Jul 29 19:15:02.032903 ip-10-0-128-173 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 29 19:15:02.032916 ip-10-0-128-173 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 29 19:15:02.032926 ip-10-0-128-173 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 29 19:15:02.033227 ip-10-0-128-173 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 29 19:15:12.033785 ip-10-0-128-173 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 29 19:15:12.033807 ip-10-0-128-173 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 3b9298500a1e4f5183f1efe93ac9916d --
Jul 29 19:17:09.061059 ip-10-0-128-173 systemd[1]: Starting Kubernetes Kubelet...
Jul 29 19:17:09.437613 ip-10-0-128-173 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 19:17:09.437613 ip-10-0-128-173 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 29 19:17:09.437613 ip-10-0-128-173 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 19:17:09.437613 ip-10-0-128-173 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 29 19:17:09.437613 ip-10-0-128-173 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 19:17:09.439242 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.439124    2557 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 29 19:17:09.445741 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445726    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 19:17:09.445741 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445742    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 19:17:09.445801 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445746    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 19:17:09.445801 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445749    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 19:17:09.445801 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445752    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 19:17:09.445801 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445755    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 19:17:09.445801 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445759    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 19:17:09.445801 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445762    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 19:17:09.445801 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445765    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 19:17:09.445801 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445767    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 19:17:09.445801 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445770    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 19:17:09.445801 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445773    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 19:17:09.445801 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445775    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 19:17:09.445801 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445778    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 19:17:09.445801 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445781    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 19:17:09.445801 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445783    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 19:17:09.445801 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445786    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 19:17:09.445801 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445788    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 19:17:09.445801 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445791    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 19:17:09.445801 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445794    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 19:17:09.445801 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445802    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 19:17:09.445801 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445806    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 19:17:09.446286 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445808    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 19:17:09.446286 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445811    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 19:17:09.446286 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445814    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 19:17:09.446286 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445817    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 19:17:09.446286 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445820    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 19:17:09.446286 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445823    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 19:17:09.446286 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445825    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 19:17:09.446286 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445828    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 19:17:09.446286 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445830    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 19:17:09.446286 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445833    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 19:17:09.446286 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445835    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 19:17:09.446286 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445838    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 19:17:09.446286 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445840    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 19:17:09.446286 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445843    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 19:17:09.446286 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445845    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 19:17:09.446286 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445848    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 19:17:09.446286 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445851    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 19:17:09.446286 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445853    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 19:17:09.446286 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445856    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 19:17:09.446286 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445858    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 19:17:09.446807 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445860    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 19:17:09.446807 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445863    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 19:17:09.446807 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445865    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 19:17:09.446807 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445868    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 19:17:09.446807 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445870    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 19:17:09.446807 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445873    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 19:17:09.446807 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445877    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 19:17:09.446807 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445879    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 19:17:09.446807 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445881    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 19:17:09.446807 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445884    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 19:17:09.446807 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445887    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 19:17:09.446807 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445889    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 19:17:09.446807 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445892    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 19:17:09.446807 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445895    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 19:17:09.446807 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445898    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 19:17:09.446807 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445900    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 19:17:09.446807 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445903    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 19:17:09.446807 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445905    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 19:17:09.446807 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445908    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 19:17:09.446807 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445911    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 19:17:09.447313 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445913    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 19:17:09.447313 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445916    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 19:17:09.447313 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445918    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 19:17:09.447313 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445921    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 19:17:09.447313 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445924    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 19:17:09.447313 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445926    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 19:17:09.447313 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445928    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 19:17:09.447313 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445931    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 19:17:09.447313 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445934    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 19:17:09.447313 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445936    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 19:17:09.447313 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445938    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 19:17:09.447313 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445941    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 19:17:09.447313 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445944    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 19:17:09.447313 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445948    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 19:17:09.447313 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445951    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 19:17:09.447313 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445953    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 19:17:09.447313 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445956    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 19:17:09.447313 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445958    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 19:17:09.447313 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445974    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 19:17:09.447313 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445981    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 19:17:09.447800 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445987    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 19:17:09.447800 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445991    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 19:17:09.447800 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445994    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 19:17:09.447800 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.445998    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 19:17:09.447800 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446002    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 19:17:09.447800 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446009    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 19:17:09.447800 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446012    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 19:17:09.447800 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446015    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 19:17:09.447800 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446018    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 19:17:09.447800 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446021    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 19:17:09.447800 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446023    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 19:17:09.447800 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446028    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 19:17:09.447800 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446031    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 19:17:09.447800 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446034    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 19:17:09.447800 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446037    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 19:17:09.447800 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446040    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 19:17:09.447800 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446042    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 19:17:09.447800 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446768    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 19:17:09.447800 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446776    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 19:17:09.448274 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446780    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 19:17:09.448274 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446782    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 19:17:09.448274 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446787    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 19:17:09.448274 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446790    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 19:17:09.448274 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446794    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 19:17:09.448274 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446797    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 19:17:09.448274 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446800    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 19:17:09.448274 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446803    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 19:17:09.448274 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446806    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 19:17:09.448274 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446809    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 19:17:09.448274 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446811    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 19:17:09.448274 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446814    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 19:17:09.448274 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446817    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 19:17:09.448274 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446820    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 19:17:09.448274 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446822    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 19:17:09.448274 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446825    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 19:17:09.448274 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446828    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 19:17:09.448274 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446831    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 19:17:09.448274 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446836    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 19:17:09.448274 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446838    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 19:17:09.448766 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446841    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 19:17:09.448766 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446843    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 19:17:09.448766 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446846    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 19:17:09.448766 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446849    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 19:17:09.448766 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446851    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 19:17:09.448766 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446854    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 19:17:09.448766 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446856    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 19:17:09.448766 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446859    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 19:17:09.448766 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446862    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 19:17:09.448766 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446864    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 19:17:09.448766 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446867    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 19:17:09.448766 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446869    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 19:17:09.448766 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446872    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 19:17:09.448766 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446874    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 19:17:09.448766 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446877    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 19:17:09.448766 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446879    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 19:17:09.448766 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446883    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 19:17:09.448766 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446886    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 19:17:09.448766 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446889    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 19:17:09.448766 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446891    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 19:17:09.449308 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446894    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 19:17:09.449308 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446897    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 19:17:09.449308 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446900    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 19:17:09.449308 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446902    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 19:17:09.449308 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446905    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 19:17:09.449308 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446908    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 19:17:09.449308 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446910    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 19:17:09.449308 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446913    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 19:17:09.449308 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446917    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 19:17:09.449308 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446921    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 19:17:09.449308 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446925    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 19:17:09.449308 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446928    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 19:17:09.449308 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446931    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 19:17:09.449308 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446934    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 19:17:09.449308 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446936    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 19:17:09.449308 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446939    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 19:17:09.449308 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446941    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 19:17:09.449308 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446944    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 19:17:09.449308 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446946    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 19:17:09.449308 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446949    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 19:17:09.449798 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446951    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 19:17:09.449798 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446956    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 19:17:09.449798 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446958    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 19:17:09.449798 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446978    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 19:17:09.449798 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446981    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 19:17:09.449798 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446983    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 19:17:09.449798 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446986    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 19:17:09.449798 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446989    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 19:17:09.449798 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446991    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 19:17:09.449798 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446994    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 19:17:09.449798 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446996    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 19:17:09.449798 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.446999    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 19:17:09.449798 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447001    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 19:17:09.449798 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447005    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 19:17:09.449798 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447007    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 19:17:09.449798 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447010    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 19:17:09.449798 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447012    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 19:17:09.449798 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447015    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 19:17:09.449798 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447018    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 19:17:09.450282 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447020    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 19:17:09.450282 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447023    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 19:17:09.450282 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447026    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 19:17:09.450282 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447030    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 19:17:09.450282 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447032    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 19:17:09.450282 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447035    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 19:17:09.450282 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447038    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 19:17:09.450282 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447040    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 19:17:09.450282 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447043    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 19:17:09.450282 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447045    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 19:17:09.450282 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447048    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 19:17:09.450282 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447050    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 19:17:09.450282 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447053    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 19:17:09.450282 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447055    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 19:17:09.450282 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447058    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 19:17:09.450282 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447061    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 19:17:09.450282 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447063    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 19:17:09.450282 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447066    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 19:17:09.450282 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447434    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 19:17:09.450282 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447438    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 19:17:09.450769 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447441    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 19:17:09.450769 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447444    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 19:17:09.450769 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447447    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 19:17:09.450769 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447449    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 19:17:09.450769 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447452    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 19:17:09.450769 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447456    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 19:17:09.450769 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447461    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 19:17:09.450769 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447465    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 19:17:09.450769 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447468    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 19:17:09.450769 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447470    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 19:17:09.450769 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447473    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 19:17:09.450769 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447476    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 19:17:09.450769 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447478    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 19:17:09.450769 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447481    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 19:17:09.450769 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447484    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 19:17:09.450769 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447487    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 19:17:09.450769 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447490    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 19:17:09.450769 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447492    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 19:17:09.450769 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447495    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 19:17:09.451236 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447498    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 19:17:09.451236 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447500    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 19:17:09.451236 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447503    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 19:17:09.451236 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447505    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 19:17:09.451236 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447508    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 19:17:09.451236 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447511    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 19:17:09.451236 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447513    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 19:17:09.451236 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447516    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 19:17:09.451236 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447519    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 19:17:09.451236 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447523    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 19:17:09.451236 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447525    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 19:17:09.451236 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447528    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 19:17:09.451236 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447531    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 19:17:09.451236 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447533    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 19:17:09.451236 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447536    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 19:17:09.451236 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447539    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 19:17:09.451236 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447541    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 19:17:09.451236 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447543    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 19:17:09.451236 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447546    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 19:17:09.451236 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447549    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 19:17:09.451725 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447552    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 19:17:09.451725 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447554    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 19:17:09.451725 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447557    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 19:17:09.451725 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447559    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 19:17:09.451725 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447561    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 19:17:09.451725 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447564    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 19:17:09.451725 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447566    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 19:17:09.451725 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447569    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 19:17:09.451725 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447573    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 19:17:09.451725 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447575    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 19:17:09.451725 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447578    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 19:17:09.451725 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447580    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 19:17:09.451725 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447584    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 19:17:09.451725 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447588    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 19:17:09.451725 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447590    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 19:17:09.451725 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447593    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 19:17:09.451725 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447596    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 19:17:09.451725 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447598    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 19:17:09.451725 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447601    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 19:17:09.452199 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447604    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 19:17:09.452199 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447607    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 19:17:09.452199 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447609    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 19:17:09.452199 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447612    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 19:17:09.452199 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447615    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 19:17:09.452199 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447617    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 19:17:09.452199 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447620    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 19:17:09.452199 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447622    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 19:17:09.452199 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447625    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 19:17:09.452199 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447628    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 19:17:09.452199 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447630    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 19:17:09.452199 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447633    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 19:17:09.452199 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447635    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 19:17:09.452199 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447637    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 19:17:09.452199 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447640    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 19:17:09.452199 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447643    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 19:17:09.452199 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447646    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 19:17:09.452199 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447648    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 19:17:09.452199 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447651    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 19:17:09.452199 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447653    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 19:17:09.452675 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447656    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 19:17:09.452675 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447660    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 19:17:09.452675 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447663    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 19:17:09.452675 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447665    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 19:17:09.452675 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447669    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 19:17:09.452675 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447672    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 19:17:09.452675 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447674    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 19:17:09.452675 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447677    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 19:17:09.452675 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447679    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 19:17:09.452675 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447682    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 19:17:09.452675 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447684    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 19:17:09.452675 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447687    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 19:17:09.452675 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447690    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 19:17:09.452675 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447692    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 19:17:09.452675 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447695    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 19:17:09.452675 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447697    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 19:17:09.452675 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447700    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 19:17:09.452675 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447703    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 19:17:09.452675 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.447705    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 19:17:09.452675 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447773    2557 flags.go:64] FLAG: --address="0.0.0.0"
Jul 29 19:17:09.452675 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447779    2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 29 19:17:09.453231 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447785    2557 flags.go:64] FLAG: --anonymous-auth="true"
Jul 29 19:17:09.453231 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447791    2557 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 29 19:17:09.453231 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447797    2557 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 29 19:17:09.453231 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447802    2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 29 19:17:09.453231 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447808    2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 29 19:17:09.453231 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447814    2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 29 19:17:09.453231 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447820    2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 29 19:17:09.453231 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447823    2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 29 19:17:09.453231 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447826    2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 29 19:17:09.453231 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447829    2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 29 19:17:09.453231 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447832    2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 29 19:17:09.453231 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447835    2557 flags.go:64] FLAG: --cgroup-root=""
Jul 29 19:17:09.453231 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447839    2557 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 29 19:17:09.453231 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447842    2557 flags.go:64] FLAG: --client-ca-file=""
Jul 29 19:17:09.453231 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447845    2557 flags.go:64] FLAG: --cloud-provider="external"
Jul 29 19:17:09.453231 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447848    2557 flags.go:64] FLAG: --cluster-dns="[]"
Jul 29 19:17:09.453231 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447851    2557 flags.go:64] FLAG: --cluster-domain=""
Jul 29 19:17:09.453231 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447854    2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 29 19:17:09.453231 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447857    2557 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 29 19:17:09.453231 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447860    2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 29 19:17:09.453231 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447864    2557 flags.go:64] FLAG: --container-log-max-files="5"
Jul 29 19:17:09.453231 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447867    2557 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 29 19:17:09.453231 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447870    2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 29 19:17:09.453785 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447873    2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 29 19:17:09.453785 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447876    2557 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 29 19:17:09.453785 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447879    2557 flags.go:64] FLAG: --contention-profiling="false"
Jul 29 19:17:09.453785 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447882    2557 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 29 19:17:09.453785 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447885    2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 29 19:17:09.453785 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447889    2557 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 29 19:17:09.453785 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447892    2557 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 29 19:17:09.453785 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447896    2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 29 19:17:09.453785 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447898    2557 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 29 19:17:09.453785 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447901    2557 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 29 19:17:09.453785 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447904    2557 flags.go:64] FLAG: --enable-load-reader="false"
Jul 29 19:17:09.453785 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447907    2557 flags.go:64] FLAG: --enable-server="true"
Jul 29 19:17:09.453785 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447910    2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 29 19:17:09.453785 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447918    2557 flags.go:64] FLAG: --event-burst="100"
Jul 29 19:17:09.453785 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447921    2557 flags.go:64] FLAG: --event-qps="50"
Jul 29 19:17:09.453785 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447924    2557 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 29 19:17:09.453785 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447932    2557 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 29 19:17:09.453785 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447935    2557 flags.go:64] FLAG: --eviction-hard=""
Jul 29 19:17:09.453785 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447939    2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 29 19:17:09.453785 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447942    2557 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 29 19:17:09.453785 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447944    2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 29 19:17:09.453785 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447947    2557 flags.go:64] FLAG: --eviction-soft=""
Jul 29 19:17:09.453785 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447951    2557 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 29 19:17:09.453785 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447954    2557 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 29 19:17:09.453785 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447957    2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 29 19:17:09.454416 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447960    2557 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 29 19:17:09.454416 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447976    2557 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 29 19:17:09.454416 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447979    2557 flags.go:64] FLAG: --fail-swap-on="true"
Jul 29 19:17:09.454416 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447982    2557 flags.go:64] FLAG: --feature-gates=""
Jul 29 19:17:09.454416 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447986    2557 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 29 19:17:09.454416 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447989    2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 29 19:17:09.454416 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447992    2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 29 19:17:09.454416 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447995    2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 29 19:17:09.454416 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.447998    2557 flags.go:64] FLAG: --healthz-port="10248"
Jul 29 19:17:09.454416 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448001    2557 flags.go:64] FLAG: --help="false"
Jul 29 19:17:09.454416 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448004    2557 flags.go:64] FLAG: --hostname-override="ip-10-0-128-173.ec2.internal"
Jul 29 19:17:09.454416 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448009    2557 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 29 19:17:09.454416 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448012    2557 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 29 19:17:09.454416 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448015    2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 29 19:17:09.454416 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448018    2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 29 19:17:09.454416 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448021    2557 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 29 19:17:09.454416 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448024    2557 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 29 19:17:09.454416 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448027    2557 flags.go:64] FLAG: --image-service-endpoint=""
Jul 29 19:17:09.454416 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448030    2557 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 29 19:17:09.454416 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448033    2557 flags.go:64] FLAG: --kube-api-burst="100"
Jul 29 19:17:09.454416 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448036    2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 29 19:17:09.454416 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448039    2557 flags.go:64] FLAG: --kube-api-qps="50"
Jul 29 19:17:09.454416 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448041    2557 flags.go:64] FLAG: --kube-reserved=""
Jul 29 19:17:09.454416 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448044    2557 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 29 19:17:09.455080 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448053    2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 29 19:17:09.455080 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448057    2557 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 29 19:17:09.455080 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448059    2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 29 19:17:09.455080 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448062    2557 flags.go:64] FLAG: --lock-file=""
Jul 29 19:17:09.455080 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448065    2557 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 29 19:17:09.455080 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448069    2557 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 29 19:17:09.455080 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448072    2557 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 29 19:17:09.455080 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448077    2557 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 29 19:17:09.455080 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448080    2557 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 29 19:17:09.455080 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448083    2557 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 29 19:17:09.455080 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448085    2557 flags.go:64] FLAG: --logging-format="text"
Jul 29 19:17:09.455080 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448088    2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 29 19:17:09.455080 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448091    2557 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 29 19:17:09.455080 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448094    2557 flags.go:64] FLAG: --manifest-url=""
Jul 29 19:17:09.455080 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448097    2557 flags.go:64] FLAG: --manifest-url-header=""
Jul 29 19:17:09.455080 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448108    2557 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 29 19:17:09.455080 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448111    2557 flags.go:64] FLAG: --max-open-files="1000000"
Jul 29 19:17:09.455080 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448114    2557 flags.go:64] FLAG: --max-pods="110"
Jul 29 19:17:09.455080 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448117    2557 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 29 19:17:09.455080 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448122    2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 29 19:17:09.455080 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448125    2557 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 29 19:17:09.455080 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448128    2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 29 19:17:09.455080 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448131    2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 29 19:17:09.455080 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448134    2557 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 29 19:17:09.455080 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448137    2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 29 19:17:09.455694 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448143    2557 flags.go:64] FLAG: --node-status-max-images="50"
Jul 29 19:17:09.455694 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448146    2557 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 29 19:17:09.455694 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448149    2557 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 29 19:17:09.455694 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448152    2557 flags.go:64] FLAG: --pod-cidr=""
Jul 29 19:17:09.455694 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448155    2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 29 19:17:09.455694 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448160    2557 flags.go:64] FLAG: --pod-manifest-path=""
Jul 29 19:17:09.455694 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448163    2557 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 29 19:17:09.455694 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448166    2557 flags.go:64] FLAG: --pods-per-core="0"
Jul 29 19:17:09.455694 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448174    2557 flags.go:64] FLAG: --port="10250"
Jul 29 19:17:09.455694 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448177    2557 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 29 19:17:09.455694 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448180    2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03790317534eb79a1"
Jul 29 19:17:09.455694 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448184    2557 flags.go:64] FLAG: --qos-reserved=""
Jul 29 19:17:09.455694 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448187    2557 flags.go:64] FLAG: --read-only-port="10255"
Jul 29 19:17:09.455694 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448189    2557 flags.go:64] FLAG: --register-node="true"
Jul 29 19:17:09.455694 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448192    2557 flags.go:64] FLAG: --register-with-taints=""
Jul 29 19:17:09.455694 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448196    2557 flags.go:64] FLAG: --registry-burst="10"
Jul 29 19:17:09.455694 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448199    2557 flags.go:64] FLAG: --registry-qps="5"
Jul 29 19:17:09.455694 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448201    2557 flags.go:64] FLAG: --reserved-cpus=""
Jul 29 19:17:09.455694 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448204    2557 flags.go:64] FLAG: --reserved-memory=""
Jul 29 19:17:09.455694 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448207    2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 29 19:17:09.455694 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448210    2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 29 19:17:09.455694 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448213    2557 flags.go:64] FLAG: --rotate-certificates="false"
Jul 29 19:17:09.455694 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448216    2557 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 29 19:17:09.455694 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448219    2557 flags.go:64] FLAG: --runonce="false"
Jul 29 19:17:09.455694 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448221    2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 29 19:17:09.456336 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448224    2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 29 19:17:09.456336 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448228    2557 flags.go:64] FLAG: --seccomp-default="false"
Jul 29 19:17:09.456336 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448231    2557 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 29 19:17:09.456336 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448234    2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 29 19:17:09.456336 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448237    2557 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 29 19:17:09.456336 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448240    2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 29 19:17:09.456336 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448243    2557 flags.go:64] FLAG: --storage-driver-password="root"
Jul 29 19:17:09.456336 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448246    2557 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 29 19:17:09.456336 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448248    2557 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 29 19:17:09.456336 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448251    2557 flags.go:64] FLAG: --storage-driver-user="root"
Jul 29 19:17:09.456336 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448254    2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 29 19:17:09.456336 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448257    2557 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 29 19:17:09.456336 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448259    2557 flags.go:64] FLAG: --system-cgroups=""
Jul 29 19:17:09.456336 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448262    2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 29 19:17:09.456336 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448267    2557 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 29 19:17:09.456336 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448270    2557 flags.go:64] FLAG: --tls-cert-file=""
Jul 29 19:17:09.456336 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448274    2557 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 29 19:17:09.456336 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448280    2557 flags.go:64] FLAG: --tls-min-version=""
Jul 29 19:17:09.456336 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448283    2557 flags.go:64] FLAG: --tls-private-key-file=""
Jul 29 19:17:09.456336 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448287    2557 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 29 19:17:09.456336 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448290    2557 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 29 19:17:09.456336 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448293    2557 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 29 19:17:09.456336 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448296    2557 flags.go:64] FLAG: --v="2"
Jul 29 19:17:09.456336 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448300    2557 flags.go:64] FLAG: --version="false"
Jul 29 19:17:09.456336 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448304    2557 flags.go:64] FLAG: --vmodule=""
Jul 29 19:17:09.456996 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448307    2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 29 19:17:09.456996 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.448310    2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 29 19:17:09.456996 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448426    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 19:17:09.456996 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448429    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 19:17:09.456996 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448433    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 19:17:09.456996 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448436    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 19:17:09.456996 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448438    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 19:17:09.456996 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448441    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 19:17:09.456996 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448445    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 19:17:09.456996 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448449    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 19:17:09.456996 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448451    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 19:17:09.456996 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448454    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 19:17:09.456996 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448456    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 19:17:09.456996 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448459    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 19:17:09.456996 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448461    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 19:17:09.456996 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448464    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 19:17:09.456996 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448466    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 19:17:09.456996 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448469    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 19:17:09.456996 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448471    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 19:17:09.456996 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448474    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 19:17:09.457496 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448476    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 19:17:09.457496 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448479    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 19:17:09.457496 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448482    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 19:17:09.457496 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448485    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 19:17:09.457496 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448488    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 19:17:09.457496 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448491    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 19:17:09.457496 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448495    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 19:17:09.457496 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448498    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 19:17:09.457496 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448500    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 19:17:09.457496 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448503    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 19:17:09.457496 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448506    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 19:17:09.457496 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448508    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 19:17:09.457496 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448511    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 19:17:09.457496 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448514    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 19:17:09.457496 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448516    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 19:17:09.457496 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448519    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 19:17:09.457496 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448521    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 19:17:09.457496 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448524    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 19:17:09.457496 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448526    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 19:17:09.457496 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448529    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 19:17:09.458006 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448534    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 19:17:09.458006 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448538    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 19:17:09.458006 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448541    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 19:17:09.458006 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448544    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 19:17:09.458006 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448547    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 19:17:09.458006 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448549    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 19:17:09.458006 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448552    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 19:17:09.458006 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448555    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 19:17:09.458006 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448558    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 19:17:09.458006 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448560    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 19:17:09.458006 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448563    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 19:17:09.458006 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448565    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 19:17:09.458006 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448568    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 19:17:09.458006 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448570    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 19:17:09.458006 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448573    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 19:17:09.458006 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448576    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 19:17:09.458006 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448578    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 19:17:09.458006 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448581    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 19:17:09.458006 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448585    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 19:17:09.458006 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448588    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 19:17:09.458536 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448591    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 19:17:09.458536 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448593    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 19:17:09.458536 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448596    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 19:17:09.458536 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448598    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 19:17:09.458536 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448601    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 19:17:09.458536 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448603    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 19:17:09.458536 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448606    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 19:17:09.458536 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448608    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 19:17:09.458536 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448611    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 19:17:09.458536 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448614    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 19:17:09.458536 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448616    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 19:17:09.458536 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448620    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 19:17:09.458536 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448625    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 19:17:09.458536 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448629    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 19:17:09.458536 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448632    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 19:17:09.458536 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448635    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 19:17:09.458536 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448638    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 19:17:09.458536 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448642    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 19:17:09.458536 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448646    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 19:17:09.459052 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448649    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 19:17:09.459052 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448652    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 19:17:09.459052 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448654    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 19:17:09.459052 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448657    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 19:17:09.459052 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448660    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 19:17:09.459052 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448663    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 19:17:09.459052 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448665    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 19:17:09.459052 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448668    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 19:17:09.459052 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448671    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 19:17:09.459052 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448673    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 19:17:09.459052 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448676    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 19:17:09.459052 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448681    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 19:17:09.459052 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448684    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 19:17:09.459052 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448686    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 19:17:09.459052 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448689    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 19:17:09.459052 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448691    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 19:17:09.459052 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448694    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 19:17:09.459052 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448696    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 19:17:09.459052 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448699    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 19:17:09.459554 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448702    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 19:17:09.459554 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448704    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 19:17:09.459554 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.448706    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 19:17:09.459554 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.449303    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 19:17:09.459554 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.456171    2557 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 29 19:17:09.459554 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.456186    2557 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 29 19:17:09.459554 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.456205    2557 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 29 19:17:09.459554 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.456212    2557 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 29 19:17:09.459554 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456267    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 19:17:09.459554 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456273    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 19:17:09.459554 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456276    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 19:17:09.459554 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456279    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 19:17:09.459554 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456282    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 19:17:09.459554 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456285    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 19:17:09.459554 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456288    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 19:17:09.459935 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456291    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 19:17:09.459935 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456293    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 19:17:09.459935 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456296    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 19:17:09.459935 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456299    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 19:17:09.459935 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456302    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 19:17:09.459935 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456305    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 19:17:09.459935 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456307    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 19:17:09.459935 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456310    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 19:17:09.459935 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456313    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 19:17:09.459935 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456315    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 19:17:09.459935 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456318    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 19:17:09.459935 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456320    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 19:17:09.459935 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456323    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 19:17:09.459935 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456325    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 19:17:09.459935 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456328    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 19:17:09.459935 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456330    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 19:17:09.459935 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456333    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 19:17:09.459935 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456335    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 19:17:09.459935 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456338    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 19:17:09.460479 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456342    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 19:17:09.460479 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456346    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 19:17:09.460479 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456349    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 19:17:09.460479 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456354    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 19:17:09.460479 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456357    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 19:17:09.460479 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456360    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 19:17:09.460479 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456363    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 19:17:09.460479 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456366    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 19:17:09.460479 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456369    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 19:17:09.460479 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456371    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 19:17:09.460479 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456374    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 19:17:09.460479 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456377    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 19:17:09.460479 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456379    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 19:17:09.460479 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456382    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 19:17:09.460479 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456385    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 19:17:09.460479 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456388    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 19:17:09.460479 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456390    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 19:17:09.460479 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456393    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 19:17:09.460479 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456395    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 19:17:09.460956 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456398    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 19:17:09.460956 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456400    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 19:17:09.460956 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456403    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 19:17:09.460956 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456406    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 19:17:09.460956 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456409    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 19:17:09.460956 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456413    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 19:17:09.460956 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456416    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 19:17:09.460956 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456418    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 19:17:09.460956 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456421    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 19:17:09.460956 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456424    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 19:17:09.460956 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456426    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 19:17:09.460956 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456429    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 19:17:09.460956 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456431    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 19:17:09.460956 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456434    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 19:17:09.460956 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456437    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 19:17:09.460956 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456439    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 19:17:09.460956 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456443    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 19:17:09.460956 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456446    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 19:17:09.460956 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456449    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 19:17:09.461433 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456451    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 19:17:09.461433 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456454    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 19:17:09.461433 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456457    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 19:17:09.461433 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456459    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 19:17:09.461433 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456462    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 19:17:09.461433 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456464    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 19:17:09.461433 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456467    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 19:17:09.461433 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456470    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 19:17:09.461433 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456473    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 19:17:09.461433 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456475    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 19:17:09.461433 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456478    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 19:17:09.461433 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456480    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 19:17:09.461433 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456484    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 19:17:09.461433 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456488    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 19:17:09.461433 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456490    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 19:17:09.461433 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456493    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 19:17:09.461433 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456496    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 19:17:09.461433 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456498    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 19:17:09.461433 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456500    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 19:17:09.461433 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456503    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 19:17:09.461945 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456506    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 19:17:09.461945 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456508    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 19:17:09.461945 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456511    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 19:17:09.461945 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456513    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 19:17:09.461945 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456516    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 19:17:09.461945 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456519    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 19:17:09.461945 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456521    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 19:17:09.461945 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456524    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 19:17:09.461945 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456527    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 19:17:09.461945 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456530    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 19:17:09.461945 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456533    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 19:17:09.461945 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456536    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 19:17:09.461945 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456538    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 19:17:09.461945 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456541    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 19:17:09.461945 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456543    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 19:17:09.462344 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.456549    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 19:17:09.462344 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456663    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 19:17:09.462344 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456668    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 19:17:09.462344 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456671    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 19:17:09.462344 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456674    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 19:17:09.462344 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456677    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 19:17:09.462344 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456680    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 19:17:09.462344 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456682    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 19:17:09.462344 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456685    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 19:17:09.462344 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456688    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 19:17:09.462344 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456691    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 19:17:09.462344 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456694    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 19:17:09.462344 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456696    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 19:17:09.462344 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456699    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 19:17:09.462344 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456702    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 19:17:09.462722 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456704    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 19:17:09.462722 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456707    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 19:17:09.462722 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456709    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 19:17:09.462722 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456712    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 19:17:09.462722 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456714    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 19:17:09.462722 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456717    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 19:17:09.462722 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456720    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 19:17:09.462722 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456722    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 19:17:09.462722 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456725    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 19:17:09.462722 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456727    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 19:17:09.462722 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456729    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 19:17:09.462722 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456733    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 19:17:09.462722 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456736    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 19:17:09.462722 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456738    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 19:17:09.462722 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456741    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 19:17:09.462722 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456744    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 19:17:09.462722 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456747    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 19:17:09.462722 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456750    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 19:17:09.462722 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456752    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 19:17:09.462722 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456755    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 19:17:09.463225 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456757    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 19:17:09.463225 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456760    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 19:17:09.463225 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456762    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 19:17:09.463225 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456765    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 19:17:09.463225 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456767    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 19:17:09.463225 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456770    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 19:17:09.463225 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456773    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 19:17:09.463225 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456775    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 19:17:09.463225 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456778    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 19:17:09.463225 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456780    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 19:17:09.463225 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456782    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 19:17:09.463225 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456785    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 19:17:09.463225 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456788    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 19:17:09.463225 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456791    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 19:17:09.463225 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456793    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 19:17:09.463225 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456804    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 19:17:09.463225 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456807    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 19:17:09.463225 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456809    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 19:17:09.463225 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456812    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 19:17:09.463225 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456814    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 19:17:09.463761 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456817    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 19:17:09.463761 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456820    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 19:17:09.463761 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456823    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 19:17:09.463761 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456829    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 19:17:09.463761 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456833    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 19:17:09.463761 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456836    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 19:17:09.463761 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456839    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 19:17:09.463761 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456842    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 19:17:09.463761 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456845    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 19:17:09.463761 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456847    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 19:17:09.463761 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456850    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 19:17:09.463761 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456852    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 19:17:09.463761 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456855    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 19:17:09.463761 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456857    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 19:17:09.463761 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456860    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 19:17:09.463761 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456862    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 19:17:09.463761 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456865    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 19:17:09.463761 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456867    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 19:17:09.463761 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456870    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 19:17:09.463761 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456872    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 19:17:09.464287 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456875    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 19:17:09.464287 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456877    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 19:17:09.464287 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456880    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 19:17:09.464287 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456883    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 19:17:09.464287 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456885    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 19:17:09.464287 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456888    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 19:17:09.464287 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456890    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 19:17:09.464287 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456892    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 19:17:09.464287 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456895    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 19:17:09.464287 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456897    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 19:17:09.464287 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456900    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 19:17:09.464287 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456902    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 19:17:09.464287 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456905    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 19:17:09.464287 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456907    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 19:17:09.464287 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456910    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 19:17:09.464287 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456913    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 19:17:09.464287 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456915    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 19:17:09.464287 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456919    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 19:17:09.464287 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456923    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 19:17:09.464925 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456926    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 19:17:09.464925 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456929    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 19:17:09.464925 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456932    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 19:17:09.464925 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456935    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 19:17:09.464925 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456937    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 19:17:09.464925 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:09.456940    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 19:17:09.464925 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.456945    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 19:17:09.464925 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.457055    2557 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 29 19:17:09.464925 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.459016    2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 29 19:17:09.464925 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.460233    2557 server.go:1019] "Starting client certificate rotation"
Jul 29 19:17:09.464925 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.460332    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 19:17:09.464925 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.460363    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 19:17:09.464925 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.462345    2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 19:17:09.466690 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.466668    2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 19:17:09.476069 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.476054    2557 log.go:25] "Validated CRI v1 runtime API"
Jul 29 19:17:09.477994 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.477979    2557 log.go:25] "Validated CRI v1 image API"
Jul 29 19:17:09.478748 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.478734    2557 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 29 19:17:09.481977 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.481935    2557 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 86604636-f8fa-4b9c-bcc2-658f005f3c67:/dev/nvme0n1p4 c22cac95-02d5-43e6-aa8b-c939073c53c7:/dev/nvme0n1p3]
Jul 29 19:17:09.482030 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.481959    2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 29 19:17:09.486551 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.486445    2557 manager.go:216] Machine: {Timestamp:2026-07-29 19:17:09.485984187 +0000 UTC m=+0.313483236 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3073497 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25e63aba7582a35db5f09290fec8e7 SystemUUID:ec25e63a-ba75-82a3-5db5-f09290fec8e7 BootID:3b929850-0a1e-4f51-83f1-efe93ac9916d Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b4:d7:f4:e3:83 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b4:d7:f4:e3:83 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:07:b0:52:8a:d2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 29 19:17:09.486551 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.486550    2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 29 19:17:09.486656 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.486617    2557 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 29 19:17:09.490219 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.490204    2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 19:17:09.490571 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.490544    2557 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 29 19:17:09.490705 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.490577    2557 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-173.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 29 19:17:09.491188 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.491177    2557 topology_manager.go:138] "Creating topology manager with none policy"
Jul 29 19:17:09.491225 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.491190    2557 container_manager_linux.go:309] "Creating device plugin manager"
Jul 29 19:17:09.491225 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.491198    2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 19:17:09.491949 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.491938    2557 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 19:17:09.492599 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.492589    2557 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 29 19:17:09.494062 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.494051    2557 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 19:17:09.494179 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.494170    2557 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 29 19:17:09.494215 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.494199    2557 kubelet.go:486] "Attempting to sync node with API server"
Jul 29 19:17:09.494215 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.494213    2557 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 29 19:17:09.494268 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.494227    2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 29 19:17:09.494268 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.494235    2557 kubelet.go:398] "Adding apiserver pod source"
Jul 29 19:17:09.494268 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.494244    2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 29 19:17:09.494562 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.494549    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 19:17:09.494597 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.494570    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 19:17:09.497802 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.497787    2557 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 29 19:17:09.499359 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.499345    2557 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 29 19:17:09.499425 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.499375    2557 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 29 19:17:09.501000 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.500987    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 29 19:17:09.501053 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.501004    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 29 19:17:09.501053 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.501011    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 29 19:17:09.501053 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.501016    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 29 19:17:09.501053 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.501024    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 29 19:17:09.501053 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.501030    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 29 19:17:09.501053 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.501038    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 29 19:17:09.501053 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.501043    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 29 19:17:09.501053 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.501051    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 29 19:17:09.501053 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.501056    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 29 19:17:09.501283 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.501065    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 29 19:17:09.501283 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.501070    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 29 19:17:09.502344 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.502332    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 29 19:17:09.502344 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.502343    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 29 19:17:09.504379 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.504360    2557 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-173.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 29 19:17:09.504379 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:09.504367    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 29 19:17:09.504468 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:09.504368    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-173.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 29 19:17:09.505400 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.505387    2557 server.go:1268] "Started kubelet"
Jul 29 19:17:09.505497 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.505472    2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 29 19:17:09.505712 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.505685    2557 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 29 19:17:09.505788 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.505722    2557 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 29 19:17:09.509568 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.509535    2557 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 29 19:17:09.509846 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.509825    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lnq2c"
Jul 29 19:17:09.510270 ip-10-0-128-173 systemd[1]: Started Kubernetes Kubelet.
Jul 29 19:17:09.511544 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.511353    2557 server.go:310] "Adding debug handlers to kubelet server"
Jul 29 19:17:09.513212 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:09.513186    2557 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 29 19:17:09.516993 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.516976    2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 29 19:17:09.517085 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:09.517058    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-173.ec2.internal\" not found"
Jul 29 19:17:09.517137 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.516957    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 29 19:17:09.517783 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.517765    2557 factory.go:55] Registering systemd factory
Jul 29 19:17:09.517783 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.517784    2557 factory.go:223] Registration of the systemd container factory successfully
Jul 29 19:17:09.518060 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.518041    2557 factory.go:153] Registering CRI-O factory
Jul 29 19:17:09.518060 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.518060    2557 factory.go:223] Registration of the crio container factory successfully
Jul 29 19:17:09.518178 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.518112    2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 29 19:17:09.518178 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.518141    2557 factory.go:103] Registering Raw factory
Jul 29 19:17:09.518178 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.518149    2557 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 29 19:17:09.518178 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.518159    2557 manager.go:1196] Started watching for new ooms in manager
Jul 29 19:17:09.518178 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.518162    2557 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 29 19:17:09.518341 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.518151    2557 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 29 19:17:09.518341 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.518258    2557 reconstruct.go:97] "Volume reconstruction finished"
Jul 29 19:17:09.518341 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.518264    2557 reconciler.go:29] "Reconciler: start to sync state"
Jul 29 19:17:09.518583 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.518570    2557 manager.go:349] Starting recovery of all containers
Jul 29 19:17:09.519191 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.519172    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lnq2c"
Jul 29 19:17:09.520652 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:09.520630    2557 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-173.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 29 19:17:09.520750 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:09.520725    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 29 19:17:09.521450 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:09.520554    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-173.ec2.internal.18c6d8c1e07a0387  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-173.ec2.internal,UID:ip-10-0-128-173.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-173.ec2.internal,},FirstTimestamp:2026-07-29 19:17:09.505360775 +0000 UTC m=+0.332859824,LastTimestamp:2026-07-29 19:17:09.505360775 +0000 UTC m=+0.332859824,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-173.ec2.internal,}"
Jul 29 19:17:09.528193 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.528169    2557 manager.go:354] Recovery completed
Jul 29 19:17:09.532386 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.532372    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 19:17:09.538930 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.538903    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-173.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 19:17:09.539031 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.538955    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-173.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 19:17:09.539031 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.538992    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-173.ec2.internal" event="NodeHasSufficientPID"
Jul 29 19:17:09.539565 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.539549    2557 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 29 19:17:09.539565 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.539563    2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 29 19:17:09.539647 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.539568    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 19:17:09.539647 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.539578    2557 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 19:17:09.540278 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.540260    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-173.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 19:17:09.540324 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.540289    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-173.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 19:17:09.540324 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.540301    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-173.ec2.internal" event="NodeHasSufficientPID"
Jul 29 19:17:09.541072 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.541057    2557 policy_none.go:49] "None policy: Start"
Jul 29 19:17:09.541131 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.541073    2557 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 29 19:17:09.541131 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.541085    2557 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 29 19:17:09.541924 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.541914    2557 policy_none.go:47] "Start"
Jul 29 19:17:09.583691 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.583514    2557 manager.go:337] "Starting Device Plugin manager"
Jul 29 19:17:09.583775 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:09.583716    2557 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 29 19:17:09.583775 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.583729    2557 server.go:87] "Starting device plugin registration server"
Jul 29 19:17:09.583995 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.583983    2557 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 29 19:17:09.584032 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.583997    2557 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 29 19:17:09.584094 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.584076    2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 29 19:17:09.584183 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.584164    2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 29 19:17:09.584183 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.584178    2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 29 19:17:09.584678 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:09.584653    2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 29 19:17:09.584765 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:09.584713    2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-173.ec2.internal\" not found"
Jul 29 19:17:09.670926 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.670887    2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 29 19:17:09.672149 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.672134    2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 29 19:17:09.672256 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.672176    2557 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 29 19:17:09.672256 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.672205    2557 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 29 19:17:09.672256 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:09.672246    2557 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 29 19:17:09.677182 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.677162    2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 19:17:09.684742 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.684729    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 19:17:09.685570 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.685555    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-173.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 19:17:09.685654 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.685585    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-173.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 19:17:09.685654 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.685598    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-173.ec2.internal" event="NodeHasSufficientPID"
Jul 29 19:17:09.685654 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.685620    2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-173.ec2.internal"
Jul 29 19:17:09.694501 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.694463    2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-173.ec2.internal"
Jul 29 19:17:09.694501 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:09.694484    2557 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-173.ec2.internal\": node \"ip-10-0-128-173.ec2.internal\" not found"
Jul 29 19:17:09.714775 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:09.714755    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-173.ec2.internal\" not found"
Jul 29 19:17:09.772854 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.772817    2557 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-173.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-173.ec2.internal"]
Jul 29 19:17:09.772940 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.772902    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 19:17:09.773848 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.773834    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-173.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 19:17:09.773898 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.773865    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-173.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 19:17:09.773898 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.773876    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-173.ec2.internal" event="NodeHasSufficientPID"
Jul 29 19:17:09.775144 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.775130    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 19:17:09.775256 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.775241    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-173.ec2.internal"
Jul 29 19:17:09.775306 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.775272    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 19:17:09.775913 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.775897    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-173.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 19:17:09.775913 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.775907    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-173.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 19:17:09.776065 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.775929    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-173.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 19:17:09.776065 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.775933    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-173.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 19:17:09.776065 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.775945    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-173.ec2.internal" event="NodeHasSufficientPID"
Jul 29 19:17:09.776065 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.775945    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-173.ec2.internal" event="NodeHasSufficientPID"
Jul 29 19:17:09.777117 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.777101    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-173.ec2.internal"
Jul 29 19:17:09.777199 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.777126    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 19:17:09.777832 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.777813    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-173.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 19:17:09.777922 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.777838    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-173.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 19:17:09.777922 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.777849    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-173.ec2.internal" event="NodeHasSufficientPID"
Jul 29 19:17:09.802705 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:09.802682    2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-173.ec2.internal\" not found" node="ip-10-0-128-173.ec2.internal"
Jul 29 19:17:09.807386 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:09.807367    2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-173.ec2.internal\" not found" node="ip-10-0-128-173.ec2.internal"
Jul 29 19:17:09.815471 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:09.815452    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-173.ec2.internal\" not found"
Jul 29 19:17:09.818970 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.818955    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6dd50be5d2585bd24e7a6b31885c789e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-173.ec2.internal\" (UID: \"6dd50be5d2585bd24e7a6b31885c789e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-173.ec2.internal"
Jul 29 19:17:09.819024 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.819007    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6dd50be5d2585bd24e7a6b31885c789e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-173.ec2.internal\" (UID: \"6dd50be5d2585bd24e7a6b31885c789e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-173.ec2.internal"
Jul 29 19:17:09.819077 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.819032    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70a21ba21338b6ba758e27d2cca1c0ab-config\") pod \"kube-apiserver-proxy-ip-10-0-128-173.ec2.internal\" (UID: \"70a21ba21338b6ba758e27d2cca1c0ab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-173.ec2.internal"
Jul 29 19:17:09.916156 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:09.916127    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-173.ec2.internal\" not found"
Jul 29 19:17:09.919620 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.919606    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6dd50be5d2585bd24e7a6b31885c789e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-173.ec2.internal\" (UID: \"6dd50be5d2585bd24e7a6b31885c789e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-173.ec2.internal"
Jul 29 19:17:09.919667 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.919630    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6dd50be5d2585bd24e7a6b31885c789e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-173.ec2.internal\" (UID: \"6dd50be5d2585bd24e7a6b31885c789e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-173.ec2.internal"
Jul 29 19:17:09.919667 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.919650    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70a21ba21338b6ba758e27d2cca1c0ab-config\") pod \"kube-apiserver-proxy-ip-10-0-128-173.ec2.internal\" (UID: \"70a21ba21338b6ba758e27d2cca1c0ab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-173.ec2.internal"
Jul 29 19:17:09.919748 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.919689    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70a21ba21338b6ba758e27d2cca1c0ab-config\") pod \"kube-apiserver-proxy-ip-10-0-128-173.ec2.internal\" (UID: \"70a21ba21338b6ba758e27d2cca1c0ab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-173.ec2.internal"
Jul 29 19:17:09.919748 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.919727    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6dd50be5d2585bd24e7a6b31885c789e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-173.ec2.internal\" (UID: \"6dd50be5d2585bd24e7a6b31885c789e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-173.ec2.internal"
Jul 29 19:17:09.919816 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:09.919727    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6dd50be5d2585bd24e7a6b31885c789e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-173.ec2.internal\" (UID: \"6dd50be5d2585bd24e7a6b31885c789e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-173.ec2.internal"
Jul 29 19:17:10.017685 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:10.017604    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-173.ec2.internal\" not found"
Jul 29 19:17:10.108297 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:10.108251    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-173.ec2.internal"
Jul 29 19:17:10.110600 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:10.110578    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-173.ec2.internal"
Jul 29 19:17:10.118013 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:10.117995    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-173.ec2.internal\" not found"
Jul 29 19:17:10.221088 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:10.221055    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-173.ec2.internal\" not found"
Jul 29 19:17:10.321877 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:10.321771    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-173.ec2.internal\" not found"
Jul 29 19:17:10.422333 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:10.422279    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-173.ec2.internal\" not found"
Jul 29 19:17:10.461450 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:10.461420    2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 29 19:17:10.461932 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:10.461604    2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 19:17:10.522495 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:10.522460    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-173.ec2.internal\" not found"
Jul 29 19:17:10.523674 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:10.523639    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 19:12:09 +0000 UTC" deadline="2028-03-08 18:43:16.876832062 +0000 UTC"
Jul 29 19:17:10.523729 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:10.523675    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14111h26m6.353160563s"
Jul 29 19:17:10.623148 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:10.623028    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-173.ec2.internal\" not found"
Jul 29 19:17:10.684754 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:10.684728    2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 19:17:10.723998 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:10.723955    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-173.ec2.internal\" not found"
Jul 29 19:17:10.814646 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:10.814613    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd50be5d2585bd24e7a6b31885c789e.slice/crio-cfcfae0a16fedfb20d06e0e89a64b5490a6a43e81809a3a88f42060d695bd43d WatchSource:0}: Error finding container cfcfae0a16fedfb20d06e0e89a64b5490a6a43e81809a3a88f42060d695bd43d: Status 404 returned error can't find the container with id cfcfae0a16fedfb20d06e0e89a64b5490a6a43e81809a3a88f42060d695bd43d
Jul 29 19:17:10.815031 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:10.815005    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70a21ba21338b6ba758e27d2cca1c0ab.slice/crio-8457dd5f206f47a28e172a2c11f418272c6b61450173c146f9842e2baf53ee03 WatchSource:0}: Error finding container 8457dd5f206f47a28e172a2c11f418272c6b61450173c146f9842e2baf53ee03: Status 404 returned error can't find the container with id 8457dd5f206f47a28e172a2c11f418272c6b61450173c146f9842e2baf53ee03
Jul 29 19:17:10.820697 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:10.820686    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 19:17:10.824567 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:10.824548    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-173.ec2.internal\" not found"
Jul 29 19:17:10.880828 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:10.880742    2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 19:17:10.904296 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:10.904275    2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 19:17:10.917565 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:10.917535    2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-173.ec2.internal"
Jul 29 19:17:10.931375 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:10.931338    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 19:17:10.933051 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:10.933033    2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-173.ec2.internal"
Jul 29 19:17:10.943338 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:10.943323    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 19:17:11.496330 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.496297    2557 apiserver.go:52] "Watching apiserver"
Jul 29 19:17:11.512374 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.511378    2557 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 29 19:17:11.512374 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.511949    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-tqpxs","openshift-network-operator/iptables-alerter-h7mz2","openshift-ovn-kubernetes/ovnkube-node-q6hd8","kube-system/kube-apiserver-proxy-ip-10-0-128-173.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jntx6","openshift-cluster-node-tuning-operator/tuned-lcx4k","openshift-dns/node-resolver-vft97","openshift-image-registry/node-ca-frvmp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-173.ec2.internal","openshift-multus/multus-additional-cni-plugins-v67cp","openshift-multus/multus-c528d","kube-system/global-pull-secret-syncer-v5x8q","kube-system/konnectivity-agent-6jmnn","openshift-multus/network-metrics-daemon-cw75m"]
Jul 29 19:17:11.513547 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.513517    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jntx6"
Jul 29 19:17:11.514955 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.514660    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cw75m"
Jul 29 19:17:11.514955 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:11.514758    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cw75m" podUID="a4f202a3-5dc7-416f-8793-e857e6086174"
Jul 29 19:17:11.515866 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.515707    2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 19:17:11.515866 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.515787    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v67cp"
Jul 29 19:17:11.517119 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.516726    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 29 19:17:11.517119 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.516943    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-l8llc\""
Jul 29 19:17:11.517298 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.517274    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.517870 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.517815    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 29 19:17:11.517994 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.517887    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 29 19:17:11.519430 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.518541    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 29 19:17:11.519430 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.518682    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.519430 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.519207    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 29 19:17:11.519430 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.519221    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 29 19:17:11.519430 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.519300    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 29 19:17:11.519430 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.519311    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 29 19:17:11.519779 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.519585    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 29 19:17:11.520866 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.520169    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6rqgc\""
Jul 29 19:17:11.520866 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.520387    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 29 19:17:11.520866 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.520403    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vft97"
Jul 29 19:17:11.521751 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.521371    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-frvmp"
Jul 29 19:17:11.521751 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.521425    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 29 19:17:11.521751 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.521521    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 29 19:17:11.521751 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.521591    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zt8hq\""
Jul 29 19:17:11.522029 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.521802    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 29 19:17:11.522029 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.521887    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 29 19:17:11.522144 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.522033    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 29 19:17:11.522513 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.522342    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 29 19:17:11.522513 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.522425    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p8jqc\""
Jul 29 19:17:11.522681 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.522535    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 29 19:17:11.522750 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.522719    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqpxs"
Jul 29 19:17:11.522844 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:11.522781    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqpxs" podUID="2677a483-66e1-42fd-94f5-3903b7af3690"
Jul 29 19:17:11.523899 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.523879    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.523996 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.523897    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 29 19:17:11.527357 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.525765    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 29 19:17:11.527357 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.526116    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 29 19:17:11.527357 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.526652    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-njst7\""
Jul 29 19:17:11.527357 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.527012    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 29 19:17:11.527593 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.527527    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hbdfk\""
Jul 29 19:17:11.527593 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.527590    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 29 19:17:11.528074 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.527838    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 29 19:17:11.528074 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.527873    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7zpns\""
Jul 29 19:17:11.528074 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.528024    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:17:11.528292 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:11.528153    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5x8q" podUID="93168299-ac4f-4fdd-8ac3-b69edb385d71"
Jul 29 19:17:11.528292 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.528278    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c4bcc5ff-6023-4514-b5d1-61657a854bba-registration-dir\") pod \"aws-ebs-csi-driver-node-jntx6\" (UID: \"c4bcc5ff-6023-4514-b5d1-61657a854bba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jntx6"
Jul 29 19:17:11.528412 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.528312    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-var-lib-openvswitch\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.528412 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.528338    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40e0ec6a-ab60-4892-8adf-45cda1a71526-tmp\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.528412 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.528360    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c4bcc5ff-6023-4514-b5d1-61657a854bba-sys-fs\") pod \"aws-ebs-csi-driver-node-jntx6\" (UID: \"c4bcc5ff-6023-4514-b5d1-61657a854bba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jntx6"
Jul 29 19:17:11.528580 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.528395    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08c90b03-c2a2-478b-8d59-bb71b15e96f8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v67cp\" (UID: \"08c90b03-c2a2-478b-8d59-bb71b15e96f8\") " pod="openshift-multus/multus-additional-cni-plugins-v67cp"
Jul 29 19:17:11.528580 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.528443    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40e0ec6a-ab60-4892-8adf-45cda1a71526-sys\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.528580 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.528493    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40e0ec6a-ab60-4892-8adf-45cda1a71526-var-lib-kubelet\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.528580 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.528520    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/067b4107-d3a3-4fb2-9c30-90fab2dab639-env-overrides\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.528580 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.528554    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40e0ec6a-ab60-4892-8adf-45cda1a71526-etc-kubernetes\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.528580 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.528580    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/40e0ec6a-ab60-4892-8adf-45cda1a71526-lib-modules\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.528886 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.528596    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-systemd-units\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.528886 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.528613    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-etc-openvswitch\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.528886 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.528628    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/40e0ec6a-ab60-4892-8adf-45cda1a71526-etc-sysconfig\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.528886 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.528669    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40e0ec6a-ab60-4892-8adf-45cda1a71526-host\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.528886 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.528685    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f737cfa7-db52-409a-b5c3-f704b9170fa4-hosts-file\") pod \"node-resolver-vft97\" (UID: \"f737cfa7-db52-409a-b5c3-f704b9170fa4\") " pod="openshift-dns/node-resolver-vft97"
Jul 29 19:17:11.528886 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.528701    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krfsd\" (UniqueName: \"kubernetes.io/projected/f737cfa7-db52-409a-b5c3-f704b9170fa4-kube-api-access-krfsd\") pod \"node-resolver-vft97\" (UID: \"f737cfa7-db52-409a-b5c3-f704b9170fa4\") " pod="openshift-dns/node-resolver-vft97"
Jul 29 19:17:11.528886 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.528717    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-run-ovn\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.528886 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.528734    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-node-log\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.528886 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.528754    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-log-socket\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.528886 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.528777    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/40e0ec6a-ab60-4892-8adf-45cda1a71526-etc-sysctl-conf\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.528886 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.528799    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/40e0ec6a-ab60-4892-8adf-45cda1a71526-etc-systemd\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.528886 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.528836    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/40e0ec6a-ab60-4892-8adf-45cda1a71526-etc-tuned\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.528886 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.528875    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dvvj\" (UniqueName: \"kubernetes.io/projected/c4bcc5ff-6023-4514-b5d1-61657a854bba-kube-api-access-5dvvj\") pod \"aws-ebs-csi-driver-node-jntx6\" (UID: \"c4bcc5ff-6023-4514-b5d1-61657a854bba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jntx6"
Jul 29 19:17:11.529565 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.528915    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmjv7\" (UniqueName: \"kubernetes.io/projected/067b4107-d3a3-4fb2-9c30-90fab2dab639-kube-api-access-hmjv7\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.529565 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.528941    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/08c90b03-c2a2-478b-8d59-bb71b15e96f8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v67cp\" (UID: \"08c90b03-c2a2-478b-8d59-bb71b15e96f8\") " pod="openshift-multus/multus-additional-cni-plugins-v67cp"
Jul 29 19:17:11.529565 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.528959    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/08c90b03-c2a2-478b-8d59-bb71b15e96f8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v67cp\" (UID: \"08c90b03-c2a2-478b-8d59-bb71b15e96f8\") " pod="openshift-multus/multus-additional-cni-plugins-v67cp"
Jul 29 19:17:11.529565 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.528982    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgx88\" (UniqueName: \"kubernetes.io/projected/08c90b03-c2a2-478b-8d59-bb71b15e96f8-kube-api-access-lgx88\") pod \"multus-additional-cni-plugins-v67cp\" (UID: \"08c90b03-c2a2-478b-8d59-bb71b15e96f8\") " pod="openshift-multus/multus-additional-cni-plugins-v67cp"
Jul 29 19:17:11.529565 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.529003    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f737cfa7-db52-409a-b5c3-f704b9170fa4-tmp-dir\") pod \"node-resolver-vft97\" (UID: \"f737cfa7-db52-409a-b5c3-f704b9170fa4\") " pod="openshift-dns/node-resolver-vft97"
Jul 29 19:17:11.529565 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.529026    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c4bcc5ff-6023-4514-b5d1-61657a854bba-socket-dir\") pod \"aws-ebs-csi-driver-node-jntx6\" (UID: \"c4bcc5ff-6023-4514-b5d1-61657a854bba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jntx6"
Jul 29 19:17:11.529565 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.529100    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hchnn\" (UniqueName: \"kubernetes.io/projected/a4f202a3-5dc7-416f-8793-e857e6086174-kube-api-access-hchnn\") pod \"network-metrics-daemon-cw75m\" (UID: \"a4f202a3-5dc7-416f-8793-e857e6086174\") " pod="openshift-multus/network-metrics-daemon-cw75m"
Jul 29 19:17:11.529565 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.529125    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/40e0ec6a-ab60-4892-8adf-45cda1a71526-run\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.529565 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.529165    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p76tm\" (UniqueName: \"kubernetes.io/projected/40e0ec6a-ab60-4892-8adf-45cda1a71526-kube-api-access-p76tm\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.529565 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.529197    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c4bcc5ff-6023-4514-b5d1-61657a854bba-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jntx6\" (UID: \"c4bcc5ff-6023-4514-b5d1-61657a854bba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jntx6"
Jul 29 19:17:11.529565 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.529223    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4f202a3-5dc7-416f-8793-e857e6086174-metrics-certs\") pod \"network-metrics-daemon-cw75m\" (UID: \"a4f202a3-5dc7-416f-8793-e857e6086174\") " pod="openshift-multus/network-metrics-daemon-cw75m"
Jul 29 19:17:11.529565 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.529249    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-run-systemd\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.529565 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.529273    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-host-cni-netd\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.529565 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.529300    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c4bcc5ff-6023-4514-b5d1-61657a854bba-device-dir\") pod \"aws-ebs-csi-driver-node-jntx6\" (UID: \"c4bcc5ff-6023-4514-b5d1-61657a854bba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jntx6"
Jul 29 19:17:11.529565 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.529332    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c4bcc5ff-6023-4514-b5d1-61657a854bba-etc-selinux\") pod \"aws-ebs-csi-driver-node-jntx6\" (UID: \"c4bcc5ff-6023-4514-b5d1-61657a854bba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jntx6"
Jul 29 19:17:11.529565 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.529365    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/067b4107-d3a3-4fb2-9c30-90fab2dab639-ovn-node-metrics-cert\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.530373 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.529404    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-host-kubelet\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.530373 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.529430    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/40e0ec6a-ab60-4892-8adf-45cda1a71526-etc-sysctl-d\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.530373 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.529490    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-host-slash\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.530373 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.529519    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-host-run-netns\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.530373 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.529563    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/40e0ec6a-ab60-4892-8adf-45cda1a71526-etc-modprobe-d\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.530373 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.529599    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08c90b03-c2a2-478b-8d59-bb71b15e96f8-cni-binary-copy\") pod \"multus-additional-cni-plugins-v67cp\" (UID: \"08c90b03-c2a2-478b-8d59-bb71b15e96f8\") " pod="openshift-multus/multus-additional-cni-plugins-v67cp"
Jul 29 19:17:11.530373 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.529645    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-run-openvswitch\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.530373 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.529690    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/067b4107-d3a3-4fb2-9c30-90fab2dab639-ovnkube-config\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.530373 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.529756    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08c90b03-c2a2-478b-8d59-bb71b15e96f8-system-cni-dir\") pod \"multus-additional-cni-plugins-v67cp\" (UID: \"08c90b03-c2a2-478b-8d59-bb71b15e96f8\") " pod="openshift-multus/multus-additional-cni-plugins-v67cp"
Jul 29 19:17:11.530373 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.529787    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08c90b03-c2a2-478b-8d59-bb71b15e96f8-cnibin\") pod \"multus-additional-cni-plugins-v67cp\" (UID: \"08c90b03-c2a2-478b-8d59-bb71b15e96f8\") " pod="openshift-multus/multus-additional-cni-plugins-v67cp"
Jul 29 19:17:11.530373 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.529823    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/08c90b03-c2a2-478b-8d59-bb71b15e96f8-os-release\") pod \"multus-additional-cni-plugins-v67cp\" (UID: \"08c90b03-c2a2-478b-8d59-bb71b15e96f8\") " pod="openshift-multus/multus-additional-cni-plugins-v67cp"
Jul 29 19:17:11.530373 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.529854    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-host-run-ovn-kubernetes\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.530373 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.529902    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-host-cni-bin\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.531009 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.529927    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.531009 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.530598    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/067b4107-d3a3-4fb2-9c30-90fab2dab639-ovnkube-script-lib\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.531009 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.530781    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6jmnn"
Jul 29 19:17:11.534319 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.532298    2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 19:17:11.534319 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.533668    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 29 19:17:11.534319 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.533874    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dg5pc\""
Jul 29 19:17:11.534530 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.534322    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 29 19:17:11.536967 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.535740    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h7mz2"
Jul 29 19:17:11.538774 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.538753    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 29 19:17:11.539031 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.538997    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bs74d\""
Jul 29 19:17:11.539223 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.539205    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 29 19:17:11.539411 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.539394    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 29 19:17:11.553546 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.553475    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fwqcx"
Jul 29 19:17:11.563248 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.562868    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fwqcx"
Jul 29 19:17:11.621171 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.620249    2557 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 29 19:17:11.631860 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.631829    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08c90b03-c2a2-478b-8d59-bb71b15e96f8-system-cni-dir\") pod \"multus-additional-cni-plugins-v67cp\" (UID: \"08c90b03-c2a2-478b-8d59-bb71b15e96f8\") " pod="openshift-multus/multus-additional-cni-plugins-v67cp"
Jul 29 19:17:11.631977 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.631876    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08c90b03-c2a2-478b-8d59-bb71b15e96f8-cnibin\") pod \"multus-additional-cni-plugins-v67cp\" (UID: \"08c90b03-c2a2-478b-8d59-bb71b15e96f8\") " pod="openshift-multus/multus-additional-cni-plugins-v67cp"
Jul 29 19:17:11.631977 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.631903    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/08c90b03-c2a2-478b-8d59-bb71b15e96f8-os-release\") pod \"multus-additional-cni-plugins-v67cp\" (UID: \"08c90b03-c2a2-478b-8d59-bb71b15e96f8\") " pod="openshift-multus/multus-additional-cni-plugins-v67cp"
Jul 29 19:17:11.631977 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.631938    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-host-run-ovn-kubernetes\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.631977 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.631963    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-host-cni-bin\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.632314 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632007    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.632314 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632036    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/067b4107-d3a3-4fb2-9c30-90fab2dab639-ovnkube-script-lib\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.632314 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632062    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c4bcc5ff-6023-4514-b5d1-61657a854bba-registration-dir\") pod \"aws-ebs-csi-driver-node-jntx6\" (UID: \"c4bcc5ff-6023-4514-b5d1-61657a854bba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jntx6"
Jul 29 19:17:11.632314 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632097    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-var-lib-openvswitch\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.632314 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632124    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40e0ec6a-ab60-4892-8adf-45cda1a71526-tmp\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.632314 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632156    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/639a786b-08de-453b-8733-28f54e9a59ff-konnectivity-ca\") pod \"konnectivity-agent-6jmnn\" (UID: \"639a786b-08de-453b-8733-28f54e9a59ff\") " pod="kube-system/konnectivity-agent-6jmnn"
Jul 29 19:17:11.632314 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632181    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-cnibin\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.632314 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632208    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-host-var-lib-cni-multus\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.632314 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632232    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-hostroot\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.632314 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632255    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-multus-conf-dir\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.632314 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632282    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c4bcc5ff-6023-4514-b5d1-61657a854bba-sys-fs\") pod \"aws-ebs-csi-driver-node-jntx6\" (UID: \"c4bcc5ff-6023-4514-b5d1-61657a854bba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jntx6"
Jul 29 19:17:11.632314 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632309    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08c90b03-c2a2-478b-8d59-bb71b15e96f8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v67cp\" (UID: \"08c90b03-c2a2-478b-8d59-bb71b15e96f8\") " pod="openshift-multus/multus-additional-cni-plugins-v67cp"
Jul 29 19:17:11.632935 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632336    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40e0ec6a-ab60-4892-8adf-45cda1a71526-sys\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.632935 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632370    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40e0ec6a-ab60-4892-8adf-45cda1a71526-var-lib-kubelet\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.632935 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632396    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rx2m\" (UniqueName: \"kubernetes.io/projected/2677a483-66e1-42fd-94f5-3903b7af3690-kube-api-access-6rx2m\") pod \"network-check-target-tqpxs\" (UID: \"2677a483-66e1-42fd-94f5-3903b7af3690\") " pod="openshift-network-diagnostics/network-check-target-tqpxs"
Jul 29 19:17:11.632935 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632423    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-cni-binary-copy\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.632935 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632469    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-host-run-multus-certs\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.632935 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632498    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-etc-kubernetes\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.632935 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632537    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/067b4107-d3a3-4fb2-9c30-90fab2dab639-env-overrides\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.632935 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632564    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40e0ec6a-ab60-4892-8adf-45cda1a71526-etc-kubernetes\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.632935 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632589    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdstk\" (UniqueName: \"kubernetes.io/projected/f4b5376d-e96c-465e-adfb-bcd266b89eea-kube-api-access-xdstk\") pod \"iptables-alerter-h7mz2\" (UID: \"f4b5376d-e96c-465e-adfb-bcd266b89eea\") " pod="openshift-network-operator/iptables-alerter-h7mz2"
Jul 29 19:17:11.632935 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632618    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22vkc\" (UniqueName: \"kubernetes.io/projected/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-kube-api-access-22vkc\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.632935 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632642    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/93168299-ac4f-4fdd-8ac3-b69edb385d71-dbus\") pod \"global-pull-secret-syncer-v5x8q\" (UID: \"93168299-ac4f-4fdd-8ac3-b69edb385d71\") " pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:17:11.632935 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632669    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/40e0ec6a-ab60-4892-8adf-45cda1a71526-lib-modules\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.632935 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632693    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmnv7\" (UniqueName: \"kubernetes.io/projected/d160081b-f780-4c60-85f9-04b9a6d1c3d9-kube-api-access-cmnv7\") pod \"node-ca-frvmp\" (UID: \"d160081b-f780-4c60-85f9-04b9a6d1c3d9\") " pod="openshift-image-registry/node-ca-frvmp"
Jul 29 19:17:11.632935 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632719    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-host-run-k8s-cni-cncf-io\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.632935 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632744    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-systemd-units\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.632935 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632771    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-etc-openvswitch\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.632935 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632801    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/40e0ec6a-ab60-4892-8adf-45cda1a71526-etc-sysconfig\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.633801 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632830    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40e0ec6a-ab60-4892-8adf-45cda1a71526-host\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.633801 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632855    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f737cfa7-db52-409a-b5c3-f704b9170fa4-hosts-file\") pod \"node-resolver-vft97\" (UID: \"f737cfa7-db52-409a-b5c3-f704b9170fa4\") " pod="openshift-dns/node-resolver-vft97"
Jul 29 19:17:11.633801 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632903    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krfsd\" (UniqueName: \"kubernetes.io/projected/f737cfa7-db52-409a-b5c3-f704b9170fa4-kube-api-access-krfsd\") pod \"node-resolver-vft97\" (UID: \"f737cfa7-db52-409a-b5c3-f704b9170fa4\") " pod="openshift-dns/node-resolver-vft97"
Jul 29 19:17:11.633801 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632936    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-run-ovn\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.633801 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632961    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-node-log\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.633801 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.632988    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-log-socket\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.633801 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.633014    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/40e0ec6a-ab60-4892-8adf-45cda1a71526-etc-sysctl-conf\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.633801 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.633055    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/40e0ec6a-ab60-4892-8adf-45cda1a71526-etc-systemd\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.633801 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.633084    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/40e0ec6a-ab60-4892-8adf-45cda1a71526-etc-tuned\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.633801 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.633111    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dvvj\" (UniqueName: \"kubernetes.io/projected/c4bcc5ff-6023-4514-b5d1-61657a854bba-kube-api-access-5dvvj\") pod \"aws-ebs-csi-driver-node-jntx6\" (UID: \"c4bcc5ff-6023-4514-b5d1-61657a854bba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jntx6"
Jul 29 19:17:11.633801 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.633137    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmjv7\" (UniqueName: \"kubernetes.io/projected/067b4107-d3a3-4fb2-9c30-90fab2dab639-kube-api-access-hmjv7\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.633801 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.633175    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f4b5376d-e96c-465e-adfb-bcd266b89eea-iptables-alerter-script\") pod \"iptables-alerter-h7mz2\" (UID: \"f4b5376d-e96c-465e-adfb-bcd266b89eea\") " pod="openshift-network-operator/iptables-alerter-h7mz2"
Jul 29 19:17:11.633801 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.633230    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f4b5376d-e96c-465e-adfb-bcd266b89eea-host-slash\") pod \"iptables-alerter-h7mz2\" (UID: \"f4b5376d-e96c-465e-adfb-bcd266b89eea\") " pod="openshift-network-operator/iptables-alerter-h7mz2"
Jul 29 19:17:11.633801 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.633258    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d160081b-f780-4c60-85f9-04b9a6d1c3d9-serviceca\") pod \"node-ca-frvmp\" (UID: \"d160081b-f780-4c60-85f9-04b9a6d1c3d9\") " pod="openshift-image-registry/node-ca-frvmp"
Jul 29 19:17:11.633801 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.633285    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-system-cni-dir\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.633801 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.633313    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-multus-socket-dir-parent\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.633801 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.633343    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-host-run-netns\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.634697 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.633372    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/08c90b03-c2a2-478b-8d59-bb71b15e96f8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v67cp\" (UID: \"08c90b03-c2a2-478b-8d59-bb71b15e96f8\") " pod="openshift-multus/multus-additional-cni-plugins-v67cp"
Jul 29 19:17:11.634697 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.633401    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/08c90b03-c2a2-478b-8d59-bb71b15e96f8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v67cp\" (UID: \"08c90b03-c2a2-478b-8d59-bb71b15e96f8\") " pod="openshift-multus/multus-additional-cni-plugins-v67cp"
Jul 29 19:17:11.634697 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.633431    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgx88\" (UniqueName: \"kubernetes.io/projected/08c90b03-c2a2-478b-8d59-bb71b15e96f8-kube-api-access-lgx88\") pod \"multus-additional-cni-plugins-v67cp\" (UID: \"08c90b03-c2a2-478b-8d59-bb71b15e96f8\") " pod="openshift-multus/multus-additional-cni-plugins-v67cp"
Jul 29 19:17:11.634697 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.633459    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f737cfa7-db52-409a-b5c3-f704b9170fa4-tmp-dir\") pod \"node-resolver-vft97\" (UID: \"f737cfa7-db52-409a-b5c3-f704b9170fa4\") " pod="openshift-dns/node-resolver-vft97"
Jul 29 19:17:11.634697 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.633487    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c4bcc5ff-6023-4514-b5d1-61657a854bba-socket-dir\") pod \"aws-ebs-csi-driver-node-jntx6\" (UID: \"c4bcc5ff-6023-4514-b5d1-61657a854bba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jntx6"
Jul 29 19:17:11.634697 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.633515    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hchnn\" (UniqueName: \"kubernetes.io/projected/a4f202a3-5dc7-416f-8793-e857e6086174-kube-api-access-hchnn\") pod \"network-metrics-daemon-cw75m\" (UID: \"a4f202a3-5dc7-416f-8793-e857e6086174\") " pod="openshift-multus/network-metrics-daemon-cw75m"
Jul 29 19:17:11.634697 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.633542    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/40e0ec6a-ab60-4892-8adf-45cda1a71526-run\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.634697 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.633570    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p76tm\" (UniqueName: \"kubernetes.io/projected/40e0ec6a-ab60-4892-8adf-45cda1a71526-kube-api-access-p76tm\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.634697 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.633598    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c4bcc5ff-6023-4514-b5d1-61657a854bba-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jntx6\" (UID: \"c4bcc5ff-6023-4514-b5d1-61657a854bba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jntx6"
Jul 29 19:17:11.634697 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.633626    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4f202a3-5dc7-416f-8793-e857e6086174-metrics-certs\") pod \"network-metrics-daemon-cw75m\" (UID: \"a4f202a3-5dc7-416f-8793-e857e6086174\") " pod="openshift-multus/network-metrics-daemon-cw75m"
Jul 29 19:17:11.634697 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.633655    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-run-systemd\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.634697 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.633685    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-host-cni-netd\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.634697 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.633710    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/639a786b-08de-453b-8733-28f54e9a59ff-agent-certs\") pod \"konnectivity-agent-6jmnn\" (UID: \"639a786b-08de-453b-8733-28f54e9a59ff\") " pod="kube-system/konnectivity-agent-6jmnn"
Jul 29 19:17:11.634697 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.633737    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-multus-cni-dir\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.634697 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.633763    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-os-release\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.634697 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.633791    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-host-var-lib-cni-bin\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.634697 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.633819    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c4bcc5ff-6023-4514-b5d1-61657a854bba-device-dir\") pod \"aws-ebs-csi-driver-node-jntx6\" (UID: \"c4bcc5ff-6023-4514-b5d1-61657a854bba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jntx6"
Jul 29 19:17:11.635515 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.633853    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c4bcc5ff-6023-4514-b5d1-61657a854bba-etc-selinux\") pod \"aws-ebs-csi-driver-node-jntx6\" (UID: \"c4bcc5ff-6023-4514-b5d1-61657a854bba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jntx6"
Jul 29 19:17:11.635515 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.633882    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/067b4107-d3a3-4fb2-9c30-90fab2dab639-ovn-node-metrics-cert\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.635515 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.633908    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-host-kubelet\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.635515 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.633936    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/40e0ec6a-ab60-4892-8adf-45cda1a71526-etc-sysctl-d\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.635515 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.633970    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-host-slash\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.635515 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.633999    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-host-run-netns\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.635515 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.634027    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/40e0ec6a-ab60-4892-8adf-45cda1a71526-etc-modprobe-d\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.635515 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.634071    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d160081b-f780-4c60-85f9-04b9a6d1c3d9-host\") pod \"node-ca-frvmp\" (UID: \"d160081b-f780-4c60-85f9-04b9a6d1c3d9\") " pod="openshift-image-registry/node-ca-frvmp"
Jul 29 19:17:11.635515 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.634109    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/93168299-ac4f-4fdd-8ac3-b69edb385d71-kubelet-config\") pod \"global-pull-secret-syncer-v5x8q\" (UID: \"93168299-ac4f-4fdd-8ac3-b69edb385d71\") " pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:17:11.635515 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.634156    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08c90b03-c2a2-478b-8d59-bb71b15e96f8-cni-binary-copy\") pod \"multus-additional-cni-plugins-v67cp\" (UID: \"08c90b03-c2a2-478b-8d59-bb71b15e96f8\") " pod="openshift-multus/multus-additional-cni-plugins-v67cp"
Jul 29 19:17:11.635515 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.634185    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-run-openvswitch\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.635515 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.634188    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08c90b03-c2a2-478b-8d59-bb71b15e96f8-system-cni-dir\") pod \"multus-additional-cni-plugins-v67cp\" (UID: \"08c90b03-c2a2-478b-8d59-bb71b15e96f8\") " pod="openshift-multus/multus-additional-cni-plugins-v67cp"
Jul 29 19:17:11.635515 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.634215    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/067b4107-d3a3-4fb2-9c30-90fab2dab639-ovnkube-config\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.635515 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.634244    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-multus-daemon-config\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.635515 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.634256    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08c90b03-c2a2-478b-8d59-bb71b15e96f8-cnibin\") pod \"multus-additional-cni-plugins-v67cp\" (UID: \"08c90b03-c2a2-478b-8d59-bb71b15e96f8\") " pod="openshift-multus/multus-additional-cni-plugins-v67cp"
Jul 29 19:17:11.635515 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.634287    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93168299-ac4f-4fdd-8ac3-b69edb385d71-original-pull-secret\") pod \"global-pull-secret-syncer-v5x8q\" (UID: \"93168299-ac4f-4fdd-8ac3-b69edb385d71\") " pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:17:11.635515 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.634317    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-host-var-lib-kubelet\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.636436 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.634361    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/08c90b03-c2a2-478b-8d59-bb71b15e96f8-os-release\") pod \"multus-additional-cni-plugins-v67cp\" (UID: \"08c90b03-c2a2-478b-8d59-bb71b15e96f8\") " pod="openshift-multus/multus-additional-cni-plugins-v67cp"
Jul 29 19:17:11.636436 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.634411    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-host-run-ovn-kubernetes\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.636436 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.634445    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-node-log\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.636436 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.634454    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-host-cni-bin\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.636436 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.634499    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-log-socket\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.636436 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.634503    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.636436 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.634633    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/40e0ec6a-ab60-4892-8adf-45cda1a71526-etc-sysctl-conf\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.636436 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.634691    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/40e0ec6a-ab60-4892-8adf-45cda1a71526-etc-systemd\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.636436 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.635009    2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 29 19:17:11.636436 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.635256    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/067b4107-d3a3-4fb2-9c30-90fab2dab639-ovnkube-script-lib\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.636436 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.635346    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c4bcc5ff-6023-4514-b5d1-61657a854bba-registration-dir\") pod \"aws-ebs-csi-driver-node-jntx6\" (UID: \"c4bcc5ff-6023-4514-b5d1-61657a854bba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jntx6"
Jul 29 19:17:11.636436 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.635391    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-var-lib-openvswitch\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.636436 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.635399    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-host-cni-netd\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.637138 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.636705    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/08c90b03-c2a2-478b-8d59-bb71b15e96f8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v67cp\" (UID: \"08c90b03-c2a2-478b-8d59-bb71b15e96f8\") " pod="openshift-multus/multus-additional-cni-plugins-v67cp"
Jul 29 19:17:11.637275 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.637249    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/08c90b03-c2a2-478b-8d59-bb71b15e96f8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v67cp\" (UID: \"08c90b03-c2a2-478b-8d59-bb71b15e96f8\") " pod="openshift-multus/multus-additional-cni-plugins-v67cp"
Jul 29 19:17:11.637792 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.637768    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f737cfa7-db52-409a-b5c3-f704b9170fa4-tmp-dir\") pod \"node-resolver-vft97\" (UID: \"f737cfa7-db52-409a-b5c3-f704b9170fa4\") " pod="openshift-dns/node-resolver-vft97"
Jul 29 19:17:11.637929 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.637908    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c4bcc5ff-6023-4514-b5d1-61657a854bba-socket-dir\") pod \"aws-ebs-csi-driver-node-jntx6\" (UID: \"c4bcc5ff-6023-4514-b5d1-61657a854bba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jntx6"
Jul 29 19:17:11.639903 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.638230    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/40e0ec6a-ab60-4892-8adf-45cda1a71526-run\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.639903 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.638329    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c4bcc5ff-6023-4514-b5d1-61657a854bba-sys-fs\") pod \"aws-ebs-csi-driver-node-jntx6\" (UID: \"c4bcc5ff-6023-4514-b5d1-61657a854bba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jntx6"
Jul 29 19:17:11.639903 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.638479    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08c90b03-c2a2-478b-8d59-bb71b15e96f8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v67cp\" (UID: \"08c90b03-c2a2-478b-8d59-bb71b15e96f8\") " pod="openshift-multus/multus-additional-cni-plugins-v67cp"
Jul 29 19:17:11.639903 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.638539    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40e0ec6a-ab60-4892-8adf-45cda1a71526-sys\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.639903 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:11.638578    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:17:11.639903 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.638593    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40e0ec6a-ab60-4892-8adf-45cda1a71526-var-lib-kubelet\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.639903 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:11.638659    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4f202a3-5dc7-416f-8793-e857e6086174-metrics-certs podName:a4f202a3-5dc7-416f-8793-e857e6086174 nodeName:}" failed. No retries permitted until 2026-07-29 19:17:12.138636375 +0000 UTC m=+2.966135436 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a4f202a3-5dc7-416f-8793-e857e6086174-metrics-certs") pod "network-metrics-daemon-cw75m" (UID: "a4f202a3-5dc7-416f-8793-e857e6086174") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:17:11.639903 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.638877    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-systemd-units\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.639903 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.638942    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40e0ec6a-ab60-4892-8adf-45cda1a71526-etc-kubernetes\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.639903 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.639057    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/067b4107-d3a3-4fb2-9c30-90fab2dab639-env-overrides\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.639903 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.639086    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/40e0ec6a-ab60-4892-8adf-45cda1a71526-lib-modules\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.639903 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.639136    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-etc-openvswitch\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.639903 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.639171    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f737cfa7-db52-409a-b5c3-f704b9170fa4-hosts-file\") pod \"node-resolver-vft97\" (UID: \"f737cfa7-db52-409a-b5c3-f704b9170fa4\") " pod="openshift-dns/node-resolver-vft97"
Jul 29 19:17:11.639903 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.639180    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c4bcc5ff-6023-4514-b5d1-61657a854bba-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jntx6\" (UID: \"c4bcc5ff-6023-4514-b5d1-61657a854bba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jntx6"
Jul 29 19:17:11.639903 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.639235    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40e0ec6a-ab60-4892-8adf-45cda1a71526-host\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.639903 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.639282    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-host-run-netns\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.639903 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.639327    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/40e0ec6a-ab60-4892-8adf-45cda1a71526-etc-sysconfig\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.639903 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.639500    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40e0ec6a-ab60-4892-8adf-45cda1a71526-tmp\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.640905 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.639581    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-run-openvswitch\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.640905 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.639595    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c4bcc5ff-6023-4514-b5d1-61657a854bba-device-dir\") pod \"aws-ebs-csi-driver-node-jntx6\" (UID: \"c4bcc5ff-6023-4514-b5d1-61657a854bba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jntx6"
Jul 29 19:17:11.640905 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.639604    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-run-systemd\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.640905 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.639649    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-host-slash\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.640905 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.639701    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c4bcc5ff-6023-4514-b5d1-61657a854bba-etc-selinux\") pod \"aws-ebs-csi-driver-node-jntx6\" (UID: \"c4bcc5ff-6023-4514-b5d1-61657a854bba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jntx6"
Jul 29 19:17:11.640905 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.639704    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-host-kubelet\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.640905 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.639757    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/067b4107-d3a3-4fb2-9c30-90fab2dab639-run-ovn\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.640905 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.639769    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08c90b03-c2a2-478b-8d59-bb71b15e96f8-cni-binary-copy\") pod \"multus-additional-cni-plugins-v67cp\" (UID: \"08c90b03-c2a2-478b-8d59-bb71b15e96f8\") " pod="openshift-multus/multus-additional-cni-plugins-v67cp"
Jul 29 19:17:11.640905 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.639797    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/40e0ec6a-ab60-4892-8adf-45cda1a71526-etc-sysctl-d\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.640905 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.639874    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/40e0ec6a-ab60-4892-8adf-45cda1a71526-etc-modprobe-d\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.640905 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.639926    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/067b4107-d3a3-4fb2-9c30-90fab2dab639-ovnkube-config\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.640905 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.640545    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/40e0ec6a-ab60-4892-8adf-45cda1a71526-etc-tuned\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.642402 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.642374    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/067b4107-d3a3-4fb2-9c30-90fab2dab639-ovn-node-metrics-cert\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.644851 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.644799    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dvvj\" (UniqueName: \"kubernetes.io/projected/c4bcc5ff-6023-4514-b5d1-61657a854bba-kube-api-access-5dvvj\") pod \"aws-ebs-csi-driver-node-jntx6\" (UID: \"c4bcc5ff-6023-4514-b5d1-61657a854bba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jntx6"
Jul 29 19:17:11.645908 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.645885    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmjv7\" (UniqueName: \"kubernetes.io/projected/067b4107-d3a3-4fb2-9c30-90fab2dab639-kube-api-access-hmjv7\") pod \"ovnkube-node-q6hd8\" (UID: \"067b4107-d3a3-4fb2-9c30-90fab2dab639\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.646549 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.646456    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgx88\" (UniqueName: \"kubernetes.io/projected/08c90b03-c2a2-478b-8d59-bb71b15e96f8-kube-api-access-lgx88\") pod \"multus-additional-cni-plugins-v67cp\" (UID: \"08c90b03-c2a2-478b-8d59-bb71b15e96f8\") " pod="openshift-multus/multus-additional-cni-plugins-v67cp"
Jul 29 19:17:11.647705 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.647664    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p76tm\" (UniqueName: \"kubernetes.io/projected/40e0ec6a-ab60-4892-8adf-45cda1a71526-kube-api-access-p76tm\") pod \"tuned-lcx4k\" (UID: \"40e0ec6a-ab60-4892-8adf-45cda1a71526\") " pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.658154 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.658036    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krfsd\" (UniqueName: \"kubernetes.io/projected/f737cfa7-db52-409a-b5c3-f704b9170fa4-kube-api-access-krfsd\") pod \"node-resolver-vft97\" (UID: \"f737cfa7-db52-409a-b5c3-f704b9170fa4\") " pod="openshift-dns/node-resolver-vft97"
Jul 29 19:17:11.658619 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.658577    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hchnn\" (UniqueName: \"kubernetes.io/projected/a4f202a3-5dc7-416f-8793-e857e6086174-kube-api-access-hchnn\") pod \"network-metrics-daemon-cw75m\" (UID: \"a4f202a3-5dc7-416f-8793-e857e6086174\") " pod="openshift-multus/network-metrics-daemon-cw75m"
Jul 29 19:17:11.679282 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.679224    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-173.ec2.internal" event={"ID":"6dd50be5d2585bd24e7a6b31885c789e","Type":"ContainerStarted","Data":"cfcfae0a16fedfb20d06e0e89a64b5490a6a43e81809a3a88f42060d695bd43d"}
Jul 29 19:17:11.680710 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.680682    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-173.ec2.internal" event={"ID":"70a21ba21338b6ba758e27d2cca1c0ab","Type":"ContainerStarted","Data":"8457dd5f206f47a28e172a2c11f418272c6b61450173c146f9842e2baf53ee03"}
Jul 29 19:17:11.735250 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.735213    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-host-var-lib-kubelet\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.735429 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.735264    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/639a786b-08de-453b-8733-28f54e9a59ff-konnectivity-ca\") pod \"konnectivity-agent-6jmnn\" (UID: \"639a786b-08de-453b-8733-28f54e9a59ff\") " pod="kube-system/konnectivity-agent-6jmnn"
Jul 29 19:17:11.735429 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.735293    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-cnibin\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.735429 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.735322    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-host-var-lib-cni-multus\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.735429 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.735346    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-hostroot\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.735429 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.735370    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-multus-conf-dir\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.735429 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.735377    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-cnibin\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.735429 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.735401    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rx2m\" (UniqueName: \"kubernetes.io/projected/2677a483-66e1-42fd-94f5-3903b7af3690-kube-api-access-6rx2m\") pod \"network-check-target-tqpxs\" (UID: \"2677a483-66e1-42fd-94f5-3903b7af3690\") " pod="openshift-network-diagnostics/network-check-target-tqpxs"
Jul 29 19:17:11.735429 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.735342    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-host-var-lib-kubelet\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.735429 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.735413    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-hostroot\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.735429 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.735427    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-cni-binary-copy\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.735955 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.735466    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-multus-conf-dir\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.735955 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.735385    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-host-var-lib-cni-multus\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.735955 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.735516    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-host-run-multus-certs\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.735955 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.735553    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-etc-kubernetes\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.735955 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.735588    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdstk\" (UniqueName: \"kubernetes.io/projected/f4b5376d-e96c-465e-adfb-bcd266b89eea-kube-api-access-xdstk\") pod \"iptables-alerter-h7mz2\" (UID: \"f4b5376d-e96c-465e-adfb-bcd266b89eea\") " pod="openshift-network-operator/iptables-alerter-h7mz2"
Jul 29 19:17:11.735955 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.735614    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22vkc\" (UniqueName: \"kubernetes.io/projected/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-kube-api-access-22vkc\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.735955 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.735650    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/93168299-ac4f-4fdd-8ac3-b69edb385d71-dbus\") pod \"global-pull-secret-syncer-v5x8q\" (UID: \"93168299-ac4f-4fdd-8ac3-b69edb385d71\") " pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:17:11.735955 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.735672    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-host-run-multus-certs\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.735955 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.735677    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmnv7\" (UniqueName: \"kubernetes.io/projected/d160081b-f780-4c60-85f9-04b9a6d1c3d9-kube-api-access-cmnv7\") pod \"node-ca-frvmp\" (UID: \"d160081b-f780-4c60-85f9-04b9a6d1c3d9\") " pod="openshift-image-registry/node-ca-frvmp"
Jul 29 19:17:11.735955 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.735721    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-host-run-k8s-cni-cncf-io\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.735955 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.735772    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f4b5376d-e96c-465e-adfb-bcd266b89eea-iptables-alerter-script\") pod \"iptables-alerter-h7mz2\" (UID: \"f4b5376d-e96c-465e-adfb-bcd266b89eea\") " pod="openshift-network-operator/iptables-alerter-h7mz2"
Jul 29 19:17:11.735955 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.735794    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f4b5376d-e96c-465e-adfb-bcd266b89eea-host-slash\") pod \"iptables-alerter-h7mz2\" (UID: \"f4b5376d-e96c-465e-adfb-bcd266b89eea\") " pod="openshift-network-operator/iptables-alerter-h7mz2"
Jul 29 19:17:11.735955 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.735809    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d160081b-f780-4c60-85f9-04b9a6d1c3d9-serviceca\") pod \"node-ca-frvmp\" (UID: \"d160081b-f780-4c60-85f9-04b9a6d1c3d9\") " pod="openshift-image-registry/node-ca-frvmp"
Jul 29 19:17:11.735955 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.735825    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-system-cni-dir\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.735955 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.735841    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-multus-socket-dir-parent\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.735955 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.735856    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-host-run-netns\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.735955 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.735868    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/639a786b-08de-453b-8733-28f54e9a59ff-konnectivity-ca\") pod \"konnectivity-agent-6jmnn\" (UID: \"639a786b-08de-453b-8733-28f54e9a59ff\") " pod="kube-system/konnectivity-agent-6jmnn"
Jul 29 19:17:11.735955 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.735888    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/639a786b-08de-453b-8733-28f54e9a59ff-agent-certs\") pod \"konnectivity-agent-6jmnn\" (UID: \"639a786b-08de-453b-8733-28f54e9a59ff\") " pod="kube-system/konnectivity-agent-6jmnn"
Jul 29 19:17:11.736759 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.735933    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-etc-kubernetes\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.736759 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.735958    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-host-run-k8s-cni-cncf-io\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.736759 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.736030    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-cni-binary-copy\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.736759 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.736081    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/93168299-ac4f-4fdd-8ac3-b69edb385d71-dbus\") pod \"global-pull-secret-syncer-v5x8q\" (UID: \"93168299-ac4f-4fdd-8ac3-b69edb385d71\") " pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:17:11.736759 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.736124    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-multus-socket-dir-parent\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.736759 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.736124    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-system-cni-dir\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.736759 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.736182    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-host-run-netns\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.736759 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.736319    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-multus-cni-dir\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.736759 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.736359    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-os-release\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.736759 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.736364    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f4b5376d-e96c-465e-adfb-bcd266b89eea-host-slash\") pod \"iptables-alerter-h7mz2\" (UID: \"f4b5376d-e96c-465e-adfb-bcd266b89eea\") " pod="openshift-network-operator/iptables-alerter-h7mz2"
Jul 29 19:17:11.736759 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.736389    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-host-var-lib-cni-bin\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.736759 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.736423    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d160081b-f780-4c60-85f9-04b9a6d1c3d9-host\") pod \"node-ca-frvmp\" (UID: \"d160081b-f780-4c60-85f9-04b9a6d1c3d9\") " pod="openshift-image-registry/node-ca-frvmp"
Jul 29 19:17:11.736759 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.736444    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-multus-cni-dir\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.736759 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.736448    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/93168299-ac4f-4fdd-8ac3-b69edb385d71-kubelet-config\") pod \"global-pull-secret-syncer-v5x8q\" (UID: \"93168299-ac4f-4fdd-8ac3-b69edb385d71\") " pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:17:11.736759 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.736480    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-multus-daemon-config\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.736759 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.736482    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-host-var-lib-cni-bin\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.736759 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.736508    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93168299-ac4f-4fdd-8ac3-b69edb385d71-original-pull-secret\") pod \"global-pull-secret-syncer-v5x8q\" (UID: \"93168299-ac4f-4fdd-8ac3-b69edb385d71\") " pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:17:11.736759 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.736522    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d160081b-f780-4c60-85f9-04b9a6d1c3d9-host\") pod \"node-ca-frvmp\" (UID: \"d160081b-f780-4c60-85f9-04b9a6d1c3d9\") " pod="openshift-image-registry/node-ca-frvmp"
Jul 29 19:17:11.737623 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.736537    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d160081b-f780-4c60-85f9-04b9a6d1c3d9-serviceca\") pod \"node-ca-frvmp\" (UID: \"d160081b-f780-4c60-85f9-04b9a6d1c3d9\") " pod="openshift-image-registry/node-ca-frvmp"
Jul 29 19:17:11.737623 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.736506    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-os-release\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.737623 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.736550    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/93168299-ac4f-4fdd-8ac3-b69edb385d71-kubelet-config\") pod \"global-pull-secret-syncer-v5x8q\" (UID: \"93168299-ac4f-4fdd-8ac3-b69edb385d71\") " pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:17:11.737623 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:11.736675    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 19:17:11.737623 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:11.736736    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93168299-ac4f-4fdd-8ac3-b69edb385d71-original-pull-secret podName:93168299-ac4f-4fdd-8ac3-b69edb385d71 nodeName:}" failed. No retries permitted until 2026-07-29 19:17:12.236717597 +0000 UTC m=+3.064216646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/93168299-ac4f-4fdd-8ac3-b69edb385d71-original-pull-secret") pod "global-pull-secret-syncer-v5x8q" (UID: "93168299-ac4f-4fdd-8ac3-b69edb385d71") : object "kube-system"/"original-pull-secret" not registered
Jul 29 19:17:11.737623 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.736941    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f4b5376d-e96c-465e-adfb-bcd266b89eea-iptables-alerter-script\") pod \"iptables-alerter-h7mz2\" (UID: \"f4b5376d-e96c-465e-adfb-bcd266b89eea\") " pod="openshift-network-operator/iptables-alerter-h7mz2"
Jul 29 19:17:11.737623 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.737007    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-multus-daemon-config\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.739281 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.739220    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/639a786b-08de-453b-8733-28f54e9a59ff-agent-certs\") pod \"konnectivity-agent-6jmnn\" (UID: \"639a786b-08de-453b-8733-28f54e9a59ff\") " pod="kube-system/konnectivity-agent-6jmnn"
Jul 29 19:17:11.748111 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:11.748055    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 19:17:11.748111 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:11.748082    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 19:17:11.748273 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:11.748127    2557 projected.go:196] Error preparing data for projected volume kube-api-access-6rx2m for pod openshift-network-diagnostics/network-check-target-tqpxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:17:11.748273 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:11.748189    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2677a483-66e1-42fd-94f5-3903b7af3690-kube-api-access-6rx2m podName:2677a483-66e1-42fd-94f5-3903b7af3690 nodeName:}" failed. No retries permitted until 2026-07-29 19:17:12.248170914 +0000 UTC m=+3.075669967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6rx2m" (UniqueName: "kubernetes.io/projected/2677a483-66e1-42fd-94f5-3903b7af3690-kube-api-access-6rx2m") pod "network-check-target-tqpxs" (UID: "2677a483-66e1-42fd-94f5-3903b7af3690") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:17:11.753646 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.753622    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdstk\" (UniqueName: \"kubernetes.io/projected/f4b5376d-e96c-465e-adfb-bcd266b89eea-kube-api-access-xdstk\") pod \"iptables-alerter-h7mz2\" (UID: \"f4b5376d-e96c-465e-adfb-bcd266b89eea\") " pod="openshift-network-operator/iptables-alerter-h7mz2"
Jul 29 19:17:11.754293 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.754268    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmnv7\" (UniqueName: \"kubernetes.io/projected/d160081b-f780-4c60-85f9-04b9a6d1c3d9-kube-api-access-cmnv7\") pod \"node-ca-frvmp\" (UID: \"d160081b-f780-4c60-85f9-04b9a6d1c3d9\") " pod="openshift-image-registry/node-ca-frvmp"
Jul 29 19:17:11.754418 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.754399    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22vkc\" (UniqueName: \"kubernetes.io/projected/4c408e45-c274-45fa-9aab-8c6a4c6fdd4e-kube-api-access-22vkc\") pod \"multus-c528d\" (UID: \"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e\") " pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.831567 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.831532    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jntx6"
Jul 29 19:17:11.840970 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.840941    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v67cp"
Jul 29 19:17:11.849473 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.849381    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:11.856098 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.856078    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lcx4k"
Jul 29 19:17:11.864099 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.864078    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vft97"
Jul 29 19:17:11.871908 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.871879    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-frvmp"
Jul 29 19:17:11.883383 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.883359    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c528d"
Jul 29 19:17:11.891093 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.891067    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6jmnn"
Jul 29 19:17:11.898878 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:11.898857    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h7mz2"
Jul 29 19:17:12.139569 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:12.139530    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4f202a3-5dc7-416f-8793-e857e6086174-metrics-certs\") pod \"network-metrics-daemon-cw75m\" (UID: \"a4f202a3-5dc7-416f-8793-e857e6086174\") " pod="openshift-multus/network-metrics-daemon-cw75m"
Jul 29 19:17:12.139768 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:12.139665    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:17:12.139768 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:12.139729    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4f202a3-5dc7-416f-8793-e857e6086174-metrics-certs podName:a4f202a3-5dc7-416f-8793-e857e6086174 nodeName:}" failed. No retries permitted until 2026-07-29 19:17:13.139710207 +0000 UTC m=+3.967209267 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a4f202a3-5dc7-416f-8793-e857e6086174-metrics-certs") pod "network-metrics-daemon-cw75m" (UID: "a4f202a3-5dc7-416f-8793-e857e6086174") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:17:12.239929 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:12.239883    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93168299-ac4f-4fdd-8ac3-b69edb385d71-original-pull-secret\") pod \"global-pull-secret-syncer-v5x8q\" (UID: \"93168299-ac4f-4fdd-8ac3-b69edb385d71\") " pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:17:12.240113 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:12.240063    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 19:17:12.240170 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:12.240132    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93168299-ac4f-4fdd-8ac3-b69edb385d71-original-pull-secret podName:93168299-ac4f-4fdd-8ac3-b69edb385d71 nodeName:}" failed. No retries permitted until 2026-07-29 19:17:13.240116202 +0000 UTC m=+4.067615240 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/93168299-ac4f-4fdd-8ac3-b69edb385d71-original-pull-secret") pod "global-pull-secret-syncer-v5x8q" (UID: "93168299-ac4f-4fdd-8ac3-b69edb385d71") : object "kube-system"/"original-pull-secret" not registered
Jul 29 19:17:12.340790 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:12.340751    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rx2m\" (UniqueName: \"kubernetes.io/projected/2677a483-66e1-42fd-94f5-3903b7af3690-kube-api-access-6rx2m\") pod \"network-check-target-tqpxs\" (UID: \"2677a483-66e1-42fd-94f5-3903b7af3690\") " pod="openshift-network-diagnostics/network-check-target-tqpxs"
Jul 29 19:17:12.340968 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:12.340943    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 19:17:12.341068 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:12.340994    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 19:17:12.341068 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:12.341011    2557 projected.go:196] Error preparing data for projected volume kube-api-access-6rx2m for pod openshift-network-diagnostics/network-check-target-tqpxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:17:12.341157 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:12.341073    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2677a483-66e1-42fd-94f5-3903b7af3690-kube-api-access-6rx2m podName:2677a483-66e1-42fd-94f5-3903b7af3690 nodeName:}" failed. No retries permitted until 2026-07-29 19:17:13.341058047 +0000 UTC m=+4.168557110 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6rx2m" (UniqueName: "kubernetes.io/projected/2677a483-66e1-42fd-94f5-3903b7af3690-kube-api-access-6rx2m") pod "network-check-target-tqpxs" (UID: "2677a483-66e1-42fd-94f5-3903b7af3690") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:17:12.565745 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:12.565658    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 19:12:11 +0000 UTC" deadline="2028-02-13 23:15:18.514545041 +0000 UTC"
Jul 29 19:17:12.565745 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:12.565696    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13539h58m5.948853037s"
Jul 29 19:17:12.672843 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:12.672811    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cw75m"
Jul 29 19:17:12.673035 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:12.672956    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cw75m" podUID="a4f202a3-5dc7-416f-8793-e857e6086174"
Jul 29 19:17:12.871866 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:12.871834    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067b4107_d3a3_4fb2_9c30_90fab2dab639.slice/crio-588e478c1c61eb8b3d7a36f9d996884c35d6d7ae1eea0a38cbaa6a7004df8034 WatchSource:0}: Error finding container 588e478c1c61eb8b3d7a36f9d996884c35d6d7ae1eea0a38cbaa6a7004df8034: Status 404 returned error can't find the container with id 588e478c1c61eb8b3d7a36f9d996884c35d6d7ae1eea0a38cbaa6a7004df8034
Jul 29 19:17:12.873207 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:12.873179    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c408e45_c274_45fa_9aab_8c6a4c6fdd4e.slice/crio-2e9ed8cb0e2968206940cfb87bdab5024af1e3c86526919735ea5e4fdaf559bc WatchSource:0}: Error finding container 2e9ed8cb0e2968206940cfb87bdab5024af1e3c86526919735ea5e4fdaf559bc: Status 404 returned error can't find the container with id 2e9ed8cb0e2968206940cfb87bdab5024af1e3c86526919735ea5e4fdaf559bc
Jul 29 19:17:12.875086 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:12.874908    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c90b03_c2a2_478b_8d59_bb71b15e96f8.slice/crio-1348f9bf4f63195d79c59cfaa896c517801fb1dadfe2a2a67ebc6624a46c3daf WatchSource:0}: Error finding container 1348f9bf4f63195d79c59cfaa896c517801fb1dadfe2a2a67ebc6624a46c3daf: Status 404 returned error can't find the container with id 1348f9bf4f63195d79c59cfaa896c517801fb1dadfe2a2a67ebc6624a46c3daf
Jul 29 19:17:12.877605 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:12.877506    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b5376d_e96c_465e_adfb_bcd266b89eea.slice/crio-2edd1ee99240563a82064b895dcee6ece932c85cc3411e96b82911d48d6637e4 WatchSource:0}: Error finding container 2edd1ee99240563a82064b895dcee6ece932c85cc3411e96b82911d48d6637e4: Status 404 returned error can't find the container with id 2edd1ee99240563a82064b895dcee6ece932c85cc3411e96b82911d48d6637e4
Jul 29 19:17:12.878609 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:12.878588    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e0ec6a_ab60_4892_8adf_45cda1a71526.slice/crio-8c23103bd9d813debeffc60b503a9dc4765bd81dc7c76b63f7838e8abef51fcb WatchSource:0}: Error finding container 8c23103bd9d813debeffc60b503a9dc4765bd81dc7c76b63f7838e8abef51fcb: Status 404 returned error can't find the container with id 8c23103bd9d813debeffc60b503a9dc4765bd81dc7c76b63f7838e8abef51fcb
Jul 29 19:17:13.147076 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:13.146869    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4f202a3-5dc7-416f-8793-e857e6086174-metrics-certs\") pod \"network-metrics-daemon-cw75m\" (UID: \"a4f202a3-5dc7-416f-8793-e857e6086174\") " pod="openshift-multus/network-metrics-daemon-cw75m"
Jul 29 19:17:13.147204 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:13.147043    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:17:13.147204 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:13.147186    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4f202a3-5dc7-416f-8793-e857e6086174-metrics-certs podName:a4f202a3-5dc7-416f-8793-e857e6086174 nodeName:}" failed. No retries permitted until 2026-07-29 19:17:15.147171345 +0000 UTC m=+5.974670382 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a4f202a3-5dc7-416f-8793-e857e6086174-metrics-certs") pod "network-metrics-daemon-cw75m" (UID: "a4f202a3-5dc7-416f-8793-e857e6086174") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:17:13.247660 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:13.247621    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93168299-ac4f-4fdd-8ac3-b69edb385d71-original-pull-secret\") pod \"global-pull-secret-syncer-v5x8q\" (UID: \"93168299-ac4f-4fdd-8ac3-b69edb385d71\") " pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:17:13.247794 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:13.247757    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 19:17:13.247852 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:13.247812    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93168299-ac4f-4fdd-8ac3-b69edb385d71-original-pull-secret podName:93168299-ac4f-4fdd-8ac3-b69edb385d71 nodeName:}" failed. No retries permitted until 2026-07-29 19:17:15.247797366 +0000 UTC m=+6.075296402 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/93168299-ac4f-4fdd-8ac3-b69edb385d71-original-pull-secret") pod "global-pull-secret-syncer-v5x8q" (UID: "93168299-ac4f-4fdd-8ac3-b69edb385d71") : object "kube-system"/"original-pull-secret" not registered
Jul 29 19:17:13.348370 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:13.348337    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rx2m\" (UniqueName: \"kubernetes.io/projected/2677a483-66e1-42fd-94f5-3903b7af3690-kube-api-access-6rx2m\") pod \"network-check-target-tqpxs\" (UID: \"2677a483-66e1-42fd-94f5-3903b7af3690\") " pod="openshift-network-diagnostics/network-check-target-tqpxs"
Jul 29 19:17:13.348572 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:13.348489    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 19:17:13.348572 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:13.348513    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 19:17:13.348572 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:13.348527    2557 projected.go:196] Error preparing data for projected volume kube-api-access-6rx2m for pod openshift-network-diagnostics/network-check-target-tqpxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:17:13.348763 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:13.348595    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2677a483-66e1-42fd-94f5-3903b7af3690-kube-api-access-6rx2m podName:2677a483-66e1-42fd-94f5-3903b7af3690 nodeName:}" failed. No retries permitted until 2026-07-29 19:17:15.348576095 +0000 UTC m=+6.176075156 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6rx2m" (UniqueName: "kubernetes.io/projected/2677a483-66e1-42fd-94f5-3903b7af3690-kube-api-access-6rx2m") pod "network-check-target-tqpxs" (UID: "2677a483-66e1-42fd-94f5-3903b7af3690") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:17:13.566923 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:13.566801    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 19:12:11 +0000 UTC" deadline="2028-04-12 21:10:45.756856626 +0000 UTC"
Jul 29 19:17:13.566923 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:13.566870    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14953h53m32.18999091s"
Jul 29 19:17:13.676466 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:13.675797    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqpxs"
Jul 29 19:17:13.676466 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:13.675920    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqpxs" podUID="2677a483-66e1-42fd-94f5-3903b7af3690"
Jul 29 19:17:13.676466 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:13.676319    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:17:13.676466 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:13.676411    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5x8q" podUID="93168299-ac4f-4fdd-8ac3-b69edb385d71"
Jul 29 19:17:13.693116 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:13.693085    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c528d" event={"ID":"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e","Type":"ContainerStarted","Data":"2e9ed8cb0e2968206940cfb87bdab5024af1e3c86526919735ea5e4fdaf559bc"}
Jul 29 19:17:13.705398 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:13.705364    2557 generic.go:356] "Generic (PLEG): container finished" podID="6dd50be5d2585bd24e7a6b31885c789e" containerID="88b68f360ef5939ee467909d8b915822312849a057f89be038402646c42c36f1" exitCode=0
Jul 29 19:17:13.705547 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:13.705451    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-173.ec2.internal" event={"ID":"6dd50be5d2585bd24e7a6b31885c789e","Type":"ContainerDied","Data":"88b68f360ef5939ee467909d8b915822312849a057f89be038402646c42c36f1"}
Jul 29 19:17:13.722268 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:13.722238    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-frvmp" event={"ID":"d160081b-f780-4c60-85f9-04b9a6d1c3d9","Type":"ContainerStarted","Data":"4d2a51478f3764c5a1bbf5d7048944cff3d2abf23398b4a2d48f65e6d3714e6f"}
Jul 29 19:17:13.729999 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:13.729955    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h7mz2" event={"ID":"f4b5376d-e96c-465e-adfb-bcd266b89eea","Type":"ContainerStarted","Data":"2edd1ee99240563a82064b895dcee6ece932c85cc3411e96b82911d48d6637e4"}
Jul 29 19:17:13.765729 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:13.765694    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v67cp" event={"ID":"08c90b03-c2a2-478b-8d59-bb71b15e96f8","Type":"ContainerStarted","Data":"1348f9bf4f63195d79c59cfaa896c517801fb1dadfe2a2a67ebc6624a46c3daf"}
Jul 29 19:17:13.769235 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:13.769209    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8" event={"ID":"067b4107-d3a3-4fb2-9c30-90fab2dab639","Type":"ContainerStarted","Data":"588e478c1c61eb8b3d7a36f9d996884c35d6d7ae1eea0a38cbaa6a7004df8034"}
Jul 29 19:17:13.796600 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:13.795871    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-173.ec2.internal" event={"ID":"70a21ba21338b6ba758e27d2cca1c0ab","Type":"ContainerStarted","Data":"687b7bd4bad41acaf012fd772a4b590bdaa01de635c9d3fa339ed74bc322bd02"}
Jul 29 19:17:13.799313 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:13.799241    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6jmnn" event={"ID":"639a786b-08de-453b-8733-28f54e9a59ff","Type":"ContainerStarted","Data":"6e3426d037eb25dfd815e702e171cdf7a7be9a61b8dff94030aecefb4473b081"}
Jul 29 19:17:13.807084 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:13.806958    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vft97" event={"ID":"f737cfa7-db52-409a-b5c3-f704b9170fa4","Type":"ContainerStarted","Data":"0d2009a08e10824fbf9b4c5b0f3b45c72afa478c454a5f0405fbae9a488390a1"}
Jul 29 19:17:13.812001 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:13.811913    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-173.ec2.internal" podStartSLOduration=2.811895095 podStartE2EDuration="2.811895095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:17:13.811517245 +0000 UTC m=+4.639016304" watchObservedRunningTime="2026-07-29 19:17:13.811895095 +0000 UTC m=+4.639394155"
Jul 29 19:17:13.831929 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:13.831806    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jntx6" event={"ID":"c4bcc5ff-6023-4514-b5d1-61657a854bba","Type":"ContainerStarted","Data":"b5bad343d61f72b1befd983f137a697dbe16a214794baa822d5045fce09300fd"}
Jul 29 19:17:13.861983 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:13.861567    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lcx4k" event={"ID":"40e0ec6a-ab60-4892-8adf-45cda1a71526","Type":"ContainerStarted","Data":"8c23103bd9d813debeffc60b503a9dc4765bd81dc7c76b63f7838e8abef51fcb"}
Jul 29 19:17:14.672687 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:14.672653    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cw75m"
Jul 29 19:17:14.673186 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:14.672790    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cw75m" podUID="a4f202a3-5dc7-416f-8793-e857e6086174"
Jul 29 19:17:14.896315 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:14.896183    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-173.ec2.internal" event={"ID":"6dd50be5d2585bd24e7a6b31885c789e","Type":"ContainerStarted","Data":"1e32220290be1c0bf379737c21c47d9026482f3ff053c82b9a1df9b56c602ea3"}
Jul 29 19:17:15.165005 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:15.164974    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4f202a3-5dc7-416f-8793-e857e6086174-metrics-certs\") pod \"network-metrics-daemon-cw75m\" (UID: \"a4f202a3-5dc7-416f-8793-e857e6086174\") " pod="openshift-multus/network-metrics-daemon-cw75m"
Jul 29 19:17:15.165193 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:15.165109    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:17:15.165193 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:15.165173    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4f202a3-5dc7-416f-8793-e857e6086174-metrics-certs podName:a4f202a3-5dc7-416f-8793-e857e6086174 nodeName:}" failed. No retries permitted until 2026-07-29 19:17:19.165154034 +0000 UTC m=+9.992653078 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a4f202a3-5dc7-416f-8793-e857e6086174-metrics-certs") pod "network-metrics-daemon-cw75m" (UID: "a4f202a3-5dc7-416f-8793-e857e6086174") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:17:15.265993 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:15.265372    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93168299-ac4f-4fdd-8ac3-b69edb385d71-original-pull-secret\") pod \"global-pull-secret-syncer-v5x8q\" (UID: \"93168299-ac4f-4fdd-8ac3-b69edb385d71\") " pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:17:15.265993 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:15.265509    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 19:17:15.265993 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:15.265572    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93168299-ac4f-4fdd-8ac3-b69edb385d71-original-pull-secret podName:93168299-ac4f-4fdd-8ac3-b69edb385d71 nodeName:}" failed. No retries permitted until 2026-07-29 19:17:19.26555495 +0000 UTC m=+10.093053999 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/93168299-ac4f-4fdd-8ac3-b69edb385d71-original-pull-secret") pod "global-pull-secret-syncer-v5x8q" (UID: "93168299-ac4f-4fdd-8ac3-b69edb385d71") : object "kube-system"/"original-pull-secret" not registered
Jul 29 19:17:15.366158 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:15.366120    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rx2m\" (UniqueName: \"kubernetes.io/projected/2677a483-66e1-42fd-94f5-3903b7af3690-kube-api-access-6rx2m\") pod \"network-check-target-tqpxs\" (UID: \"2677a483-66e1-42fd-94f5-3903b7af3690\") " pod="openshift-network-diagnostics/network-check-target-tqpxs"
Jul 29 19:17:15.366393 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:15.366374    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 19:17:15.366466 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:15.366401    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 19:17:15.366466 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:15.366414    2557 projected.go:196] Error preparing data for projected volume kube-api-access-6rx2m for pod openshift-network-diagnostics/network-check-target-tqpxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:17:15.366567 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:15.366473    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2677a483-66e1-42fd-94f5-3903b7af3690-kube-api-access-6rx2m podName:2677a483-66e1-42fd-94f5-3903b7af3690 nodeName:}" failed. No retries permitted until 2026-07-29 19:17:19.366454703 +0000 UTC m=+10.193953755 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6rx2m" (UniqueName: "kubernetes.io/projected/2677a483-66e1-42fd-94f5-3903b7af3690-kube-api-access-6rx2m") pod "network-check-target-tqpxs" (UID: "2677a483-66e1-42fd-94f5-3903b7af3690") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:17:15.674126 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:15.674096    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:17:15.674588 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:15.674228    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5x8q" podUID="93168299-ac4f-4fdd-8ac3-b69edb385d71"
Jul 29 19:17:15.674873 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:15.674851    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqpxs"
Jul 29 19:17:15.674991 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:15.674957    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqpxs" podUID="2677a483-66e1-42fd-94f5-3903b7af3690"
Jul 29 19:17:16.672922 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:16.672889    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cw75m"
Jul 29 19:17:16.673132 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:16.673059    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cw75m" podUID="a4f202a3-5dc7-416f-8793-e857e6086174"
Jul 29 19:17:17.673346 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:17.673309    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqpxs"
Jul 29 19:17:17.673346 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:17.673335    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:17:17.673841 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:17.673450    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqpxs" podUID="2677a483-66e1-42fd-94f5-3903b7af3690"
Jul 29 19:17:17.673841 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:17.673577    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5x8q" podUID="93168299-ac4f-4fdd-8ac3-b69edb385d71"
Jul 29 19:17:18.673027 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:18.672555    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cw75m"
Jul 29 19:17:18.673526 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:18.673467    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cw75m" podUID="a4f202a3-5dc7-416f-8793-e857e6086174"
Jul 29 19:17:19.199061 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:19.198491    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4f202a3-5dc7-416f-8793-e857e6086174-metrics-certs\") pod \"network-metrics-daemon-cw75m\" (UID: \"a4f202a3-5dc7-416f-8793-e857e6086174\") " pod="openshift-multus/network-metrics-daemon-cw75m"
Jul 29 19:17:19.199061 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:19.198658    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:17:19.199061 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:19.198724    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4f202a3-5dc7-416f-8793-e857e6086174-metrics-certs podName:a4f202a3-5dc7-416f-8793-e857e6086174 nodeName:}" failed. No retries permitted until 2026-07-29 19:17:27.198705147 +0000 UTC m=+18.026204204 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a4f202a3-5dc7-416f-8793-e857e6086174-metrics-certs") pod "network-metrics-daemon-cw75m" (UID: "a4f202a3-5dc7-416f-8793-e857e6086174") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:17:19.299544 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:19.299511    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93168299-ac4f-4fdd-8ac3-b69edb385d71-original-pull-secret\") pod \"global-pull-secret-syncer-v5x8q\" (UID: \"93168299-ac4f-4fdd-8ac3-b69edb385d71\") " pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:17:19.299724 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:19.299670    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 19:17:19.299792 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:19.299730    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93168299-ac4f-4fdd-8ac3-b69edb385d71-original-pull-secret podName:93168299-ac4f-4fdd-8ac3-b69edb385d71 nodeName:}" failed. No retries permitted until 2026-07-29 19:17:27.299712349 +0000 UTC m=+18.127211397 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/93168299-ac4f-4fdd-8ac3-b69edb385d71-original-pull-secret") pod "global-pull-secret-syncer-v5x8q" (UID: "93168299-ac4f-4fdd-8ac3-b69edb385d71") : object "kube-system"/"original-pull-secret" not registered
Jul 29 19:17:19.401170 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:19.400524    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rx2m\" (UniqueName: \"kubernetes.io/projected/2677a483-66e1-42fd-94f5-3903b7af3690-kube-api-access-6rx2m\") pod \"network-check-target-tqpxs\" (UID: \"2677a483-66e1-42fd-94f5-3903b7af3690\") " pod="openshift-network-diagnostics/network-check-target-tqpxs"
Jul 29 19:17:19.401170 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:19.400731    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 19:17:19.401170 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:19.400752    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 19:17:19.401170 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:19.400765    2557 projected.go:196] Error preparing data for projected volume kube-api-access-6rx2m for pod openshift-network-diagnostics/network-check-target-tqpxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:17:19.401170 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:19.400822    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2677a483-66e1-42fd-94f5-3903b7af3690-kube-api-access-6rx2m podName:2677a483-66e1-42fd-94f5-3903b7af3690 nodeName:}" failed. No retries permitted until 2026-07-29 19:17:27.400803233 +0000 UTC m=+18.228302277 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6rx2m" (UniqueName: "kubernetes.io/projected/2677a483-66e1-42fd-94f5-3903b7af3690-kube-api-access-6rx2m") pod "network-check-target-tqpxs" (UID: "2677a483-66e1-42fd-94f5-3903b7af3690") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:17:19.677640 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:19.677211    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:17:19.677640 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:19.677327    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5x8q" podUID="93168299-ac4f-4fdd-8ac3-b69edb385d71"
Jul 29 19:17:19.677640 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:19.677415    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqpxs"
Jul 29 19:17:19.677640 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:19.677515    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqpxs" podUID="2677a483-66e1-42fd-94f5-3903b7af3690"
Jul 29 19:17:20.673282 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:20.673151    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cw75m"
Jul 29 19:17:20.673452 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:20.673300    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cw75m" podUID="a4f202a3-5dc7-416f-8793-e857e6086174"
Jul 29 19:17:21.676673 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:21.676604    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:17:21.676673 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:21.676631    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqpxs"
Jul 29 19:17:21.677244 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:21.676728    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5x8q" podUID="93168299-ac4f-4fdd-8ac3-b69edb385d71"
Jul 29 19:17:21.677244 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:21.677138    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqpxs" podUID="2677a483-66e1-42fd-94f5-3903b7af3690"
Jul 29 19:17:22.672493 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:22.672458    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cw75m"
Jul 29 19:17:22.672665 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:22.672593    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cw75m" podUID="a4f202a3-5dc7-416f-8793-e857e6086174"
Jul 29 19:17:23.672783 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:23.672754    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqpxs"
Jul 29 19:17:23.673247 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:23.672861    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqpxs" podUID="2677a483-66e1-42fd-94f5-3903b7af3690"
Jul 29 19:17:23.673247 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:23.672923    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:17:23.673247 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:23.673053    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5x8q" podUID="93168299-ac4f-4fdd-8ac3-b69edb385d71"
Jul 29 19:17:24.672644 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:24.672607    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cw75m"
Jul 29 19:17:24.672843 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:24.672745    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cw75m" podUID="a4f202a3-5dc7-416f-8793-e857e6086174"
Jul 29 19:17:25.672807 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:25.672771    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqpxs"
Jul 29 19:17:25.673000 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:25.672904    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqpxs" podUID="2677a483-66e1-42fd-94f5-3903b7af3690"
Jul 29 19:17:25.673000 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:25.672977    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:17:25.673392 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:25.673105    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5x8q" podUID="93168299-ac4f-4fdd-8ac3-b69edb385d71"
Jul 29 19:17:26.673288 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:26.673252    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cw75m"
Jul 29 19:17:26.673761 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:26.673379    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cw75m" podUID="a4f202a3-5dc7-416f-8793-e857e6086174"
Jul 29 19:17:27.260678 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:27.260640    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4f202a3-5dc7-416f-8793-e857e6086174-metrics-certs\") pod \"network-metrics-daemon-cw75m\" (UID: \"a4f202a3-5dc7-416f-8793-e857e6086174\") " pod="openshift-multus/network-metrics-daemon-cw75m"
Jul 29 19:17:27.260818 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:27.260785    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:17:27.260865 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:27.260841    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4f202a3-5dc7-416f-8793-e857e6086174-metrics-certs podName:a4f202a3-5dc7-416f-8793-e857e6086174 nodeName:}" failed. No retries permitted until 2026-07-29 19:17:43.260826671 +0000 UTC m=+34.088325707 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a4f202a3-5dc7-416f-8793-e857e6086174-metrics-certs") pod "network-metrics-daemon-cw75m" (UID: "a4f202a3-5dc7-416f-8793-e857e6086174") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:17:27.361258 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:27.361223    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93168299-ac4f-4fdd-8ac3-b69edb385d71-original-pull-secret\") pod \"global-pull-secret-syncer-v5x8q\" (UID: \"93168299-ac4f-4fdd-8ac3-b69edb385d71\") " pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:17:27.361388 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:27.361365    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 19:17:27.361449 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:27.361438    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93168299-ac4f-4fdd-8ac3-b69edb385d71-original-pull-secret podName:93168299-ac4f-4fdd-8ac3-b69edb385d71 nodeName:}" failed. No retries permitted until 2026-07-29 19:17:43.361420661 +0000 UTC m=+34.188919700 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/93168299-ac4f-4fdd-8ac3-b69edb385d71-original-pull-secret") pod "global-pull-secret-syncer-v5x8q" (UID: "93168299-ac4f-4fdd-8ac3-b69edb385d71") : object "kube-system"/"original-pull-secret" not registered
Jul 29 19:17:27.461587 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:27.461553    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rx2m\" (UniqueName: \"kubernetes.io/projected/2677a483-66e1-42fd-94f5-3903b7af3690-kube-api-access-6rx2m\") pod \"network-check-target-tqpxs\" (UID: \"2677a483-66e1-42fd-94f5-3903b7af3690\") " pod="openshift-network-diagnostics/network-check-target-tqpxs"
Jul 29 19:17:27.461749 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:27.461694    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 19:17:27.461749 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:27.461711    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 19:17:27.461749 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:27.461720    2557 projected.go:196] Error preparing data for projected volume kube-api-access-6rx2m for pod openshift-network-diagnostics/network-check-target-tqpxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:17:27.461886 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:27.461772    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2677a483-66e1-42fd-94f5-3903b7af3690-kube-api-access-6rx2m podName:2677a483-66e1-42fd-94f5-3903b7af3690 nodeName:}" failed. No retries permitted until 2026-07-29 19:17:43.461758059 +0000 UTC m=+34.289257099 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6rx2m" (UniqueName: "kubernetes.io/projected/2677a483-66e1-42fd-94f5-3903b7af3690-kube-api-access-6rx2m") pod "network-check-target-tqpxs" (UID: "2677a483-66e1-42fd-94f5-3903b7af3690") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:17:27.672812 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:27.672773    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqpxs"
Jul 29 19:17:27.673006 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:27.672899    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqpxs" podUID="2677a483-66e1-42fd-94f5-3903b7af3690"
Jul 29 19:17:27.673006 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:27.672977    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:17:27.673201 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:27.673098    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5x8q" podUID="93168299-ac4f-4fdd-8ac3-b69edb385d71"
Jul 29 19:17:28.672514 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:28.672482    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cw75m"
Jul 29 19:17:28.672945 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:28.672616    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cw75m" podUID="a4f202a3-5dc7-416f-8793-e857e6086174"
Jul 29 19:17:29.674186 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:29.674150    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqpxs"
Jul 29 19:17:29.674565 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:29.674245    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:17:29.674565 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:29.674283    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqpxs" podUID="2677a483-66e1-42fd-94f5-3903b7af3690"
Jul 29 19:17:29.674565 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:29.674321    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5x8q" podUID="93168299-ac4f-4fdd-8ac3-b69edb385d71"
Jul 29 19:17:30.672365 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:30.672333    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cw75m"
Jul 29 19:17:30.672537 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:30.672440    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cw75m" podUID="a4f202a3-5dc7-416f-8793-e857e6086174"
Jul 29 19:17:31.673157 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:31.672998    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqpxs"
Jul 29 19:17:31.673953 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:31.673229    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqpxs" podUID="2677a483-66e1-42fd-94f5-3903b7af3690"
Jul 29 19:17:31.673953 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:31.673283    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:17:31.673953 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:31.673326    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5x8q" podUID="93168299-ac4f-4fdd-8ac3-b69edb385d71"
Jul 29 19:17:31.930469 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:31.930436    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-frvmp" event={"ID":"d160081b-f780-4c60-85f9-04b9a6d1c3d9","Type":"ContainerStarted","Data":"4b7a6b4a1fdaccc72365b6dfb271df2986ddde5e64ba4164ae4fe71d32820220"}
Jul 29 19:17:31.931890 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:31.931867    2557 generic.go:356] "Generic (PLEG): container finished" podID="08c90b03-c2a2-478b-8d59-bb71b15e96f8" containerID="7048a531f421f1271e6c4f1c494ac7c49731e2f53fc6237dd2c5d576bbdea72c" exitCode=0
Jul 29 19:17:31.932006 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:31.931919    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v67cp" event={"ID":"08c90b03-c2a2-478b-8d59-bb71b15e96f8","Type":"ContainerDied","Data":"7048a531f421f1271e6c4f1c494ac7c49731e2f53fc6237dd2c5d576bbdea72c"}
Jul 29 19:17:31.934449 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:31.934424    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8" event={"ID":"067b4107-d3a3-4fb2-9c30-90fab2dab639","Type":"ContainerStarted","Data":"1cfc8bd1b6e1fcffae8b016b7e2383b26881c7ee104c32c484f854bae4f2688a"}
Jul 29 19:17:31.934537 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:31.934504    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8" event={"ID":"067b4107-d3a3-4fb2-9c30-90fab2dab639","Type":"ContainerStarted","Data":"f3a374b4447cdc1b331e8bd4ef9dff28a402f1525aa77e8c7d7f4ef2be7f377d"}
Jul 29 19:17:31.934594 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:31.934547    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8" event={"ID":"067b4107-d3a3-4fb2-9c30-90fab2dab639","Type":"ContainerStarted","Data":"b0f4959b84edcddbbf535c22efe52369554d4c1c1aad39e87e5336f973462615"}
Jul 29 19:17:31.934594 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:31.934584    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8" event={"ID":"067b4107-d3a3-4fb2-9c30-90fab2dab639","Type":"ContainerStarted","Data":"9a165b9d25fdfb469d0c06358219650e1e81fbb65a290b74828c5a7d0764dd9b"}
Jul 29 19:17:31.936020 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:31.935998    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6jmnn" event={"ID":"639a786b-08de-453b-8733-28f54e9a59ff","Type":"ContainerStarted","Data":"240368d360880fd291360d59f926ad880a91d04c5d7647cf5ea85a3c6a0c4ef6"}
Jul 29 19:17:31.937434 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:31.937395    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vft97" event={"ID":"f737cfa7-db52-409a-b5c3-f704b9170fa4","Type":"ContainerStarted","Data":"04dae1d674260b2213ac211cb175d0e021ef96c00e4585d0b9e16aea70e989e8"}
Jul 29 19:17:31.938947 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:31.938924    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jntx6" event={"ID":"c4bcc5ff-6023-4514-b5d1-61657a854bba","Type":"ContainerStarted","Data":"330890789e338e50f4e6e16dd352d1089c5a73882b91e6dcba823bb5763e27e0"}
Jul 29 19:17:31.942568 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:31.942549    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lcx4k" event={"ID":"40e0ec6a-ab60-4892-8adf-45cda1a71526","Type":"ContainerStarted","Data":"c5cc2c9fc13a43f3098a82110b3caf78a1f007b7aeb5df1cf1adf8ddc5ae1387"}
Jul 29 19:17:31.943887 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:31.943868    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c528d" event={"ID":"4c408e45-c274-45fa-9aab-8c6a4c6fdd4e","Type":"ContainerStarted","Data":"b5287607b9255f15024ec211b589c4b46e34d3289da9c31e948ff4a5fa866bae"}
Jul 29 19:17:31.956773 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:31.956737    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-173.ec2.internal" podStartSLOduration=20.956726662 podStartE2EDuration="20.956726662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:17:14.92537425 +0000 UTC m=+5.752873308" watchObservedRunningTime="2026-07-29 19:17:31.956726662 +0000 UTC m=+22.784225720"
Jul 29 19:17:31.985818 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:31.985720    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-frvmp" podStartSLOduration=3.473594836 podStartE2EDuration="21.985707713s" totalImagesPullingTime="18.512112877s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:17:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:17:31.95699662 +0000 UTC m=+22.784495678" watchObservedRunningTime="2026-07-29 19:17:31.985707713 +0000 UTC m=+22.813206774"
Jul 29 19:17:32.006691 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:32.006656    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-c528d" podStartSLOduration=3.4415659180000002 podStartE2EDuration="22.006644512s" totalImagesPullingTime="18.565078594s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:17:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:17:32.006237445 +0000 UTC m=+22.833736503" watchObservedRunningTime="2026-07-29 19:17:32.006644512 +0000 UTC m=+22.834143579"
Jul 29 19:17:32.043092 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:32.043058    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6jmnn" podStartSLOduration=7.718404029 podStartE2EDuration="22.043045698s" totalImagesPullingTime="14.324641669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:17:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:17:32.024203276 +0000 UTC m=+22.851702354" watchObservedRunningTime="2026-07-29 19:17:32.043045698 +0000 UTC m=+22.870544756"
Jul 29 19:17:32.043396 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:32.043375    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lcx4k" podStartSLOduration=3.530193564 podStartE2EDuration="22.043369034s" totalImagesPullingTime="18.51317547s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:17:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:17:32.042874775 +0000 UTC m=+22.870373830" watchObservedRunningTime="2026-07-29 19:17:32.043369034 +0000 UTC m=+22.870868091"
Jul 29 19:17:32.059746 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:32.059709    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vft97" podStartSLOduration=3.566849309 podStartE2EDuration="22.059699003s" totalImagesPullingTime="18.492849694s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:17:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:17:32.059374719 +0000 UTC m=+22.886873778" watchObservedRunningTime="2026-07-29 19:17:32.059699003 +0000 UTC m=+22.887198060"
Jul 29 19:17:32.672747 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:32.672721    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cw75m"
Jul 29 19:17:32.672866 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:32.672841    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cw75m" podUID="a4f202a3-5dc7-416f-8793-e857e6086174"
Jul 29 19:17:32.947978 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:32.947920    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h7mz2" event={"ID":"f4b5376d-e96c-465e-adfb-bcd266b89eea","Type":"ContainerStarted","Data":"c58166bca6c8092169b7c54dfe7d64005323482d86de8172a3e790fb458fa0da"}
Jul 29 19:17:32.951583 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:32.951548    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8" event={"ID":"067b4107-d3a3-4fb2-9c30-90fab2dab639","Type":"ContainerStarted","Data":"34f381a1c2c5ab92f2794eb8f7822eaa94e81a5a1c6b83d202032e7ef21bb460"}
Jul 29 19:17:32.951684 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:32.951641    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8" event={"ID":"067b4107-d3a3-4fb2-9c30-90fab2dab639","Type":"ContainerStarted","Data":"6fc5350e730f192c804556e93de76df8a0aaaeb29eb13b15f7cf78abc24e9921"}
Jul 29 19:17:32.965165 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:32.965117    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-h7mz2" podStartSLOduration=4.451738097 podStartE2EDuration="22.965101878s" totalImagesPullingTime="18.513363781s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:17:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:17:32.964472934 +0000 UTC m=+23.791971992" watchObservedRunningTime="2026-07-29 19:17:32.965101878 +0000 UTC m=+23.792600935"
Jul 29 19:17:33.010053 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:33.010022    2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 29 19:17:33.597817 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:33.597704    2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T19:17:33.010046891Z","UUID":"3830c8fb-889d-421e-84d2-cf81d6463ef0","Handler":null,"Name":"","Endpoint":""}
Jul 29 19:17:33.599353 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:33.599319    2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 29 19:17:33.599447 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:33.599366    2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 29 19:17:33.673199 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:33.673163    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqpxs"
Jul 29 19:17:33.673354 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:33.673291    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqpxs" podUID="2677a483-66e1-42fd-94f5-3903b7af3690"
Jul 29 19:17:33.673354 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:33.673341    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:17:33.673458 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:33.673439    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5x8q" podUID="93168299-ac4f-4fdd-8ac3-b69edb385d71"
Jul 29 19:17:33.955980 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:33.955924    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jntx6" event={"ID":"c4bcc5ff-6023-4514-b5d1-61657a854bba","Type":"ContainerStarted","Data":"81bdb8c56c88d40a1c403a41479f778a089296314cd681f34dc31c105b613e9a"}
Jul 29 19:17:34.258829 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:34.258801    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6jmnn"
Jul 29 19:17:34.259362 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:34.259342    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6jmnn"
Jul 29 19:17:34.270432 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:34.270381    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6jmnn"
Jul 29 19:17:34.270845 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:34.270829    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6jmnn"
Jul 29 19:17:34.673186 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:34.673142    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cw75m"
Jul 29 19:17:34.673386 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:34.673294    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cw75m" podUID="a4f202a3-5dc7-416f-8793-e857e6086174"
Jul 29 19:17:34.962619 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:34.962519    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8" event={"ID":"067b4107-d3a3-4fb2-9c30-90fab2dab639","Type":"ContainerStarted","Data":"a4896819f17bcd9348b907f05af0def9e9345b50789f04b506e578cd33697da0"}
Jul 29 19:17:34.965511 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:34.965473    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jntx6" event={"ID":"c4bcc5ff-6023-4514-b5d1-61657a854bba","Type":"ContainerStarted","Data":"51c35094209bfbf541c7b9cb8cec6342163e0638c468251802957dc8fa421354"}
Jul 29 19:17:34.984926 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:34.984883    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jntx6" podStartSLOduration=4.964805359 podStartE2EDuration="25.984867761s" totalImagesPullingTime="21.020062402s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:17:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:17:34.983785818 +0000 UTC m=+25.811284877" watchObservedRunningTime="2026-07-29 19:17:34.984867761 +0000 UTC m=+25.812366819"
Jul 29 19:17:35.673262 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:35.673063    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:17:35.673522 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:35.673118    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqpxs"
Jul 29 19:17:35.673522 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:35.673369    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5x8q" podUID="93168299-ac4f-4fdd-8ac3-b69edb385d71"
Jul 29 19:17:35.673522 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:35.673457    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqpxs" podUID="2677a483-66e1-42fd-94f5-3903b7af3690"
Jul 29 19:17:36.673587 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:36.673369    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cw75m"
Jul 29 19:17:36.674265 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:36.673630    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cw75m" podUID="a4f202a3-5dc7-416f-8793-e857e6086174"
Jul 29 19:17:36.971153 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:36.971036    2557 generic.go:356] "Generic (PLEG): container finished" podID="08c90b03-c2a2-478b-8d59-bb71b15e96f8" containerID="60e49efc318d6e9f915f5f89dda1db4cf1cce3fbb0200b42840cafcf0391cc6e" exitCode=0
Jul 29 19:17:36.971153 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:36.971141    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v67cp" event={"ID":"08c90b03-c2a2-478b-8d59-bb71b15e96f8","Type":"ContainerDied","Data":"60e49efc318d6e9f915f5f89dda1db4cf1cce3fbb0200b42840cafcf0391cc6e"}
Jul 29 19:17:36.974757 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:36.974731    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8" event={"ID":"067b4107-d3a3-4fb2-9c30-90fab2dab639","Type":"ContainerStarted","Data":"d7a26398cae190f3abf253689655cc7253c65b95adb53649cc23ffce6485f454"}
Jul 29 19:17:36.975146 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:36.975124    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:36.990813 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:36.990794    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:37.027770 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:37.027721    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8" podStartSLOduration=8.441028737 podStartE2EDuration="27.027708385s" totalImagesPullingTime="18.586679648s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:17:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:17:37.026306985 +0000 UTC m=+27.853806043" watchObservedRunningTime="2026-07-29 19:17:37.027708385 +0000 UTC m=+27.855207443"
Jul 29 19:17:37.673047 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:37.673008    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqpxs"
Jul 29 19:17:37.673233 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:37.673017    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:17:37.673233 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:37.673139    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqpxs" podUID="2677a483-66e1-42fd-94f5-3903b7af3690"
Jul 29 19:17:37.673233 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:37.673213    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5x8q" podUID="93168299-ac4f-4fdd-8ac3-b69edb385d71"
Jul 29 19:17:37.977804 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:37.977526    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:37.977804 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:37.977617    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:37.993553 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:37.993526    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:17:38.450074 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:38.450049    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tqpxs"]
Jul 29 19:17:38.450232 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:38.450152    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqpxs"
Jul 29 19:17:38.450292 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:38.450241    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqpxs" podUID="2677a483-66e1-42fd-94f5-3903b7af3690"
Jul 29 19:17:38.453245 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:38.453221    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cw75m"]
Jul 29 19:17:38.453366 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:38.453345    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cw75m"
Jul 29 19:17:38.453468 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:38.453445    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cw75m" podUID="a4f202a3-5dc7-416f-8793-e857e6086174"
Jul 29 19:17:38.453869 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:38.453844    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v5x8q"]
Jul 29 19:17:38.453985 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:38.453929    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:17:38.454060 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:38.454038    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5x8q" podUID="93168299-ac4f-4fdd-8ac3-b69edb385d71"
Jul 29 19:17:38.983957 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:38.983926    2557 generic.go:356] "Generic (PLEG): container finished" podID="08c90b03-c2a2-478b-8d59-bb71b15e96f8" containerID="11edbaa6e11a3de67112699d210781ec00a702970d5b70a8565b803b778e16b4" exitCode=0
Jul 29 19:17:38.984330 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:38.984006    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v67cp" event={"ID":"08c90b03-c2a2-478b-8d59-bb71b15e96f8","Type":"ContainerDied","Data":"11edbaa6e11a3de67112699d210781ec00a702970d5b70a8565b803b778e16b4"}
Jul 29 19:17:39.674344 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:39.674113    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cw75m"
Jul 29 19:17:39.674496 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:39.674462    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cw75m" podUID="a4f202a3-5dc7-416f-8793-e857e6086174"
Jul 29 19:17:40.673140 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:40.673105    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:17:40.673615 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:40.673105    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqpxs"
Jul 29 19:17:40.673615 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:40.673216    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5x8q" podUID="93168299-ac4f-4fdd-8ac3-b69edb385d71"
Jul 29 19:17:40.673615 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:40.673276    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqpxs" podUID="2677a483-66e1-42fd-94f5-3903b7af3690"
Jul 29 19:17:40.992054 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:40.992022    2557 generic.go:356] "Generic (PLEG): container finished" podID="08c90b03-c2a2-478b-8d59-bb71b15e96f8" containerID="25244b885660c491cebb50f6d9924eb69ab75be82e66770d40de368e0c988d13" exitCode=0
Jul 29 19:17:40.992218 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:40.992084    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v67cp" event={"ID":"08c90b03-c2a2-478b-8d59-bb71b15e96f8","Type":"ContainerDied","Data":"25244b885660c491cebb50f6d9924eb69ab75be82e66770d40de368e0c988d13"}
Jul 29 19:17:41.673246 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:41.673208    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cw75m"
Jul 29 19:17:41.673870 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:41.673350    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cw75m" podUID="a4f202a3-5dc7-416f-8793-e857e6086174"
Jul 29 19:17:42.672705 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:42.672678    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:17:42.672857 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:42.672682    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqpxs"
Jul 29 19:17:42.672857 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:42.672797    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5x8q" podUID="93168299-ac4f-4fdd-8ac3-b69edb385d71"
Jul 29 19:17:42.672983 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:42.672854    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqpxs" podUID="2677a483-66e1-42fd-94f5-3903b7af3690"
Jul 29 19:17:43.282927 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:43.282891    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4f202a3-5dc7-416f-8793-e857e6086174-metrics-certs\") pod \"network-metrics-daemon-cw75m\" (UID: \"a4f202a3-5dc7-416f-8793-e857e6086174\") " pod="openshift-multus/network-metrics-daemon-cw75m"
Jul 29 19:17:43.283341 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:43.283068    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:17:43.283341 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:43.283147    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4f202a3-5dc7-416f-8793-e857e6086174-metrics-certs podName:a4f202a3-5dc7-416f-8793-e857e6086174 nodeName:}" failed. No retries permitted until 2026-07-29 19:18:15.283127868 +0000 UTC m=+66.110626908 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a4f202a3-5dc7-416f-8793-e857e6086174-metrics-certs") pod "network-metrics-daemon-cw75m" (UID: "a4f202a3-5dc7-416f-8793-e857e6086174") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:17:43.383613 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:43.383527    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93168299-ac4f-4fdd-8ac3-b69edb385d71-original-pull-secret\") pod \"global-pull-secret-syncer-v5x8q\" (UID: \"93168299-ac4f-4fdd-8ac3-b69edb385d71\") " pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:17:43.383755 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:43.383694    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 19:17:43.383803 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:43.383775    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93168299-ac4f-4fdd-8ac3-b69edb385d71-original-pull-secret podName:93168299-ac4f-4fdd-8ac3-b69edb385d71 nodeName:}" failed. No retries permitted until 2026-07-29 19:18:15.383754285 +0000 UTC m=+66.211253337 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/93168299-ac4f-4fdd-8ac3-b69edb385d71-original-pull-secret") pod "global-pull-secret-syncer-v5x8q" (UID: "93168299-ac4f-4fdd-8ac3-b69edb385d71") : object "kube-system"/"original-pull-secret" not registered
Jul 29 19:17:43.484249 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:43.484210    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rx2m\" (UniqueName: \"kubernetes.io/projected/2677a483-66e1-42fd-94f5-3903b7af3690-kube-api-access-6rx2m\") pod \"network-check-target-tqpxs\" (UID: \"2677a483-66e1-42fd-94f5-3903b7af3690\") " pod="openshift-network-diagnostics/network-check-target-tqpxs"
Jul 29 19:17:43.484429 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:43.484396    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 19:17:43.484429 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:43.484413    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 19:17:43.484429 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:43.484422    2557 projected.go:196] Error preparing data for projected volume kube-api-access-6rx2m for pod openshift-network-diagnostics/network-check-target-tqpxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:17:43.484570 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:43.484473    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2677a483-66e1-42fd-94f5-3903b7af3690-kube-api-access-6rx2m podName:2677a483-66e1-42fd-94f5-3903b7af3690 nodeName:}" failed. No retries permitted until 2026-07-29 19:18:15.484461132 +0000 UTC m=+66.311960170 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6rx2m" (UniqueName: "kubernetes.io/projected/2677a483-66e1-42fd-94f5-3903b7af3690-kube-api-access-6rx2m") pod "network-check-target-tqpxs" (UID: "2677a483-66e1-42fd-94f5-3903b7af3690") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:17:43.673422 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:43.673389    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cw75m"
Jul 29 19:17:43.673602 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:43.673522    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cw75m" podUID="a4f202a3-5dc7-416f-8793-e857e6086174"
Jul 29 19:17:44.672804 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:44.672773    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:17:44.673423 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:44.672773    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqpxs"
Jul 29 19:17:44.673423 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:44.672894    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5x8q" podUID="93168299-ac4f-4fdd-8ac3-b69edb385d71"
Jul 29 19:17:44.673423 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:44.672951    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqpxs" podUID="2677a483-66e1-42fd-94f5-3903b7af3690"
Jul 29 19:17:44.976192 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:44.976121    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-173.ec2.internal" event="NodeReady"
Jul 29 19:17:44.976342 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:44.976275    2557 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 29 19:17:45.028515 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.028466    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kq4zj"]
Jul 29 19:17:45.032208 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.032181    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kq4zj"
Jul 29 19:17:45.033840 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.033815    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-q724q"]
Jul 29 19:17:45.035692 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.035659    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-q724q"
Jul 29 19:17:45.036227 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.036203    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 29 19:17:45.036333 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.036248    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-74g8d\""
Jul 29 19:17:45.036333 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.036290    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 29 19:17:45.042434 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.042336    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 29 19:17:45.042693 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.042570    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2gz5l\""
Jul 29 19:17:45.042693 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.042636    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 29 19:17:45.043036 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.043019    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 29 19:17:45.050952 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.050928    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kq4zj"]
Jul 29 19:17:45.051543 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.051524    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-q724q"]
Jul 29 19:17:45.196629 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.196604    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7266743b-b72b-41da-8110-b47af371be5c-cert\") pod \"ingress-canary-q724q\" (UID: \"7266743b-b72b-41da-8110-b47af371be5c\") " pod="openshift-ingress-canary/ingress-canary-q724q"
Jul 29 19:17:45.196740 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.196650    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffafbd2c-ca03-47e9-ab0d-cd6125697e9e-tmp-dir\") pod \"dns-default-kq4zj\" (UID: \"ffafbd2c-ca03-47e9-ab0d-cd6125697e9e\") " pod="openshift-dns/dns-default-kq4zj"
Jul 29 19:17:45.196740 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.196705    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ffafbd2c-ca03-47e9-ab0d-cd6125697e9e-config-volume\") pod \"dns-default-kq4zj\" (UID: \"ffafbd2c-ca03-47e9-ab0d-cd6125697e9e\") " pod="openshift-dns/dns-default-kq4zj"
Jul 29 19:17:45.196740 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.196724    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffafbd2c-ca03-47e9-ab0d-cd6125697e9e-metrics-tls\") pod \"dns-default-kq4zj\" (UID: \"ffafbd2c-ca03-47e9-ab0d-cd6125697e9e\") " pod="openshift-dns/dns-default-kq4zj"
Jul 29 19:17:45.196833 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.196750    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4rsg\" (UniqueName: \"kubernetes.io/projected/ffafbd2c-ca03-47e9-ab0d-cd6125697e9e-kube-api-access-t4rsg\") pod \"dns-default-kq4zj\" (UID: \"ffafbd2c-ca03-47e9-ab0d-cd6125697e9e\") " pod="openshift-dns/dns-default-kq4zj"
Jul 29 19:17:45.196833 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.196794    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnp4n\" (UniqueName: \"kubernetes.io/projected/7266743b-b72b-41da-8110-b47af371be5c-kube-api-access-cnp4n\") pod \"ingress-canary-q724q\" (UID: \"7266743b-b72b-41da-8110-b47af371be5c\") " pod="openshift-ingress-canary/ingress-canary-q724q"
Jul 29 19:17:45.297955 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.297931    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4rsg\" (UniqueName: \"kubernetes.io/projected/ffafbd2c-ca03-47e9-ab0d-cd6125697e9e-kube-api-access-t4rsg\") pod \"dns-default-kq4zj\" (UID: \"ffafbd2c-ca03-47e9-ab0d-cd6125697e9e\") " pod="openshift-dns/dns-default-kq4zj"
Jul 29 19:17:45.298044 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.297995    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnp4n\" (UniqueName: \"kubernetes.io/projected/7266743b-b72b-41da-8110-b47af371be5c-kube-api-access-cnp4n\") pod \"ingress-canary-q724q\" (UID: \"7266743b-b72b-41da-8110-b47af371be5c\") " pod="openshift-ingress-canary/ingress-canary-q724q"
Jul 29 19:17:45.298044 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.298024    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7266743b-b72b-41da-8110-b47af371be5c-cert\") pod \"ingress-canary-q724q\" (UID: \"7266743b-b72b-41da-8110-b47af371be5c\") " pod="openshift-ingress-canary/ingress-canary-q724q"
Jul 29 19:17:45.298111 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.298050    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffafbd2c-ca03-47e9-ab0d-cd6125697e9e-tmp-dir\") pod \"dns-default-kq4zj\" (UID: \"ffafbd2c-ca03-47e9-ab0d-cd6125697e9e\") " pod="openshift-dns/dns-default-kq4zj"
Jul 29 19:17:45.298111 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.298093    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ffafbd2c-ca03-47e9-ab0d-cd6125697e9e-config-volume\") pod \"dns-default-kq4zj\" (UID: \"ffafbd2c-ca03-47e9-ab0d-cd6125697e9e\") " pod="openshift-dns/dns-default-kq4zj"
Jul 29 19:17:45.298185 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.298114    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffafbd2c-ca03-47e9-ab0d-cd6125697e9e-metrics-tls\") pod \"dns-default-kq4zj\" (UID: \"ffafbd2c-ca03-47e9-ab0d-cd6125697e9e\") " pod="openshift-dns/dns-default-kq4zj"
Jul 29 19:17:45.298229 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:45.298207    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 19:17:45.298270 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:45.298259    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffafbd2c-ca03-47e9-ab0d-cd6125697e9e-metrics-tls podName:ffafbd2c-ca03-47e9-ab0d-cd6125697e9e nodeName:}" failed. No retries permitted until 2026-07-29 19:17:45.798242059 +0000 UTC m=+36.625741102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ffafbd2c-ca03-47e9-ab0d-cd6125697e9e-metrics-tls") pod "dns-default-kq4zj" (UID: "ffafbd2c-ca03-47e9-ab0d-cd6125697e9e") : secret "dns-default-metrics-tls" not found
Jul 29 19:17:45.298448 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.298427    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffafbd2c-ca03-47e9-ab0d-cd6125697e9e-tmp-dir\") pod \"dns-default-kq4zj\" (UID: \"ffafbd2c-ca03-47e9-ab0d-cd6125697e9e\") " pod="openshift-dns/dns-default-kq4zj"
Jul 29 19:17:45.298740 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.298724    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ffafbd2c-ca03-47e9-ab0d-cd6125697e9e-config-volume\") pod \"dns-default-kq4zj\" (UID: \"ffafbd2c-ca03-47e9-ab0d-cd6125697e9e\") " pod="openshift-dns/dns-default-kq4zj"
Jul 29 19:17:45.302032 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.302012    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7266743b-b72b-41da-8110-b47af371be5c-cert\") pod \"ingress-canary-q724q\" (UID: \"7266743b-b72b-41da-8110-b47af371be5c\") " pod="openshift-ingress-canary/ingress-canary-q724q"
Jul 29 19:17:45.309069 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.309049    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4rsg\" (UniqueName: \"kubernetes.io/projected/ffafbd2c-ca03-47e9-ab0d-cd6125697e9e-kube-api-access-t4rsg\") pod \"dns-default-kq4zj\" (UID: \"ffafbd2c-ca03-47e9-ab0d-cd6125697e9e\") " pod="openshift-dns/dns-default-kq4zj"
Jul 29 19:17:45.309069 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.309065    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnp4n\" (UniqueName: \"kubernetes.io/projected/7266743b-b72b-41da-8110-b47af371be5c-kube-api-access-cnp4n\") pod \"ingress-canary-q724q\" (UID: \"7266743b-b72b-41da-8110-b47af371be5c\") " pod="openshift-ingress-canary/ingress-canary-q724q"
Jul 29 19:17:45.351149 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.351126    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-q724q"
Jul 29 19:17:45.478212 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.478180    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-q724q"]
Jul 29 19:17:45.481629 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:17:45.481604    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7266743b_b72b_41da_8110_b47af371be5c.slice/crio-423a93921b51dd0386a11f03749cfdd8436768f711cf413ee9b7d25db0c1eb7e WatchSource:0}: Error finding container 423a93921b51dd0386a11f03749cfdd8436768f711cf413ee9b7d25db0c1eb7e: Status 404 returned error can't find the container with id 423a93921b51dd0386a11f03749cfdd8436768f711cf413ee9b7d25db0c1eb7e
Jul 29 19:17:45.673343 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.673317    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cw75m"
Jul 29 19:17:45.676216 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.676193    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 19:17:45.676350 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.676202    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tgbwc\""
Jul 29 19:17:45.801839 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:45.801749    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffafbd2c-ca03-47e9-ab0d-cd6125697e9e-metrics-tls\") pod \"dns-default-kq4zj\" (UID: \"ffafbd2c-ca03-47e9-ab0d-cd6125697e9e\") " pod="openshift-dns/dns-default-kq4zj"
Jul 29 19:17:45.802009 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:45.801916    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 19:17:45.802009 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:45.801997    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffafbd2c-ca03-47e9-ab0d-cd6125697e9e-metrics-tls podName:ffafbd2c-ca03-47e9-ab0d-cd6125697e9e nodeName:}" failed. No retries permitted until 2026-07-29 19:17:46.801978635 +0000 UTC m=+37.629477689 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ffafbd2c-ca03-47e9-ab0d-cd6125697e9e-metrics-tls") pod "dns-default-kq4zj" (UID: "ffafbd2c-ca03-47e9-ab0d-cd6125697e9e") : secret "dns-default-metrics-tls" not found
Jul 29 19:17:46.007661 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:46.007621    2557 generic.go:356] "Generic (PLEG): container finished" podID="08c90b03-c2a2-478b-8d59-bb71b15e96f8" containerID="81c731c19f329059c44657b1903577059296423d465a0a46531734daa413f1ce" exitCode=0
Jul 29 19:17:46.007833 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:46.007703    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v67cp" event={"ID":"08c90b03-c2a2-478b-8d59-bb71b15e96f8","Type":"ContainerDied","Data":"81c731c19f329059c44657b1903577059296423d465a0a46531734daa413f1ce"}
Jul 29 19:17:46.009046 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:46.008773    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-q724q" event={"ID":"7266743b-b72b-41da-8110-b47af371be5c","Type":"ContainerStarted","Data":"423a93921b51dd0386a11f03749cfdd8436768f711cf413ee9b7d25db0c1eb7e"}
Jul 29 19:17:46.673252 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:46.673219    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqpxs"
Jul 29 19:17:46.673413 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:46.673230    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:17:46.676320 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:46.676301    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 19:17:46.677473 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:46.677455    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bfbzc\""
Jul 29 19:17:46.677611 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:46.677477    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 19:17:46.677759 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:46.677743    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 19:17:46.810109 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:46.810075    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffafbd2c-ca03-47e9-ab0d-cd6125697e9e-metrics-tls\") pod \"dns-default-kq4zj\" (UID: \"ffafbd2c-ca03-47e9-ab0d-cd6125697e9e\") " pod="openshift-dns/dns-default-kq4zj"
Jul 29 19:17:46.810283 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:46.810222    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 19:17:46.810283 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:46.810281    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffafbd2c-ca03-47e9-ab0d-cd6125697e9e-metrics-tls podName:ffafbd2c-ca03-47e9-ab0d-cd6125697e9e nodeName:}" failed. No retries permitted until 2026-07-29 19:17:48.810266838 +0000 UTC m=+39.637765878 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ffafbd2c-ca03-47e9-ab0d-cd6125697e9e-metrics-tls") pod "dns-default-kq4zj" (UID: "ffafbd2c-ca03-47e9-ab0d-cd6125697e9e") : secret "dns-default-metrics-tls" not found
Jul 29 19:17:47.014255 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:47.014170    2557 generic.go:356] "Generic (PLEG): container finished" podID="08c90b03-c2a2-478b-8d59-bb71b15e96f8" containerID="82e8a13b3435fc1eca0a9ae19e18c7c232704ea55ef11f46f1ba5ef1560195ae" exitCode=0
Jul 29 19:17:47.014408 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:47.014233    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v67cp" event={"ID":"08c90b03-c2a2-478b-8d59-bb71b15e96f8","Type":"ContainerDied","Data":"82e8a13b3435fc1eca0a9ae19e18c7c232704ea55ef11f46f1ba5ef1560195ae"}
Jul 29 19:17:48.020080 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:48.019885    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v67cp" event={"ID":"08c90b03-c2a2-478b-8d59-bb71b15e96f8","Type":"ContainerStarted","Data":"cd22519180ebe0cd0af41fcf2b5607de9bf570d02aa9a226008b141a2167741d"}
Jul 29 19:17:48.047232 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:48.047188    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-v67cp" podStartSLOduration=8.853224407999999 podStartE2EDuration="38.047174292s" totalImagesPullingTime="29.193949884s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:17:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:17:48.045336595 +0000 UTC m=+38.872835683" watchObservedRunningTime="2026-07-29 19:17:48.047174292 +0000 UTC m=+38.874673349"
Jul 29 19:17:48.825713 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:48.825683    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffafbd2c-ca03-47e9-ab0d-cd6125697e9e-metrics-tls\") pod \"dns-default-kq4zj\" (UID: \"ffafbd2c-ca03-47e9-ab0d-cd6125697e9e\") " pod="openshift-dns/dns-default-kq4zj"
Jul 29 19:17:48.825855 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:48.825838    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 19:17:48.825918 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:48.825906    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffafbd2c-ca03-47e9-ab0d-cd6125697e9e-metrics-tls podName:ffafbd2c-ca03-47e9-ab0d-cd6125697e9e nodeName:}" failed. No retries permitted until 2026-07-29 19:17:52.82588813 +0000 UTC m=+43.653387165 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ffafbd2c-ca03-47e9-ab0d-cd6125697e9e-metrics-tls") pod "dns-default-kq4zj" (UID: "ffafbd2c-ca03-47e9-ab0d-cd6125697e9e") : secret "dns-default-metrics-tls" not found
Jul 29 19:17:49.023391 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:49.023356    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-q724q" event={"ID":"7266743b-b72b-41da-8110-b47af371be5c","Type":"ContainerStarted","Data":"1947f028bdc6a0e96dcb8d85480341b86a28d0c04e607e0e9a7f0ac2535d072a"}
Jul 29 19:17:49.041879 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:49.041835    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-q724q" podStartSLOduration=1.225071495 podStartE2EDuration="4.041822234s" totalImagesPullingTime="2.816750739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:17:49.041740121 +0000 UTC m=+39.869239178" watchObservedRunningTime="2026-07-29 19:17:49.041822234 +0000 UTC m=+39.869321291"
Jul 29 19:17:52.853800 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:17:52.853756    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffafbd2c-ca03-47e9-ab0d-cd6125697e9e-metrics-tls\") pod \"dns-default-kq4zj\" (UID: \"ffafbd2c-ca03-47e9-ab0d-cd6125697e9e\") " pod="openshift-dns/dns-default-kq4zj"
Jul 29 19:17:52.854300 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:52.853931    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 19:17:52.854300 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:17:52.854047    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffafbd2c-ca03-47e9-ab0d-cd6125697e9e-metrics-tls podName:ffafbd2c-ca03-47e9-ab0d-cd6125697e9e nodeName:}" failed. No retries permitted until 2026-07-29 19:18:00.854025819 +0000 UTC m=+51.681524855 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ffafbd2c-ca03-47e9-ab0d-cd6125697e9e-metrics-tls") pod "dns-default-kq4zj" (UID: "ffafbd2c-ca03-47e9-ab0d-cd6125697e9e") : secret "dns-default-metrics-tls" not found
Jul 29 19:18:00.912130 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:00.912085    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffafbd2c-ca03-47e9-ab0d-cd6125697e9e-metrics-tls\") pod \"dns-default-kq4zj\" (UID: \"ffafbd2c-ca03-47e9-ab0d-cd6125697e9e\") " pod="openshift-dns/dns-default-kq4zj"
Jul 29 19:18:00.912591 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:00.912237    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 19:18:00.912591 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:00.912311    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffafbd2c-ca03-47e9-ab0d-cd6125697e9e-metrics-tls podName:ffafbd2c-ca03-47e9-ab0d-cd6125697e9e nodeName:}" failed. No retries permitted until 2026-07-29 19:18:16.912293504 +0000 UTC m=+67.739792540 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ffafbd2c-ca03-47e9-ab0d-cd6125697e9e-metrics-tls") pod "dns-default-kq4zj" (UID: "ffafbd2c-ca03-47e9-ab0d-cd6125697e9e") : secret "dns-default-metrics-tls" not found
Jul 29 19:18:09.137624 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.137590    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kwjqt"]
Jul 29 19:18:09.158024 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.157994    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kwjqt"
Jul 29 19:18:09.158860 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.158839    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kwjqt"]
Jul 29 19:18:09.164615 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.164590    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-dmgdk\""
Jul 29 19:18:09.165065 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.164863    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 29 19:18:09.165782 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.165761    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 29 19:18:09.165866 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.165797    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 29 19:18:09.169515 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.169495    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8djv\" (UniqueName: \"kubernetes.io/projected/94a768fa-47eb-480c-85e8-fb69c78da295-kube-api-access-b8djv\") pod \"cluster-samples-operator-6748489656-kwjqt\" (UID: \"94a768fa-47eb-480c-85e8-fb69c78da295\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kwjqt"
Jul 29 19:18:09.169578 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.169562    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/94a768fa-47eb-480c-85e8-fb69c78da295-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-kwjqt\" (UID: \"94a768fa-47eb-480c-85e8-fb69c78da295\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kwjqt"
Jul 29 19:18:09.270345 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.270304    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/94a768fa-47eb-480c-85e8-fb69c78da295-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-kwjqt\" (UID: \"94a768fa-47eb-480c-85e8-fb69c78da295\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kwjqt"
Jul 29 19:18:09.270494 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.270369    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8djv\" (UniqueName: \"kubernetes.io/projected/94a768fa-47eb-480c-85e8-fb69c78da295-kube-api-access-b8djv\") pod \"cluster-samples-operator-6748489656-kwjqt\" (UID: \"94a768fa-47eb-480c-85e8-fb69c78da295\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kwjqt"
Jul 29 19:18:09.270494 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:09.270468    2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 19:18:09.270578 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:09.270551    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94a768fa-47eb-480c-85e8-fb69c78da295-samples-operator-tls podName:94a768fa-47eb-480c-85e8-fb69c78da295 nodeName:}" failed. No retries permitted until 2026-07-29 19:18:09.770530137 +0000 UTC m=+60.598029174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/94a768fa-47eb-480c-85e8-fb69c78da295-samples-operator-tls") pod "cluster-samples-operator-6748489656-kwjqt" (UID: "94a768fa-47eb-480c-85e8-fb69c78da295") : secret "samples-operator-tls" not found
Jul 29 19:18:09.287811 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.287784    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8djv\" (UniqueName: \"kubernetes.io/projected/94a768fa-47eb-480c-85e8-fb69c78da295-kube-api-access-b8djv\") pod \"cluster-samples-operator-6748489656-kwjqt\" (UID: \"94a768fa-47eb-480c-85e8-fb69c78da295\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kwjqt"
Jul 29 19:18:09.341959 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.341929    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-dh7rf"]
Jul 29 19:18:09.358315 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.358290    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7fc6497669-xk7lp"]
Jul 29 19:18:09.358452 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.358430    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-dh7rf"
Jul 29 19:18:09.361393 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.361371    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 29 19:18:09.361501 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.361397    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-wmf52\""
Jul 29 19:18:09.362553 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.362534    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 29 19:18:09.371398 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.371374    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/259710be-9c9b-4457-81c2-301f7d7cb110-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-dh7rf\" (UID: \"259710be-9c9b-4457-81c2-301f7d7cb110\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-dh7rf"
Jul 29 19:18:09.371508 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.371405    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/259710be-9c9b-4457-81c2-301f7d7cb110-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-dh7rf\" (UID: \"259710be-9c9b-4457-81c2-301f7d7cb110\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-dh7rf"
Jul 29 19:18:09.374915 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.374896    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-dh7rf"]
Jul 29 19:18:09.375022 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.374993    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7fc6497669-xk7lp"
Jul 29 19:18:09.376169 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.376148    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7fc6497669-xk7lp"]
Jul 29 19:18:09.379361 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.379341    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 29 19:18:09.379448 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.379428    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 29 19:18:09.379662 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.379645    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 29 19:18:09.379780 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.379669    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 29 19:18:09.379780 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.379704    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-znt5r\""
Jul 29 19:18:09.379880 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.379830    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 29 19:18:09.379920 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.379913    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 29 19:18:09.429473 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.429400    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-cfmh8"]
Jul 29 19:18:09.446486 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.446460    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-cfmh8"]
Jul 29 19:18:09.446616 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.446572    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-cfmh8"
Jul 29 19:18:09.449791 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.449767    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 29 19:18:09.450007 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.449992    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 29 19:18:09.450146 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.450127    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 29 19:18:09.451326 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.451310    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-w9vj6\""
Jul 29 19:18:09.451495 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.451482    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 29 19:18:09.455689 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.455672    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 29 19:18:09.471733 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.471708    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbwg2\" (UniqueName: \"kubernetes.io/projected/dd24fa62-fd72-4e18-af46-6930625f22da-kube-api-access-fbwg2\") pod \"router-default-7fc6497669-xk7lp\" (UID: \"dd24fa62-fd72-4e18-af46-6930625f22da\") " pod="openshift-ingress/router-default-7fc6497669-xk7lp"
Jul 29 19:18:09.471828 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.471754    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac-snapshots\") pod \"insights-operator-5885fdb759-cfmh8\" (UID: \"2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac\") " pod="openshift-insights/insights-operator-5885fdb759-cfmh8"
Jul 29 19:18:09.471828 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.471784    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/dd24fa62-fd72-4e18-af46-6930625f22da-default-certificate\") pod \"router-default-7fc6497669-xk7lp\" (UID: \"dd24fa62-fd72-4e18-af46-6930625f22da\") " pod="openshift-ingress/router-default-7fc6497669-xk7lp"
Jul 29 19:18:09.471828 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.471807    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd24fa62-fd72-4e18-af46-6930625f22da-metrics-certs\") pod \"router-default-7fc6497669-xk7lp\" (UID: \"dd24fa62-fd72-4e18-af46-6930625f22da\") " pod="openshift-ingress/router-default-7fc6497669-xk7lp"
Jul 29 19:18:09.471950 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.471828    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac-serving-cert\") pod \"insights-operator-5885fdb759-cfmh8\" (UID: \"2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac\") " pod="openshift-insights/insights-operator-5885fdb759-cfmh8"
Jul 29 19:18:09.471950 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.471910    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/259710be-9c9b-4457-81c2-301f7d7cb110-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-dh7rf\" (UID: \"259710be-9c9b-4457-81c2-301f7d7cb110\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-dh7rf"
Jul 29 19:18:09.471950 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.471944    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb82w\" (UniqueName: \"kubernetes.io/projected/2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac-kube-api-access-jb82w\") pod \"insights-operator-5885fdb759-cfmh8\" (UID: \"2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac\") " pod="openshift-insights/insights-operator-5885fdb759-cfmh8"
Jul 29 19:18:09.472098 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.471987    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-cfmh8\" (UID: \"2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac\") " pod="openshift-insights/insights-operator-5885fdb759-cfmh8"
Jul 29 19:18:09.472098 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:09.471994    2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 19:18:09.472098 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.472017    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/259710be-9c9b-4457-81c2-301f7d7cb110-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-dh7rf\" (UID: \"259710be-9c9b-4457-81c2-301f7d7cb110\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-dh7rf"
Jul 29 19:18:09.472098 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.472041    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd24fa62-fd72-4e18-af46-6930625f22da-service-ca-bundle\") pod \"router-default-7fc6497669-xk7lp\" (UID: \"dd24fa62-fd72-4e18-af46-6930625f22da\") " pod="openshift-ingress/router-default-7fc6497669-xk7lp"
Jul 29 19:18:09.472098 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:09.472052    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/259710be-9c9b-4457-81c2-301f7d7cb110-networking-console-plugin-cert podName:259710be-9c9b-4457-81c2-301f7d7cb110 nodeName:}" failed. No retries permitted until 2026-07-29 19:18:09.97203698 +0000 UTC m=+60.799536016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/259710be-9c9b-4457-81c2-301f7d7cb110-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-dh7rf" (UID: "259710be-9c9b-4457-81c2-301f7d7cb110") : secret "networking-console-plugin-cert" not found
Jul 29 19:18:09.472098 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.472083    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/dd24fa62-fd72-4e18-af46-6930625f22da-stats-auth\") pod \"router-default-7fc6497669-xk7lp\" (UID: \"dd24fa62-fd72-4e18-af46-6930625f22da\") " pod="openshift-ingress/router-default-7fc6497669-xk7lp"
Jul 29 19:18:09.472343 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.472165    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac-service-ca-bundle\") pod \"insights-operator-5885fdb759-cfmh8\" (UID: \"2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac\") " pod="openshift-insights/insights-operator-5885fdb759-cfmh8"
Jul 29 19:18:09.472343 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.472197    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac-tmp\") pod \"insights-operator-5885fdb759-cfmh8\" (UID: \"2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac\") " pod="openshift-insights/insights-operator-5885fdb759-cfmh8"
Jul 29 19:18:09.472798 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.472777    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/259710be-9c9b-4457-81c2-301f7d7cb110-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-dh7rf\" (UID: \"259710be-9c9b-4457-81c2-301f7d7cb110\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-dh7rf"
Jul 29 19:18:09.573042 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.573011    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb82w\" (UniqueName: \"kubernetes.io/projected/2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac-kube-api-access-jb82w\") pod \"insights-operator-5885fdb759-cfmh8\" (UID: \"2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac\") " pod="openshift-insights/insights-operator-5885fdb759-cfmh8"
Jul 29 19:18:09.573195 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.573040    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-cfmh8\" (UID: \"2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac\") " pod="openshift-insights/insights-operator-5885fdb759-cfmh8"
Jul 29 19:18:09.573195 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.573078    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd24fa62-fd72-4e18-af46-6930625f22da-service-ca-bundle\") pod \"router-default-7fc6497669-xk7lp\" (UID: \"dd24fa62-fd72-4e18-af46-6930625f22da\") " pod="openshift-ingress/router-default-7fc6497669-xk7lp"
Jul 29 19:18:09.573195 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.573094    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/dd24fa62-fd72-4e18-af46-6930625f22da-stats-auth\") pod \"router-default-7fc6497669-xk7lp\" (UID: \"dd24fa62-fd72-4e18-af46-6930625f22da\") " pod="openshift-ingress/router-default-7fc6497669-xk7lp"
Jul 29 19:18:09.573195 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.573112    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac-service-ca-bundle\") pod \"insights-operator-5885fdb759-cfmh8\" (UID: \"2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac\") " pod="openshift-insights/insights-operator-5885fdb759-cfmh8"
Jul 29 19:18:09.573195 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.573178    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac-tmp\") pod \"insights-operator-5885fdb759-cfmh8\" (UID: \"2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac\") " pod="openshift-insights/insights-operator-5885fdb759-cfmh8"
Jul 29 19:18:09.573415 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.573284    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbwg2\" (UniqueName: \"kubernetes.io/projected/dd24fa62-fd72-4e18-af46-6930625f22da-kube-api-access-fbwg2\") pod \"router-default-7fc6497669-xk7lp\" (UID: \"dd24fa62-fd72-4e18-af46-6930625f22da\") " pod="openshift-ingress/router-default-7fc6497669-xk7lp"
Jul 29 19:18:09.573415 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.573310    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac-snapshots\") pod \"insights-operator-5885fdb759-cfmh8\" (UID: \"2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac\") " pod="openshift-insights/insights-operator-5885fdb759-cfmh8"
Jul 29 19:18:09.573415 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.573333    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/dd24fa62-fd72-4e18-af46-6930625f22da-default-certificate\") pod \"router-default-7fc6497669-xk7lp\" (UID: \"dd24fa62-fd72-4e18-af46-6930625f22da\") " pod="openshift-ingress/router-default-7fc6497669-xk7lp"
Jul 29 19:18:09.573415 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.573350    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd24fa62-fd72-4e18-af46-6930625f22da-metrics-certs\") pod \"router-default-7fc6497669-xk7lp\" (UID: \"dd24fa62-fd72-4e18-af46-6930625f22da\") " pod="openshift-ingress/router-default-7fc6497669-xk7lp"
Jul 29 19:18:09.573415 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.573366    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac-serving-cert\") pod \"insights-operator-5885fdb759-cfmh8\" (UID: \"2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac\") " pod="openshift-insights/insights-operator-5885fdb759-cfmh8"
Jul 29 19:18:09.573639 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.573589    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac-tmp\") pod \"insights-operator-5885fdb759-cfmh8\" (UID: \"2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac\") " pod="openshift-insights/insights-operator-5885fdb759-cfmh8"
Jul 29 19:18:09.573843 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.573820    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac-snapshots\") pod \"insights-operator-5885fdb759-cfmh8\" (UID: \"2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac\") " pod="openshift-insights/insights-operator-5885fdb759-cfmh8"
Jul 29 19:18:09.575943 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.575927    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 29 19:18:09.576018 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.575959    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 29 19:18:09.577396 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.577381    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 29 19:18:09.577480 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.577401    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 29 19:18:09.577480 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.577424    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 29 19:18:09.577667 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.577652    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 29 19:18:09.583519 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:09.583502    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd24fa62-fd72-4e18-af46-6930625f22da-service-ca-bundle podName:dd24fa62-fd72-4e18-af46-6930625f22da nodeName:}" failed. No retries permitted until 2026-07-29 19:18:10.083483935 +0000 UTC m=+60.910982970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dd24fa62-fd72-4e18-af46-6930625f22da-service-ca-bundle") pod "router-default-7fc6497669-xk7lp" (UID: "dd24fa62-fd72-4e18-af46-6930625f22da") : configmap references non-existent config key: service-ca.crt
Jul 29 19:18:09.583617 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:09.583554    2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 29 19:18:09.583673 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:09.583624    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd24fa62-fd72-4e18-af46-6930625f22da-metrics-certs podName:dd24fa62-fd72-4e18-af46-6930625f22da nodeName:}" failed. No retries permitted until 2026-07-29 19:18:10.083606274 +0000 UTC m=+60.911105325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd24fa62-fd72-4e18-af46-6930625f22da-metrics-certs") pod "router-default-7fc6497669-xk7lp" (UID: "dd24fa62-fd72-4e18-af46-6930625f22da") : secret "router-metrics-certs-default" not found
Jul 29 19:18:09.583943 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.583912    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac-service-ca-bundle\") pod \"insights-operator-5885fdb759-cfmh8\" (UID: \"2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac\") " pod="openshift-insights/insights-operator-5885fdb759-cfmh8"
Jul 29 19:18:09.585391 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.585367    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 29 19:18:09.585919 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.585895    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac-serving-cert\") pod \"insights-operator-5885fdb759-cfmh8\" (UID: \"2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac\") " pod="openshift-insights/insights-operator-5885fdb759-cfmh8"
Jul 29 19:18:09.586023 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.585936    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/dd24fa62-fd72-4e18-af46-6930625f22da-stats-auth\") pod \"router-default-7fc6497669-xk7lp\" (UID: \"dd24fa62-fd72-4e18-af46-6930625f22da\") " pod="openshift-ingress/router-default-7fc6497669-xk7lp"
Jul 29 19:18:09.586023 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.586003    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/dd24fa62-fd72-4e18-af46-6930625f22da-default-certificate\") pod \"router-default-7fc6497669-xk7lp\" (UID: \"dd24fa62-fd72-4e18-af46-6930625f22da\") " pod="openshift-ingress/router-default-7fc6497669-xk7lp"
Jul 29 19:18:09.587087 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.587061    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 29 19:18:09.589335 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.589310    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 29 19:18:09.596830 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.596813    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 29 19:18:09.599714 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.599698    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 29 19:18:09.606871 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.606851    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb82w\" (UniqueName: \"kubernetes.io/projected/2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac-kube-api-access-jb82w\") pod \"insights-operator-5885fdb759-cfmh8\" (UID: \"2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac\") " pod="openshift-insights/insights-operator-5885fdb759-cfmh8"
Jul 29 19:18:09.609683 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.609665    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbwg2\" (UniqueName: \"kubernetes.io/projected/dd24fa62-fd72-4e18-af46-6930625f22da-kube-api-access-fbwg2\") pod \"router-default-7fc6497669-xk7lp\" (UID: \"dd24fa62-fd72-4e18-af46-6930625f22da\") " pod="openshift-ingress/router-default-7fc6497669-xk7lp"
Jul 29 19:18:09.617417 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.617394    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-cfmh8\" (UID: \"2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac\") " pod="openshift-insights/insights-operator-5885fdb759-cfmh8"
Jul 29 19:18:09.759061 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.758997    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-w9vj6\""
Jul 29 19:18:09.766572 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.766554    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-cfmh8"
Jul 29 19:18:09.774053 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.774033    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/94a768fa-47eb-480c-85e8-fb69c78da295-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-kwjqt\" (UID: \"94a768fa-47eb-480c-85e8-fb69c78da295\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kwjqt"
Jul 29 19:18:09.774185 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:09.774168    2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 19:18:09.774235 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:09.774225    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94a768fa-47eb-480c-85e8-fb69c78da295-samples-operator-tls podName:94a768fa-47eb-480c-85e8-fb69c78da295 nodeName:}" failed. No retries permitted until 2026-07-29 19:18:10.774210126 +0000 UTC m=+61.601709162 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/94a768fa-47eb-480c-85e8-fb69c78da295-samples-operator-tls") pod "cluster-samples-operator-6748489656-kwjqt" (UID: "94a768fa-47eb-480c-85e8-fb69c78da295") : secret "samples-operator-tls" not found
Jul 29 19:18:09.919164 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.919128    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-cfmh8"]
Jul 29 19:18:09.923078 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:18:09.923047    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eacb5ca_ad77_4c2c_8fca_7f99a1c8c3ac.slice/crio-81b81273452d4fe1ab4addf06e190b48654e28b76009c7fa71b784d2fcc278f1 WatchSource:0}: Error finding container 81b81273452d4fe1ab4addf06e190b48654e28b76009c7fa71b784d2fcc278f1: Status 404 returned error can't find the container with id 81b81273452d4fe1ab4addf06e190b48654e28b76009c7fa71b784d2fcc278f1
Jul 29 19:18:09.976676 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:09.976643    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/259710be-9c9b-4457-81c2-301f7d7cb110-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-dh7rf\" (UID: \"259710be-9c9b-4457-81c2-301f7d7cb110\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-dh7rf"
Jul 29 19:18:09.976825 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:09.976792    2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 19:18:09.976870 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:09.976854    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/259710be-9c9b-4457-81c2-301f7d7cb110-networking-console-plugin-cert podName:259710be-9c9b-4457-81c2-301f7d7cb110 nodeName:}" failed. No retries permitted until 2026-07-29 19:18:10.976838248 +0000 UTC m=+61.804337284 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/259710be-9c9b-4457-81c2-301f7d7cb110-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-dh7rf" (UID: "259710be-9c9b-4457-81c2-301f7d7cb110") : secret "networking-console-plugin-cert" not found
Jul 29 19:18:10.004107 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:10.004079    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q6hd8"
Jul 29 19:18:10.071277 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:10.071204    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-cfmh8" event={"ID":"2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac","Type":"ContainerStarted","Data":"81b81273452d4fe1ab4addf06e190b48654e28b76009c7fa71b784d2fcc278f1"}
Jul 29 19:18:10.177952 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:10.177917    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd24fa62-fd72-4e18-af46-6930625f22da-service-ca-bundle\") pod \"router-default-7fc6497669-xk7lp\" (UID: \"dd24fa62-fd72-4e18-af46-6930625f22da\") " pod="openshift-ingress/router-default-7fc6497669-xk7lp"
Jul 29 19:18:10.178359 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:10.178050    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd24fa62-fd72-4e18-af46-6930625f22da-metrics-certs\") pod \"router-default-7fc6497669-xk7lp\" (UID: \"dd24fa62-fd72-4e18-af46-6930625f22da\") " pod="openshift-ingress/router-default-7fc6497669-xk7lp"
Jul 29 19:18:10.178359 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:10.178068    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd24fa62-fd72-4e18-af46-6930625f22da-service-ca-bundle podName:dd24fa62-fd72-4e18-af46-6930625f22da nodeName:}" failed. No retries permitted until 2026-07-29 19:18:11.17805043 +0000 UTC m=+62.005549469 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dd24fa62-fd72-4e18-af46-6930625f22da-service-ca-bundle") pod "router-default-7fc6497669-xk7lp" (UID: "dd24fa62-fd72-4e18-af46-6930625f22da") : configmap references non-existent config key: service-ca.crt
Jul 29 19:18:10.178359 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:10.178139    2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 29 19:18:10.178359 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:10.178193    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd24fa62-fd72-4e18-af46-6930625f22da-metrics-certs podName:dd24fa62-fd72-4e18-af46-6930625f22da nodeName:}" failed. No retries permitted until 2026-07-29 19:18:11.178176558 +0000 UTC m=+62.005675608 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd24fa62-fd72-4e18-af46-6930625f22da-metrics-certs") pod "router-default-7fc6497669-xk7lp" (UID: "dd24fa62-fd72-4e18-af46-6930625f22da") : secret "router-metrics-certs-default" not found
Jul 29 19:18:10.785217 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:10.785175    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/94a768fa-47eb-480c-85e8-fb69c78da295-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-kwjqt\" (UID: \"94a768fa-47eb-480c-85e8-fb69c78da295\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kwjqt"
Jul 29 19:18:10.785404 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:10.785342    2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 19:18:10.785456 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:10.785412    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94a768fa-47eb-480c-85e8-fb69c78da295-samples-operator-tls podName:94a768fa-47eb-480c-85e8-fb69c78da295 nodeName:}" failed. No retries permitted until 2026-07-29 19:18:12.785393716 +0000 UTC m=+63.612892752 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/94a768fa-47eb-480c-85e8-fb69c78da295-samples-operator-tls") pod "cluster-samples-operator-6748489656-kwjqt" (UID: "94a768fa-47eb-480c-85e8-fb69c78da295") : secret "samples-operator-tls" not found
Jul 29 19:18:10.986502 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:10.986463    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/259710be-9c9b-4457-81c2-301f7d7cb110-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-dh7rf\" (UID: \"259710be-9c9b-4457-81c2-301f7d7cb110\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-dh7rf"
Jul 29 19:18:10.986693 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:10.986637    2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 19:18:10.986746 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:10.986722    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/259710be-9c9b-4457-81c2-301f7d7cb110-networking-console-plugin-cert podName:259710be-9c9b-4457-81c2-301f7d7cb110 nodeName:}" failed. No retries permitted until 2026-07-29 19:18:12.986700316 +0000 UTC m=+63.814199354 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/259710be-9c9b-4457-81c2-301f7d7cb110-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-dh7rf" (UID: "259710be-9c9b-4457-81c2-301f7d7cb110") : secret "networking-console-plugin-cert" not found
Jul 29 19:18:11.187991 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:11.187940    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd24fa62-fd72-4e18-af46-6930625f22da-metrics-certs\") pod \"router-default-7fc6497669-xk7lp\" (UID: \"dd24fa62-fd72-4e18-af46-6930625f22da\") " pod="openshift-ingress/router-default-7fc6497669-xk7lp"
Jul 29 19:18:11.188396 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:11.188057    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd24fa62-fd72-4e18-af46-6930625f22da-service-ca-bundle\") pod \"router-default-7fc6497669-xk7lp\" (UID: \"dd24fa62-fd72-4e18-af46-6930625f22da\") " pod="openshift-ingress/router-default-7fc6497669-xk7lp"
Jul 29 19:18:11.188396 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:11.188084    2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 29 19:18:11.188396 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:11.188255    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd24fa62-fd72-4e18-af46-6930625f22da-metrics-certs podName:dd24fa62-fd72-4e18-af46-6930625f22da nodeName:}" failed. No retries permitted until 2026-07-29 19:18:13.188138372 +0000 UTC m=+64.015637408 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd24fa62-fd72-4e18-af46-6930625f22da-metrics-certs") pod "router-default-7fc6497669-xk7lp" (UID: "dd24fa62-fd72-4e18-af46-6930625f22da") : secret "router-metrics-certs-default" not found
Jul 29 19:18:11.188396 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:11.188283    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd24fa62-fd72-4e18-af46-6930625f22da-service-ca-bundle podName:dd24fa62-fd72-4e18-af46-6930625f22da nodeName:}" failed. No retries permitted until 2026-07-29 19:18:13.188272298 +0000 UTC m=+64.015771333 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dd24fa62-fd72-4e18-af46-6930625f22da-service-ca-bundle") pod "router-default-7fc6497669-xk7lp" (UID: "dd24fa62-fd72-4e18-af46-6930625f22da") : configmap references non-existent config key: service-ca.crt
Jul 29 19:18:12.077948 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:12.077907    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-cfmh8" event={"ID":"2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac","Type":"ContainerStarted","Data":"b0add6812fd061fd47bf81d64630c543bdd9a97668fcd8b6915f30d376a7aaac"}
Jul 29 19:18:12.098252 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:12.098205    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-cfmh8" podStartSLOduration=1.094264567 podStartE2EDuration="3.098190325s" totalImagesPullingTime="2.003925758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:18:12.097266793 +0000 UTC m=+62.924765850" watchObservedRunningTime="2026-07-29 19:18:12.098190325 +0000 UTC m=+62.925689382"
Jul 29 19:18:12.801846 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:12.801812    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/94a768fa-47eb-480c-85e8-fb69c78da295-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-kwjqt\" (UID: \"94a768fa-47eb-480c-85e8-fb69c78da295\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kwjqt"
Jul 29 19:18:12.802219 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:12.801917    2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 19:18:12.802219 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:12.801985    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94a768fa-47eb-480c-85e8-fb69c78da295-samples-operator-tls podName:94a768fa-47eb-480c-85e8-fb69c78da295 nodeName:}" failed. No retries permitted until 2026-07-29 19:18:16.801959799 +0000 UTC m=+67.629458835 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/94a768fa-47eb-480c-85e8-fb69c78da295-samples-operator-tls") pod "cluster-samples-operator-6748489656-kwjqt" (UID: "94a768fa-47eb-480c-85e8-fb69c78da295") : secret "samples-operator-tls" not found
Jul 29 19:18:13.003453 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:13.003406    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/259710be-9c9b-4457-81c2-301f7d7cb110-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-dh7rf\" (UID: \"259710be-9c9b-4457-81c2-301f7d7cb110\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-dh7rf"
Jul 29 19:18:13.003631 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:13.003567    2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 19:18:13.003680 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:13.003638    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/259710be-9c9b-4457-81c2-301f7d7cb110-networking-console-plugin-cert podName:259710be-9c9b-4457-81c2-301f7d7cb110 nodeName:}" failed. No retries permitted until 2026-07-29 19:18:17.003623669 +0000 UTC m=+67.831122705 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/259710be-9c9b-4457-81c2-301f7d7cb110-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-dh7rf" (UID: "259710be-9c9b-4457-81c2-301f7d7cb110") : secret "networking-console-plugin-cert" not found
Jul 29 19:18:13.088570 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:13.088491    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bpq7l"]
Jul 29 19:18:13.090680 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:13.090657    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bpq7l"
Jul 29 19:18:13.093416 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:13.093388    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"insights-runtime-extractor-bpq7l\" is forbidden: User \"system:node:ip-10-0-128-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-128-173.ec2.internal' and this object" podUID="6e8a45f8-a6cf-4d48-b1cb-92a91e190c50" pod="openshift-insights/insights-runtime-extractor-bpq7l"
Jul 29 19:18:13.094426 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:13.094401    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"insights-runtime-extractor-sa-dockercfg-knhpr\" is forbidden: User \"system:node:ip-10-0-128-173.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-128-173.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-knhpr\"" type="*v1.Secret"
Jul 29 19:18:13.094596 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:13.094578    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"insights-runtime-extractor-tls\" is forbidden: User \"system:node:ip-10-0-128-173.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-128-173.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret"
Jul 29 19:18:13.094656 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:13.094588    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-rbac-proxy\" is forbidden: User \"system:node:ip-10-0-128-173.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-128-173.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap"
Jul 29 19:18:13.114475 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:13.114455    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bpq7l"]
Jul 29 19:18:13.206054 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:13.206025    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-crio-socket\") pod \"insights-runtime-extractor-bpq7l\" (UID: \"6e8a45f8-a6cf-4d48-b1cb-92a91e190c50\") " pod="openshift-insights/insights-runtime-extractor-bpq7l"
Jul 29 19:18:13.206054 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:13.206056    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7g5f\" (UniqueName: \"kubernetes.io/projected/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-kube-api-access-h7g5f\") pod \"insights-runtime-extractor-bpq7l\" (UID: \"6e8a45f8-a6cf-4d48-b1cb-92a91e190c50\") " pod="openshift-insights/insights-runtime-extractor-bpq7l"
Jul 29 19:18:13.206228 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:13.206081    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd24fa62-fd72-4e18-af46-6930625f22da-service-ca-bundle\") pod \"router-default-7fc6497669-xk7lp\" (UID: \"dd24fa62-fd72-4e18-af46-6930625f22da\") " pod="openshift-ingress/router-default-7fc6497669-xk7lp"
Jul 29 19:18:13.206228 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:13.206137    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bpq7l\" (UID: \"6e8a45f8-a6cf-4d48-b1cb-92a91e190c50\") " pod="openshift-insights/insights-runtime-extractor-bpq7l"
Jul 29 19:18:13.206228 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:13.206168    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd24fa62-fd72-4e18-af46-6930625f22da-service-ca-bundle podName:dd24fa62-fd72-4e18-af46-6930625f22da nodeName:}" failed. No retries permitted until 2026-07-29 19:18:17.206155213 +0000 UTC m=+68.033654253 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dd24fa62-fd72-4e18-af46-6930625f22da-service-ca-bundle") pod "router-default-7fc6497669-xk7lp" (UID: "dd24fa62-fd72-4e18-af46-6930625f22da") : configmap references non-existent config key: service-ca.crt
Jul 29 19:18:13.206228 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:13.206193    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bpq7l\" (UID: \"6e8a45f8-a6cf-4d48-b1cb-92a91e190c50\") " pod="openshift-insights/insights-runtime-extractor-bpq7l"
Jul 29 19:18:13.206378 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:13.206249    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-data-volume\") pod \"insights-runtime-extractor-bpq7l\" (UID: \"6e8a45f8-a6cf-4d48-b1cb-92a91e190c50\") " pod="openshift-insights/insights-runtime-extractor-bpq7l"
Jul 29 19:18:13.206378 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:13.206268    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd24fa62-fd72-4e18-af46-6930625f22da-metrics-certs\") pod \"router-default-7fc6497669-xk7lp\" (UID: \"dd24fa62-fd72-4e18-af46-6930625f22da\") " pod="openshift-ingress/router-default-7fc6497669-xk7lp"
Jul 29 19:18:13.206378 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:13.206347    2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 29 19:18:13.206378 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:13.206374    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd24fa62-fd72-4e18-af46-6930625f22da-metrics-certs podName:dd24fa62-fd72-4e18-af46-6930625f22da nodeName:}" failed. No retries permitted until 2026-07-29 19:18:17.206366685 +0000 UTC m=+68.033865720 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd24fa62-fd72-4e18-af46-6930625f22da-metrics-certs") pod "router-default-7fc6497669-xk7lp" (UID: "dd24fa62-fd72-4e18-af46-6930625f22da") : secret "router-metrics-certs-default" not found
Jul 29 19:18:13.306853 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:13.306805    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bpq7l\" (UID: \"6e8a45f8-a6cf-4d48-b1cb-92a91e190c50\") " pod="openshift-insights/insights-runtime-extractor-bpq7l"
Jul 29 19:18:13.306853 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:13.306854    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bpq7l\" (UID: \"6e8a45f8-a6cf-4d48-b1cb-92a91e190c50\") " pod="openshift-insights/insights-runtime-extractor-bpq7l"
Jul 29 19:18:13.307108 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:13.306926    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-data-volume\") pod \"insights-runtime-extractor-bpq7l\" (UID: \"6e8a45f8-a6cf-4d48-b1cb-92a91e190c50\") " pod="openshift-insights/insights-runtime-extractor-bpq7l"
Jul 29 19:18:13.307108 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:13.307013    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-crio-socket\") pod \"insights-runtime-extractor-bpq7l\" (UID: \"6e8a45f8-a6cf-4d48-b1cb-92a91e190c50\") " pod="openshift-insights/insights-runtime-extractor-bpq7l"
Jul 29 19:18:13.307108 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:13.307038    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7g5f\" (UniqueName: \"kubernetes.io/projected/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-kube-api-access-h7g5f\") pod \"insights-runtime-extractor-bpq7l\" (UID: \"6e8a45f8-a6cf-4d48-b1cb-92a91e190c50\") " pod="openshift-insights/insights-runtime-extractor-bpq7l"
Jul 29 19:18:13.307231 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:13.307115    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-crio-socket\") pod \"insights-runtime-extractor-bpq7l\" (UID: \"6e8a45f8-a6cf-4d48-b1cb-92a91e190c50\") " pod="openshift-insights/insights-runtime-extractor-bpq7l"
Jul 29 19:18:13.307389 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:13.307367    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-data-volume\") pod \"insights-runtime-extractor-bpq7l\" (UID: \"6e8a45f8-a6cf-4d48-b1cb-92a91e190c50\") " pod="openshift-insights/insights-runtime-extractor-bpq7l"
Jul 29 19:18:13.316670 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:13.316647    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7g5f\" (UniqueName: \"kubernetes.io/projected/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-kube-api-access-h7g5f\") pod \"insights-runtime-extractor-bpq7l\" (UID: \"6e8a45f8-a6cf-4d48-b1cb-92a91e190c50\") " pod="openshift-insights/insights-runtime-extractor-bpq7l"
Jul 29 19:18:14.191628 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:14.191598    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 29 19:18:14.198007 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:14.197987    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 19:18:14.198098 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:14.198047    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-kube-rbac-proxy-cm podName:6e8a45f8-a6cf-4d48-b1cb-92a91e190c50 nodeName:}" failed. No retries permitted until 2026-07-29 19:18:14.698031423 +0000 UTC m=+65.525530459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bpq7l" (UID: "6e8a45f8-a6cf-4d48-b1cb-92a91e190c50") : configmap "kube-rbac-proxy" not found
Jul 29 19:18:14.222329 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:14.222309    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 29 19:18:14.227587 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:14.227573    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 19:18:14.227634 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:14.227619    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-insights-runtime-extractor-tls podName:6e8a45f8-a6cf-4d48-b1cb-92a91e190c50 nodeName:}" failed. No retries permitted until 2026-07-29 19:18:14.727607769 +0000 UTC m=+65.555106805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bpq7l" (UID: "6e8a45f8-a6cf-4d48-b1cb-92a91e190c50") : secret "insights-runtime-extractor-tls" not found
Jul 29 19:18:14.607230 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:14.607160    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-knhpr\""
Jul 29 19:18:14.720160 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:14.720129    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bpq7l\" (UID: \"6e8a45f8-a6cf-4d48-b1cb-92a91e190c50\") " pod="openshift-insights/insights-runtime-extractor-bpq7l"
Jul 29 19:18:14.720300 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:14.720249    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 19:18:14.720338 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:14.720311    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-kube-rbac-proxy-cm podName:6e8a45f8-a6cf-4d48-b1cb-92a91e190c50 nodeName:}" failed. No retries permitted until 2026-07-29 19:18:15.720295811 +0000 UTC m=+66.547794851 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bpq7l" (UID: "6e8a45f8-a6cf-4d48-b1cb-92a91e190c50") : configmap "kube-rbac-proxy" not found
Jul 29 19:18:14.821132 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:14.821103    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bpq7l\" (UID: \"6e8a45f8-a6cf-4d48-b1cb-92a91e190c50\") " pod="openshift-insights/insights-runtime-extractor-bpq7l"
Jul 29 19:18:14.821280 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:14.821209    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 19:18:14.821280 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:14.821259    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-insights-runtime-extractor-tls podName:6e8a45f8-a6cf-4d48-b1cb-92a91e190c50 nodeName:}" failed. No retries permitted until 2026-07-29 19:18:15.821246228 +0000 UTC m=+66.648745267 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bpq7l" (UID: "6e8a45f8-a6cf-4d48-b1cb-92a91e190c50") : secret "insights-runtime-extractor-tls" not found
Jul 29 19:18:15.144151 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:15.144123    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vft97_f737cfa7-db52-409a-b5c3-f704b9170fa4/dns-node-resolver/0.log"
Jul 29 19:18:15.326427 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:15.326393    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4f202a3-5dc7-416f-8793-e857e6086174-metrics-certs\") pod \"network-metrics-daemon-cw75m\" (UID: \"a4f202a3-5dc7-416f-8793-e857e6086174\") " pod="openshift-multus/network-metrics-daemon-cw75m"
Jul 29 19:18:15.329104 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:15.329085    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 19:18:15.337545 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:15.337526    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 29 19:18:15.337618 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:15.337581    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4f202a3-5dc7-416f-8793-e857e6086174-metrics-certs podName:a4f202a3-5dc7-416f-8793-e857e6086174 nodeName:}" failed. No retries permitted until 2026-07-29 19:19:19.337564943 +0000 UTC m=+130.165063982 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a4f202a3-5dc7-416f-8793-e857e6086174-metrics-certs") pod "network-metrics-daemon-cw75m" (UID: "a4f202a3-5dc7-416f-8793-e857e6086174") : secret "metrics-daemon-secret" not found
Jul 29 19:18:15.427801 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:15.427730    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93168299-ac4f-4fdd-8ac3-b69edb385d71-original-pull-secret\") pod \"global-pull-secret-syncer-v5x8q\" (UID: \"93168299-ac4f-4fdd-8ac3-b69edb385d71\") " pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:18:15.431201 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:15.431185    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 19:18:15.440218 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:15.440204    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93168299-ac4f-4fdd-8ac3-b69edb385d71-original-pull-secret\") pod \"global-pull-secret-syncer-v5x8q\" (UID: \"93168299-ac4f-4fdd-8ac3-b69edb385d71\") " pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:18:15.490321 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:15.490298    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5x8q"
Jul 29 19:18:15.528653 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:15.528626    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rx2m\" (UniqueName: \"kubernetes.io/projected/2677a483-66e1-42fd-94f5-3903b7af3690-kube-api-access-6rx2m\") pod \"network-check-target-tqpxs\" (UID: \"2677a483-66e1-42fd-94f5-3903b7af3690\") " pod="openshift-network-diagnostics/network-check-target-tqpxs"
Jul 29 19:18:15.531752 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:15.531731    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 19:18:15.542252 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:15.542029    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 19:18:15.552416 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:15.552369    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rx2m\" (UniqueName: \"kubernetes.io/projected/2677a483-66e1-42fd-94f5-3903b7af3690-kube-api-access-6rx2m\") pod \"network-check-target-tqpxs\" (UID: \"2677a483-66e1-42fd-94f5-3903b7af3690\") " pod="openshift-network-diagnostics/network-check-target-tqpxs"
Jul 29 19:18:15.626145 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:15.626116    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v5x8q"]
Jul 29 19:18:15.628824 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:18:15.628796    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93168299_ac4f_4fdd_8ac3_b69edb385d71.slice/crio-f546e4d0c2ccf11c50272527859d08ccd39bd3613b6cb62def44f36e1f91be3b WatchSource:0}: Error finding container f546e4d0c2ccf11c50272527859d08ccd39bd3613b6cb62def44f36e1f91be3b: Status 404 returned error can't find the container with id f546e4d0c2ccf11c50272527859d08ccd39bd3613b6cb62def44f36e1f91be3b
Jul 29 19:18:15.729813 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:15.729748    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bpq7l\" (UID: \"6e8a45f8-a6cf-4d48-b1cb-92a91e190c50\") " pod="openshift-insights/insights-runtime-extractor-bpq7l"
Jul 29 19:18:15.729922 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:15.729880    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 19:18:15.729981 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:15.729939    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-kube-rbac-proxy-cm podName:6e8a45f8-a6cf-4d48-b1cb-92a91e190c50 nodeName:}" failed. No retries permitted until 2026-07-29 19:18:17.729925026 +0000 UTC m=+68.557424062 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bpq7l" (UID: "6e8a45f8-a6cf-4d48-b1cb-92a91e190c50") : configmap "kube-rbac-proxy" not found
Jul 29 19:18:15.786478 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:15.786445    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bfbzc\""
Jul 29 19:18:15.793903 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:15.793884    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqpxs"
Jul 29 19:18:15.830599 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:15.830565    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bpq7l\" (UID: \"6e8a45f8-a6cf-4d48-b1cb-92a91e190c50\") " pod="openshift-insights/insights-runtime-extractor-bpq7l"
Jul 29 19:18:15.830742 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:15.830727    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 19:18:15.830808 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:15.830797    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-insights-runtime-extractor-tls podName:6e8a45f8-a6cf-4d48-b1cb-92a91e190c50 nodeName:}" failed. No retries permitted until 2026-07-29 19:18:17.830779633 +0000 UTC m=+68.658278669 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bpq7l" (UID: "6e8a45f8-a6cf-4d48-b1cb-92a91e190c50") : secret "insights-runtime-extractor-tls" not found
Jul 29 19:18:15.924470 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:15.924438    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tqpxs"]
Jul 29 19:18:15.927665 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:18:15.927634    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2677a483_66e1_42fd_94f5_3903b7af3690.slice/crio-eee0cdd3542ada4d44470344931dba30f838178745093d158df95e46b9e7979e WatchSource:0}: Error finding container eee0cdd3542ada4d44470344931dba30f838178745093d158df95e46b9e7979e: Status 404 returned error can't find the container with id eee0cdd3542ada4d44470344931dba30f838178745093d158df95e46b9e7979e
Jul 29 19:18:15.942168 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:15.942145    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-frvmp_d160081b-f780-4c60-85f9-04b9a6d1c3d9/node-ca/0.log"
Jul 29 19:18:16.088698 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:16.088616    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v5x8q" event={"ID":"93168299-ac4f-4fdd-8ac3-b69edb385d71","Type":"ContainerStarted","Data":"f546e4d0c2ccf11c50272527859d08ccd39bd3613b6cb62def44f36e1f91be3b"}
Jul 29 19:18:16.089673 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:16.089653    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tqpxs" event={"ID":"2677a483-66e1-42fd-94f5-3903b7af3690","Type":"ContainerStarted","Data":"eee0cdd3542ada4d44470344931dba30f838178745093d158df95e46b9e7979e"}
Jul 29 19:18:16.753886 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:16.753851    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-qcsv9"]
Jul 29 19:18:16.775280 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:16.775255    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-qcsv9"]
Jul 29 19:18:16.775424 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:16.775353    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-qcsv9"
Jul 29 19:18:16.778726 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:16.778696    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 29 19:18:16.778866 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:16.778745    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-65ll2\""
Jul 29 19:18:16.780121 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:16.780099    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 29 19:18:16.780121 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:16.780116    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 29 19:18:16.780271 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:16.780151    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 29 19:18:16.837640 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:16.837610    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c451a40b-fc19-46dd-aa1b-2331a3207b8f-signing-key\") pod \"service-ca-658d7b7c78-qcsv9\" (UID: \"c451a40b-fc19-46dd-aa1b-2331a3207b8f\") " pod="openshift-service-ca/service-ca-658d7b7c78-qcsv9"
Jul 29 19:18:16.837782 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:16.837701    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx552\" (UniqueName: \"kubernetes.io/projected/c451a40b-fc19-46dd-aa1b-2331a3207b8f-kube-api-access-rx552\") pod \"service-ca-658d7b7c78-qcsv9\" (UID: \"c451a40b-fc19-46dd-aa1b-2331a3207b8f\") " pod="openshift-service-ca/service-ca-658d7b7c78-qcsv9"
Jul 29 19:18:16.837858 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:16.837776    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/94a768fa-47eb-480c-85e8-fb69c78da295-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-kwjqt\" (UID: \"94a768fa-47eb-480c-85e8-fb69c78da295\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kwjqt"
Jul 29 19:18:16.837858 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:16.837824    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c451a40b-fc19-46dd-aa1b-2331a3207b8f-signing-cabundle\") pod \"service-ca-658d7b7c78-qcsv9\" (UID: \"c451a40b-fc19-46dd-aa1b-2331a3207b8f\") " pod="openshift-service-ca/service-ca-658d7b7c78-qcsv9"
Jul 29 19:18:16.837955 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:16.837877    2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 19:18:16.837955 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:16.837950    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94a768fa-47eb-480c-85e8-fb69c78da295-samples-operator-tls podName:94a768fa-47eb-480c-85e8-fb69c78da295 nodeName:}" failed. No retries permitted until 2026-07-29 19:18:24.837933393 +0000 UTC m=+75.665432429 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/94a768fa-47eb-480c-85e8-fb69c78da295-samples-operator-tls") pod "cluster-samples-operator-6748489656-kwjqt" (UID: "94a768fa-47eb-480c-85e8-fb69c78da295") : secret "samples-operator-tls" not found
Jul 29 19:18:16.938503 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:16.938468    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx552\" (UniqueName: \"kubernetes.io/projected/c451a40b-fc19-46dd-aa1b-2331a3207b8f-kube-api-access-rx552\") pod \"service-ca-658d7b7c78-qcsv9\" (UID: \"c451a40b-fc19-46dd-aa1b-2331a3207b8f\") " pod="openshift-service-ca/service-ca-658d7b7c78-qcsv9"
Jul 29 19:18:16.938678 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:16.938531    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c451a40b-fc19-46dd-aa1b-2331a3207b8f-signing-cabundle\") pod \"service-ca-658d7b7c78-qcsv9\" (UID: \"c451a40b-fc19-46dd-aa1b-2331a3207b8f\") " pod="openshift-service-ca/service-ca-658d7b7c78-qcsv9"
Jul 29 19:18:16.938678 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:16.938610    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c451a40b-fc19-46dd-aa1b-2331a3207b8f-signing-key\") pod \"service-ca-658d7b7c78-qcsv9\" (UID: \"c451a40b-fc19-46dd-aa1b-2331a3207b8f\") " pod="openshift-service-ca/service-ca-658d7b7c78-qcsv9"
Jul 29 19:18:16.938678 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:16.938642    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffafbd2c-ca03-47e9-ab0d-cd6125697e9e-metrics-tls\") pod \"dns-default-kq4zj\" (UID: \"ffafbd2c-ca03-47e9-ab0d-cd6125697e9e\") " pod="openshift-dns/dns-default-kq4zj"
Jul 29 19:18:16.938852 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:16.938744    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 19:18:16.938852 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:16.938806    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffafbd2c-ca03-47e9-ab0d-cd6125697e9e-metrics-tls podName:ffafbd2c-ca03-47e9-ab0d-cd6125697e9e nodeName:}" failed. No retries permitted until 2026-07-29 19:18:48.938788832 +0000 UTC m=+99.766287870 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ffafbd2c-ca03-47e9-ab0d-cd6125697e9e-metrics-tls") pod "dns-default-kq4zj" (UID: "ffafbd2c-ca03-47e9-ab0d-cd6125697e9e") : secret "dns-default-metrics-tls" not found
Jul 29 19:18:16.939061 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:16.939022    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-q724q_7266743b-b72b-41da-8110-b47af371be5c/serve-healthcheck-canary/0.log"
Jul 29 19:18:16.939411 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:16.939387    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c451a40b-fc19-46dd-aa1b-2331a3207b8f-signing-cabundle\") pod \"service-ca-658d7b7c78-qcsv9\" (UID: \"c451a40b-fc19-46dd-aa1b-2331a3207b8f\") " pod="openshift-service-ca/service-ca-658d7b7c78-qcsv9"
Jul 29 19:18:16.941790 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:16.941744    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c451a40b-fc19-46dd-aa1b-2331a3207b8f-signing-key\") pod \"service-ca-658d7b7c78-qcsv9\" (UID: \"c451a40b-fc19-46dd-aa1b-2331a3207b8f\") " pod="openshift-service-ca/service-ca-658d7b7c78-qcsv9"
Jul 29 19:18:16.947151 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:16.947121    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx552\" (UniqueName: \"kubernetes.io/projected/c451a40b-fc19-46dd-aa1b-2331a3207b8f-kube-api-access-rx552\") pod \"service-ca-658d7b7c78-qcsv9\" (UID: \"c451a40b-fc19-46dd-aa1b-2331a3207b8f\") " pod="openshift-service-ca/service-ca-658d7b7c78-qcsv9"
Jul 29 19:18:17.039303 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:17.039221    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/259710be-9c9b-4457-81c2-301f7d7cb110-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-dh7rf\" (UID: \"259710be-9c9b-4457-81c2-301f7d7cb110\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-dh7rf"
Jul 29 19:18:17.039466 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:17.039448    2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 19:18:17.039520 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:17.039511    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/259710be-9c9b-4457-81c2-301f7d7cb110-networking-console-plugin-cert podName:259710be-9c9b-4457-81c2-301f7d7cb110 nodeName:}" failed. No retries permitted until 2026-07-29 19:18:25.039497015 +0000 UTC m=+75.866996051 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/259710be-9c9b-4457-81c2-301f7d7cb110-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-dh7rf" (UID: "259710be-9c9b-4457-81c2-301f7d7cb110") : secret "networking-console-plugin-cert" not found
Jul 29 19:18:17.085979 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:17.085937    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-qcsv9"
Jul 29 19:18:17.241732 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:17.241698    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd24fa62-fd72-4e18-af46-6930625f22da-metrics-certs\") pod \"router-default-7fc6497669-xk7lp\" (UID: \"dd24fa62-fd72-4e18-af46-6930625f22da\") " pod="openshift-ingress/router-default-7fc6497669-xk7lp"
Jul 29 19:18:17.243328 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:17.241841    2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 29 19:18:17.243328 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:17.243019    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd24fa62-fd72-4e18-af46-6930625f22da-service-ca-bundle\") pod \"router-default-7fc6497669-xk7lp\" (UID: \"dd24fa62-fd72-4e18-af46-6930625f22da\") " pod="openshift-ingress/router-default-7fc6497669-xk7lp"
Jul 29 19:18:17.243328 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:17.243065    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd24fa62-fd72-4e18-af46-6930625f22da-metrics-certs podName:dd24fa62-fd72-4e18-af46-6930625f22da nodeName:}" failed. No retries permitted until 2026-07-29 19:18:25.243037832 +0000 UTC m=+76.070536889 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd24fa62-fd72-4e18-af46-6930625f22da-metrics-certs") pod "router-default-7fc6497669-xk7lp" (UID: "dd24fa62-fd72-4e18-af46-6930625f22da") : secret "router-metrics-certs-default" not found
Jul 29 19:18:17.243328 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:17.243123    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd24fa62-fd72-4e18-af46-6930625f22da-service-ca-bundle podName:dd24fa62-fd72-4e18-af46-6930625f22da nodeName:}" failed. No retries permitted until 2026-07-29 19:18:25.243112512 +0000 UTC m=+76.070611564 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dd24fa62-fd72-4e18-af46-6930625f22da-service-ca-bundle") pod "router-default-7fc6497669-xk7lp" (UID: "dd24fa62-fd72-4e18-af46-6930625f22da") : configmap references non-existent config key: service-ca.crt
Jul 29 19:18:17.243624 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:17.243476    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-qcsv9"]
Jul 29 19:18:17.248219 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:18:17.248190    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc451a40b_fc19_46dd_aa1b_2331a3207b8f.slice/crio-923fa502cd3a3301ccaf3eef16cb9000cab67b4e2cc39e421b0c49dde4f11c24 WatchSource:0}: Error finding container 923fa502cd3a3301ccaf3eef16cb9000cab67b4e2cc39e421b0c49dde4f11c24: Status 404 returned error can't find the container with id 923fa502cd3a3301ccaf3eef16cb9000cab67b4e2cc39e421b0c49dde4f11c24
Jul 29 19:18:17.749269 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:17.749236    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bpq7l\" (UID: \"6e8a45f8-a6cf-4d48-b1cb-92a91e190c50\") " pod="openshift-insights/insights-runtime-extractor-bpq7l"
Jul 29 19:18:17.749425 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:17.749374    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 19:18:17.749505 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:17.749453    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-kube-rbac-proxy-cm podName:6e8a45f8-a6cf-4d48-b1cb-92a91e190c50 nodeName:}" failed. No retries permitted until 2026-07-29 19:18:21.749433655 +0000 UTC m=+72.576932704 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bpq7l" (UID: "6e8a45f8-a6cf-4d48-b1cb-92a91e190c50") : configmap "kube-rbac-proxy" not found
Jul 29 19:18:17.850844 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:17.850776    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bpq7l\" (UID: \"6e8a45f8-a6cf-4d48-b1cb-92a91e190c50\") " pod="openshift-insights/insights-runtime-extractor-bpq7l"
Jul 29 19:18:17.851331 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:17.850939    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 19:18:17.851331 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:17.851036    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-insights-runtime-extractor-tls podName:6e8a45f8-a6cf-4d48-b1cb-92a91e190c50 nodeName:}" failed. No retries permitted until 2026-07-29 19:18:21.851013751 +0000 UTC m=+72.678512810 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bpq7l" (UID: "6e8a45f8-a6cf-4d48-b1cb-92a91e190c50") : secret "insights-runtime-extractor-tls" not found
Jul 29 19:18:18.096530 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:18.096435    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-qcsv9" event={"ID":"c451a40b-fc19-46dd-aa1b-2331a3207b8f","Type":"ContainerStarted","Data":"923fa502cd3a3301ccaf3eef16cb9000cab67b4e2cc39e421b0c49dde4f11c24"}
Jul 29 19:18:21.783274 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:21.783239    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bpq7l\" (UID: \"6e8a45f8-a6cf-4d48-b1cb-92a91e190c50\") " pod="openshift-insights/insights-runtime-extractor-bpq7l"
Jul 29 19:18:21.783634 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:21.783346    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 19:18:21.783634 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:21.783398    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-kube-rbac-proxy-cm podName:6e8a45f8-a6cf-4d48-b1cb-92a91e190c50 nodeName:}" failed. No retries permitted until 2026-07-29 19:18:29.783384374 +0000 UTC m=+80.610883409 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bpq7l" (UID: "6e8a45f8-a6cf-4d48-b1cb-92a91e190c50") : configmap "kube-rbac-proxy" not found
Jul 29 19:18:21.884246 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:21.884212    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bpq7l\" (UID: \"6e8a45f8-a6cf-4d48-b1cb-92a91e190c50\") " pod="openshift-insights/insights-runtime-extractor-bpq7l"
Jul 29 19:18:21.884393 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:21.884329    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 19:18:21.884393 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:21.884379    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-insights-runtime-extractor-tls podName:6e8a45f8-a6cf-4d48-b1cb-92a91e190c50 nodeName:}" failed. No retries permitted until 2026-07-29 19:18:29.884366438 +0000 UTC m=+80.711865473 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bpq7l" (UID: "6e8a45f8-a6cf-4d48-b1cb-92a91e190c50") : secret "insights-runtime-extractor-tls" not found
Jul 29 19:18:22.109187 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:22.109091    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tqpxs" event={"ID":"2677a483-66e1-42fd-94f5-3903b7af3690","Type":"ContainerStarted","Data":"ff695b895ceabe094d9e2aecdc61feb3a715f5ab3a3d5ba29174b52b3a5d843f"}
Jul 29 19:18:22.109187 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:22.109155    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tqpxs"
Jul 29 19:18:22.110665 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:22.110630    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-qcsv9" event={"ID":"c451a40b-fc19-46dd-aa1b-2331a3207b8f","Type":"ContainerStarted","Data":"d9367ab8aa53508fa463c262426a215c21ec14e2c10e80815bdba66b3e77bd5c"}
Jul 29 19:18:22.112046 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:22.112026    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v5x8q" event={"ID":"93168299-ac4f-4fdd-8ac3-b69edb385d71","Type":"ContainerStarted","Data":"6660c672136fd08f51dbea3e72b79343fb350e9573ff1b994ecaaf76f1541f04"}
Jul 29 19:18:22.130792 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:22.130742    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tqpxs" podStartSLOduration=66.859111798 podStartE2EDuration="1m12.130729351s" totalImagesPullingTime="5.271617553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:17:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:18:22.129872938 +0000 UTC m=+72.957371996" watchObservedRunningTime="2026-07-29 19:18:22.130729351 +0000 UTC m=+72.958228403"
Jul 29 19:18:22.147625 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:22.147584    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-qcsv9" podStartSLOduration=2.203544386 podStartE2EDuration="6.147568616s" totalImagesPullingTime="3.94402423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:18:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:18:22.147330961 +0000 UTC m=+72.974830020" watchObservedRunningTime="2026-07-29 19:18:22.147568616 +0000 UTC m=+72.975067676"
Jul 29 19:18:22.163727 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:22.163689    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-v5x8q" podStartSLOduration=66.60491507 podStartE2EDuration="1m12.163676511s" totalImagesPullingTime="5.558761441s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:17:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:18:22.163124469 +0000 UTC m=+72.990623543" watchObservedRunningTime="2026-07-29 19:18:22.163676511 +0000 UTC m=+72.991175568"
Jul 29 19:18:24.913110 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:24.913077    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/94a768fa-47eb-480c-85e8-fb69c78da295-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-kwjqt\" (UID: \"94a768fa-47eb-480c-85e8-fb69c78da295\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kwjqt"
Jul 29 19:18:24.915443 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:24.915418    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/94a768fa-47eb-480c-85e8-fb69c78da295-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-kwjqt\" (UID: \"94a768fa-47eb-480c-85e8-fb69c78da295\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kwjqt"
Jul 29 19:18:25.070762 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:25.070724    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-dmgdk\""
Jul 29 19:18:25.078446 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:25.078420    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kwjqt"
Jul 29 19:18:25.114774 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:25.114736    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/259710be-9c9b-4457-81c2-301f7d7cb110-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-dh7rf\" (UID: \"259710be-9c9b-4457-81c2-301f7d7cb110\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-dh7rf"
Jul 29 19:18:25.114916 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:25.114878    2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 19:18:25.114956 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:25.114946    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/259710be-9c9b-4457-81c2-301f7d7cb110-networking-console-plugin-cert podName:259710be-9c9b-4457-81c2-301f7d7cb110 nodeName:}" failed. No retries permitted until 2026-07-29 19:18:41.11493094 +0000 UTC m=+91.942429975 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/259710be-9c9b-4457-81c2-301f7d7cb110-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-dh7rf" (UID: "259710be-9c9b-4457-81c2-301f7d7cb110") : secret "networking-console-plugin-cert" not found
Jul 29 19:18:25.208056 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:25.207985    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kwjqt"]
Jul 29 19:18:25.316740 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:25.316703    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd24fa62-fd72-4e18-af46-6930625f22da-metrics-certs\") pod \"router-default-7fc6497669-xk7lp\" (UID: \"dd24fa62-fd72-4e18-af46-6930625f22da\") " pod="openshift-ingress/router-default-7fc6497669-xk7lp"
Jul 29 19:18:25.316896 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:25.316755    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd24fa62-fd72-4e18-af46-6930625f22da-service-ca-bundle\") pod \"router-default-7fc6497669-xk7lp\" (UID: \"dd24fa62-fd72-4e18-af46-6930625f22da\") " pod="openshift-ingress/router-default-7fc6497669-xk7lp"
Jul 29 19:18:25.316896 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:25.316873    2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 29 19:18:25.316995 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:25.316878    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd24fa62-fd72-4e18-af46-6930625f22da-service-ca-bundle podName:dd24fa62-fd72-4e18-af46-6930625f22da nodeName:}" failed. No retries permitted until 2026-07-29 19:18:41.316863438 +0000 UTC m=+92.144362486 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dd24fa62-fd72-4e18-af46-6930625f22da-service-ca-bundle") pod "router-default-7fc6497669-xk7lp" (UID: "dd24fa62-fd72-4e18-af46-6930625f22da") : configmap references non-existent config key: service-ca.crt
Jul 29 19:18:25.316995 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:25.316951    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd24fa62-fd72-4e18-af46-6930625f22da-metrics-certs podName:dd24fa62-fd72-4e18-af46-6930625f22da nodeName:}" failed. No retries permitted until 2026-07-29 19:18:41.316933364 +0000 UTC m=+92.144432406 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd24fa62-fd72-4e18-af46-6930625f22da-metrics-certs") pod "router-default-7fc6497669-xk7lp" (UID: "dd24fa62-fd72-4e18-af46-6930625f22da") : secret "router-metrics-certs-default" not found
Jul 29 19:18:26.124584 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:26.124544    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kwjqt" event={"ID":"94a768fa-47eb-480c-85e8-fb69c78da295","Type":"ContainerStarted","Data":"b5ea3353ef9547afd6b4d5af6062607486c967e2aae071ace941d68cb49d9ebe"}
Jul 29 19:18:27.129109 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:27.129066    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kwjqt" event={"ID":"94a768fa-47eb-480c-85e8-fb69c78da295","Type":"ContainerStarted","Data":"8648757b03819dddb783dd1cc6468a1d81eac8343dd5923792863b0f71dfa1c3"}
Jul 29 19:18:28.136923 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:28.136887    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kwjqt" event={"ID":"94a768fa-47eb-480c-85e8-fb69c78da295","Type":"ContainerStarted","Data":"827b67cd304a4f23943c90c5258892d9d12aa743b7dfc551a2c741f4bb8cd43a"}
Jul 29 19:18:28.155822 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:28.155776    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-kwjqt" podStartSLOduration=17.363558639 podStartE2EDuration="19.155762855s" totalImagesPullingTime="1.792204216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:18:28.155457005 +0000 UTC m=+78.982956064" watchObservedRunningTime="2026-07-29 19:18:28.155762855 +0000 UTC m=+78.983261913"
Jul 29 19:18:29.853684 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:29.853650    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bpq7l\" (UID: \"6e8a45f8-a6cf-4d48-b1cb-92a91e190c50\") " pod="openshift-insights/insights-runtime-extractor-bpq7l"
Jul 29 19:18:29.854177 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:29.854114    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bpq7l\" (UID: \"6e8a45f8-a6cf-4d48-b1cb-92a91e190c50\") " pod="openshift-insights/insights-runtime-extractor-bpq7l"
Jul 29 19:18:29.954570 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:29.954538    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bpq7l\" (UID: \"6e8a45f8-a6cf-4d48-b1cb-92a91e190c50\") " pod="openshift-insights/insights-runtime-extractor-bpq7l"
Jul 29 19:18:29.954734 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:29.954700    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 19:18:29.954785 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:29.954774    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-insights-runtime-extractor-tls podName:6e8a45f8-a6cf-4d48-b1cb-92a91e190c50 nodeName:}" failed. No retries permitted until 2026-07-29 19:18:45.954758432 +0000 UTC m=+96.782257468 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bpq7l" (UID: "6e8a45f8-a6cf-4d48-b1cb-92a91e190c50") : secret "insights-runtime-extractor-tls" not found
Jul 29 19:18:39.609205 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:39.609094    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7544fd9566-nbrt9"]
Jul 29 19:18:39.613508 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:39.613488    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7544fd9566-nbrt9"
Jul 29 19:18:39.617449 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:39.617422    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-tpg4n\""
Jul 29 19:18:39.617449 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:39.617439    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 29 19:18:39.617449 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:39.617422    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 29 19:18:39.617667 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:39.617439    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 29 19:18:39.617667 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:39.617526    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 29 19:18:39.625747 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:39.625727    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7544fd9566-nbrt9"]
Jul 29 19:18:39.712080 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:39.712054    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-84drj"]
Jul 29 19:18:39.714914 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:39.714897    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-84drj"
Jul 29 19:18:39.718138 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:39.718121    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-mh88n\""
Jul 29 19:18:39.718302 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:39.718290    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 29 19:18:39.720305 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:39.720288    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 29 19:18:39.732361 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:39.732337    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-84drj"]
Jul 29 19:18:39.734658 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:39.734635    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2xwl\" (UniqueName: \"kubernetes.io/projected/1015f945-2ed3-4b75-b6f6-c9a2d6870a3a-kube-api-access-k2xwl\") pod \"managed-serviceaccount-addon-agent-7544fd9566-nbrt9\" (UID: \"1015f945-2ed3-4b75-b6f6-c9a2d6870a3a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7544fd9566-nbrt9"
Jul 29 19:18:39.734759 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:39.734689    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1015f945-2ed3-4b75-b6f6-c9a2d6870a3a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7544fd9566-nbrt9\" (UID: \"1015f945-2ed3-4b75-b6f6-c9a2d6870a3a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7544fd9566-nbrt9"
Jul 29 19:18:39.835763 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:39.835737    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9xns\" (UniqueName: \"kubernetes.io/projected/b92baea5-f190-4e2d-9e6e-2fcfc1a61176-kube-api-access-v9xns\") pod \"downloads-7c97dc4555-84drj\" (UID: \"b92baea5-f190-4e2d-9e6e-2fcfc1a61176\") " pod="openshift-console/downloads-7c97dc4555-84drj"
Jul 29 19:18:39.835917 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:39.835784    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2xwl\" (UniqueName: \"kubernetes.io/projected/1015f945-2ed3-4b75-b6f6-c9a2d6870a3a-kube-api-access-k2xwl\") pod \"managed-serviceaccount-addon-agent-7544fd9566-nbrt9\" (UID: \"1015f945-2ed3-4b75-b6f6-c9a2d6870a3a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7544fd9566-nbrt9"
Jul 29 19:18:39.835917 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:39.835845    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1015f945-2ed3-4b75-b6f6-c9a2d6870a3a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7544fd9566-nbrt9\" (UID: \"1015f945-2ed3-4b75-b6f6-c9a2d6870a3a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7544fd9566-nbrt9"
Jul 29 19:18:39.838333 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:39.838311    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1015f945-2ed3-4b75-b6f6-c9a2d6870a3a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7544fd9566-nbrt9\" (UID: \"1015f945-2ed3-4b75-b6f6-c9a2d6870a3a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7544fd9566-nbrt9"
Jul 29 19:18:39.845784 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:39.845762    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2xwl\" (UniqueName: \"kubernetes.io/projected/1015f945-2ed3-4b75-b6f6-c9a2d6870a3a-kube-api-access-k2xwl\") pod \"managed-serviceaccount-addon-agent-7544fd9566-nbrt9\" (UID: \"1015f945-2ed3-4b75-b6f6-c9a2d6870a3a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7544fd9566-nbrt9"
Jul 29 19:18:39.934174 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:39.934124    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7544fd9566-nbrt9"
Jul 29 19:18:39.936821 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:39.936454    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9xns\" (UniqueName: \"kubernetes.io/projected/b92baea5-f190-4e2d-9e6e-2fcfc1a61176-kube-api-access-v9xns\") pod \"downloads-7c97dc4555-84drj\" (UID: \"b92baea5-f190-4e2d-9e6e-2fcfc1a61176\") " pod="openshift-console/downloads-7c97dc4555-84drj"
Jul 29 19:18:39.945012 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:39.944986    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9xns\" (UniqueName: \"kubernetes.io/projected/b92baea5-f190-4e2d-9e6e-2fcfc1a61176-kube-api-access-v9xns\") pod \"downloads-7c97dc4555-84drj\" (UID: \"b92baea5-f190-4e2d-9e6e-2fcfc1a61176\") " pod="openshift-console/downloads-7c97dc4555-84drj"
Jul 29 19:18:40.023371 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:40.023345    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-84drj"
Jul 29 19:18:40.057771 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:40.057734    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7544fd9566-nbrt9"]
Jul 29 19:18:40.061117 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:18:40.061081    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1015f945_2ed3_4b75_b6f6_c9a2d6870a3a.slice/crio-edf725d7605e75e6e238908104783709fe7e89ee8461f63a2ca9c5beef783992 WatchSource:0}: Error finding container edf725d7605e75e6e238908104783709fe7e89ee8461f63a2ca9c5beef783992: Status 404 returned error can't find the container with id edf725d7605e75e6e238908104783709fe7e89ee8461f63a2ca9c5beef783992
Jul 29 19:18:40.147361 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:40.147332    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-84drj"]
Jul 29 19:18:40.150498 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:18:40.150470    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb92baea5_f190_4e2d_9e6e_2fcfc1a61176.slice/crio-88a1938bf5969351ec38d1149afcf273ffdf0de6b2ef43131ddd0e8933f84c12 WatchSource:0}: Error finding container 88a1938bf5969351ec38d1149afcf273ffdf0de6b2ef43131ddd0e8933f84c12: Status 404 returned error can't find the container with id 88a1938bf5969351ec38d1149afcf273ffdf0de6b2ef43131ddd0e8933f84c12
Jul 29 19:18:40.175282 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:40.175250    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7544fd9566-nbrt9" event={"ID":"1015f945-2ed3-4b75-b6f6-c9a2d6870a3a","Type":"ContainerStarted","Data":"edf725d7605e75e6e238908104783709fe7e89ee8461f63a2ca9c5beef783992"}
Jul 29 19:18:40.176303 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:40.176283    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-84drj" event={"ID":"b92baea5-f190-4e2d-9e6e-2fcfc1a61176","Type":"ContainerStarted","Data":"88a1938bf5969351ec38d1149afcf273ffdf0de6b2ef43131ddd0e8933f84c12"}
Jul 29 19:18:41.148667 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:41.148629    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/259710be-9c9b-4457-81c2-301f7d7cb110-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-dh7rf\" (UID: \"259710be-9c9b-4457-81c2-301f7d7cb110\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-dh7rf"
Jul 29 19:18:41.160147 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:41.160113    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/259710be-9c9b-4457-81c2-301f7d7cb110-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-dh7rf\" (UID: \"259710be-9c9b-4457-81c2-301f7d7cb110\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-dh7rf"
Jul 29 19:18:41.173760 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:41.173735    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-wmf52\""
Jul 29 19:18:41.180997 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:41.180775    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-dh7rf"
Jul 29 19:18:41.351047 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:41.351012    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd24fa62-fd72-4e18-af46-6930625f22da-metrics-certs\") pod \"router-default-7fc6497669-xk7lp\" (UID: \"dd24fa62-fd72-4e18-af46-6930625f22da\") " pod="openshift-ingress/router-default-7fc6497669-xk7lp"
Jul 29 19:18:41.351226 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:41.351080    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd24fa62-fd72-4e18-af46-6930625f22da-service-ca-bundle\") pod \"router-default-7fc6497669-xk7lp\" (UID: \"dd24fa62-fd72-4e18-af46-6930625f22da\") " pod="openshift-ingress/router-default-7fc6497669-xk7lp"
Jul 29 19:18:41.352528 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:41.352501    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd24fa62-fd72-4e18-af46-6930625f22da-service-ca-bundle\") pod \"router-default-7fc6497669-xk7lp\" (UID: \"dd24fa62-fd72-4e18-af46-6930625f22da\") " pod="openshift-ingress/router-default-7fc6497669-xk7lp"
Jul 29 19:18:41.355731 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:41.355650    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd24fa62-fd72-4e18-af46-6930625f22da-metrics-certs\") pod \"router-default-7fc6497669-xk7lp\" (UID: \"dd24fa62-fd72-4e18-af46-6930625f22da\") " pod="openshift-ingress/router-default-7fc6497669-xk7lp"
Jul 29 19:18:41.383345 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:41.383281    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-dh7rf"]
Jul 29 19:18:41.386988 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:18:41.386936    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod259710be_9c9b_4457_81c2_301f7d7cb110.slice/crio-4117f67b9b84a1fee09255582f443b723afb1b17fd65127d354fb422682d0705 WatchSource:0}: Error finding container 4117f67b9b84a1fee09255582f443b723afb1b17fd65127d354fb422682d0705: Status 404 returned error can't find the container with id 4117f67b9b84a1fee09255582f443b723afb1b17fd65127d354fb422682d0705
Jul 29 19:18:41.485738 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:41.485651    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-znt5r\""
Jul 29 19:18:41.494200 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:41.494109    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7fc6497669-xk7lp"
Jul 29 19:18:41.649265 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:41.649233    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7fc6497669-xk7lp"]
Jul 29 19:18:41.652373 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:18:41.652343    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd24fa62_fd72_4e18_af46_6930625f22da.slice/crio-8b17d962c415a86cadf375366dcf9771c154cfef8e560ac67f50d29c6fe3dc89 WatchSource:0}: Error finding container 8b17d962c415a86cadf375366dcf9771c154cfef8e560ac67f50d29c6fe3dc89: Status 404 returned error can't find the container with id 8b17d962c415a86cadf375366dcf9771c154cfef8e560ac67f50d29c6fe3dc89
Jul 29 19:18:42.186119 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:42.186007    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-dh7rf" event={"ID":"259710be-9c9b-4457-81c2-301f7d7cb110","Type":"ContainerStarted","Data":"4117f67b9b84a1fee09255582f443b723afb1b17fd65127d354fb422682d0705"}
Jul 29 19:18:42.188338 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:42.188309    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7fc6497669-xk7lp" event={"ID":"dd24fa62-fd72-4e18-af46-6930625f22da","Type":"ContainerStarted","Data":"d51ccaba31672ea6a412d0498c0b503630a6e1ef3bd2d633960c17a4f6699f1b"}
Jul 29 19:18:42.188462 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:42.188355    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7fc6497669-xk7lp" event={"ID":"dd24fa62-fd72-4e18-af46-6930625f22da","Type":"ContainerStarted","Data":"8b17d962c415a86cadf375366dcf9771c154cfef8e560ac67f50d29c6fe3dc89"}
Jul 29 19:18:42.210395 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:42.210353    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7fc6497669-xk7lp" podStartSLOduration=33.21033679 podStartE2EDuration="33.21033679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:18:42.209364992 +0000 UTC m=+93.036864051" watchObservedRunningTime="2026-07-29 19:18:42.21033679 +0000 UTC m=+93.037836047"
Jul 29 19:18:42.494862 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:42.494740    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7fc6497669-xk7lp"
Jul 29 19:18:42.497370 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:42.497344    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7fc6497669-xk7lp"
Jul 29 19:18:43.192544 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:43.192506    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7fc6497669-xk7lp"
Jul 29 19:18:43.194044 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:43.193995    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7fc6497669-xk7lp"
Jul 29 19:18:44.197758 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:44.197719    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7544fd9566-nbrt9" event={"ID":"1015f945-2ed3-4b75-b6f6-c9a2d6870a3a","Type":"ContainerStarted","Data":"a1534ee708fbba1d295881e5302f3208013ae318d2e60c7e0fa5701855635951"}
Jul 29 19:18:44.199603 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:44.199576    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-dh7rf" event={"ID":"259710be-9c9b-4457-81c2-301f7d7cb110","Type":"ContainerStarted","Data":"2d5086d3d4c589d98bd75bf6a7dea1f8643976615284157719638079650fe904"}
Jul 29 19:18:44.213507 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:44.213451    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7544fd9566-nbrt9" podStartSLOduration=1.860005446 podStartE2EDuration="5.213438673s" totalImagesPullingTime="3.353433227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:18:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:18:44.213223436 +0000 UTC m=+95.040722495" watchObservedRunningTime="2026-07-29 19:18:44.213438673 +0000 UTC m=+95.040937731"
Jul 29 19:18:44.231663 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:44.231621    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-dh7rf" podStartSLOduration=33.165906252 podStartE2EDuration="35.231606714s" totalImagesPullingTime="2.065700462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:18:44.230579689 +0000 UTC m=+95.058078746" watchObservedRunningTime="2026-07-29 19:18:44.231606714 +0000 UTC m=+95.059105772"
Jul 29 19:18:45.996086 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:45.996046    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bpq7l\" (UID: \"6e8a45f8-a6cf-4d48-b1cb-92a91e190c50\") " pod="openshift-insights/insights-runtime-extractor-bpq7l"
Jul 29 19:18:45.998698 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:45.998671    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e8a45f8-a6cf-4d48-b1cb-92a91e190c50-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bpq7l\" (UID: \"6e8a45f8-a6cf-4d48-b1cb-92a91e190c50\") " pod="openshift-insights/insights-runtime-extractor-bpq7l"
Jul 29 19:18:46.098927 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:46.098893    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bpq7l"
Jul 29 19:18:46.255261 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:46.255232    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bpq7l"]
Jul 29 19:18:46.257872 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:18:46.257842    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e8a45f8_a6cf_4d48_b1cb_92a91e190c50.slice/crio-b56feb70bd81dd0bc3b89b99ab511864cf1e9d47da0cc0030a23897885c0be43 WatchSource:0}: Error finding container b56feb70bd81dd0bc3b89b99ab511864cf1e9d47da0cc0030a23897885c0be43: Status 404 returned error can't find the container with id b56feb70bd81dd0bc3b89b99ab511864cf1e9d47da0cc0030a23897885c0be43
Jul 29 19:18:47.212366 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:47.212328    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bpq7l" event={"ID":"6e8a45f8-a6cf-4d48-b1cb-92a91e190c50","Type":"ContainerStarted","Data":"b56feb70bd81dd0bc3b89b99ab511864cf1e9d47da0cc0030a23897885c0be43"}
Jul 29 19:18:49.021330 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:49.021227    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffafbd2c-ca03-47e9-ab0d-cd6125697e9e-metrics-tls\") pod \"dns-default-kq4zj\" (UID: \"ffafbd2c-ca03-47e9-ab0d-cd6125697e9e\") " pod="openshift-dns/dns-default-kq4zj"
Jul 29 19:18:49.023954 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:49.023870    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffafbd2c-ca03-47e9-ab0d-cd6125697e9e-metrics-tls\") pod \"dns-default-kq4zj\" (UID: \"ffafbd2c-ca03-47e9-ab0d-cd6125697e9e\") " pod="openshift-dns/dns-default-kq4zj"
Jul 29 19:18:49.248345 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:49.248126    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-74g8d\""
Jul 29 19:18:49.255426 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:49.255366    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kq4zj"
Jul 29 19:18:50.684500 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:50.684467    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-frhw9"]
Jul 29 19:18:50.713805 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:50.713768    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-frhw9"]
Jul 29 19:18:50.713989 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:50.713901    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-frhw9"
Jul 29 19:18:50.717005 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:50.716974    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 29 19:18:50.717005 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:50.716982    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 29 19:18:50.717956 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:50.717902    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 29 19:18:50.718371 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:50.718011    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-wp9d9\""
Jul 29 19:18:50.718371 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:50.718128    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 29 19:18:50.718371 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:50.718161    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 29 19:18:50.736180 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:50.736155    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d7bc168b-7b7c-4242-b505-92500fd9a3a3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-frhw9\" (UID: \"d7bc168b-7b7c-4242-b505-92500fd9a3a3\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-frhw9"
Jul 29 19:18:50.736314 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:50.736217    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7bc168b-7b7c-4242-b505-92500fd9a3a3-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-frhw9\" (UID: \"d7bc168b-7b7c-4242-b505-92500fd9a3a3\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-frhw9"
Jul 29 19:18:50.736314 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:50.736247    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7bc168b-7b7c-4242-b505-92500fd9a3a3-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-frhw9\" (UID: \"d7bc168b-7b7c-4242-b505-92500fd9a3a3\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-frhw9"
Jul 29 19:18:50.736428 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:50.736347    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtlf8\" (UniqueName: \"kubernetes.io/projected/d7bc168b-7b7c-4242-b505-92500fd9a3a3-kube-api-access-qtlf8\") pod \"prometheus-operator-84d69f794d-frhw9\" (UID: \"d7bc168b-7b7c-4242-b505-92500fd9a3a3\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-frhw9"
Jul 29 19:18:50.836863 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:50.836830    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d7bc168b-7b7c-4242-b505-92500fd9a3a3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-frhw9\" (UID: \"d7bc168b-7b7c-4242-b505-92500fd9a3a3\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-frhw9"
Jul 29 19:18:50.837046 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:50.836891    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7bc168b-7b7c-4242-b505-92500fd9a3a3-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-frhw9\" (UID: \"d7bc168b-7b7c-4242-b505-92500fd9a3a3\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-frhw9"
Jul 29 19:18:50.837046 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:50.836920    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7bc168b-7b7c-4242-b505-92500fd9a3a3-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-frhw9\" (UID: \"d7bc168b-7b7c-4242-b505-92500fd9a3a3\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-frhw9"
Jul 29 19:18:50.837046 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:50.836986    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtlf8\" (UniqueName: \"kubernetes.io/projected/d7bc168b-7b7c-4242-b505-92500fd9a3a3-kube-api-access-qtlf8\") pod \"prometheus-operator-84d69f794d-frhw9\" (UID: \"d7bc168b-7b7c-4242-b505-92500fd9a3a3\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-frhw9"
Jul 29 19:18:50.837239 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:50.837070    2557 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 29 19:18:50.837239 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:18:50.837153    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7bc168b-7b7c-4242-b505-92500fd9a3a3-prometheus-operator-tls podName:d7bc168b-7b7c-4242-b505-92500fd9a3a3 nodeName:}" failed. No retries permitted until 2026-07-29 19:18:51.337133198 +0000 UTC m=+102.164632238 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/d7bc168b-7b7c-4242-b505-92500fd9a3a3-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-frhw9" (UID: "d7bc168b-7b7c-4242-b505-92500fd9a3a3") : secret "prometheus-operator-tls" not found
Jul 29 19:18:50.837712 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:50.837694    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7bc168b-7b7c-4242-b505-92500fd9a3a3-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-frhw9\" (UID: \"d7bc168b-7b7c-4242-b505-92500fd9a3a3\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-frhw9"
Jul 29 19:18:50.839618 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:50.839594    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d7bc168b-7b7c-4242-b505-92500fd9a3a3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-frhw9\" (UID: \"d7bc168b-7b7c-4242-b505-92500fd9a3a3\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-frhw9"
Jul 29 19:18:50.846387 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:50.846359    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtlf8\" (UniqueName: \"kubernetes.io/projected/d7bc168b-7b7c-4242-b505-92500fd9a3a3-kube-api-access-qtlf8\") pod \"prometheus-operator-84d69f794d-frhw9\" (UID: \"d7bc168b-7b7c-4242-b505-92500fd9a3a3\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-frhw9"
Jul 29 19:18:51.341784 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:51.341734    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7bc168b-7b7c-4242-b505-92500fd9a3a3-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-frhw9\" (UID: \"d7bc168b-7b7c-4242-b505-92500fd9a3a3\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-frhw9"
Jul 29 19:18:51.344663 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:51.344635    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7bc168b-7b7c-4242-b505-92500fd9a3a3-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-frhw9\" (UID: \"d7bc168b-7b7c-4242-b505-92500fd9a3a3\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-frhw9"
Jul 29 19:18:51.623722 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:51.623636    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-frhw9"
Jul 29 19:18:53.118834 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:53.118791    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tqpxs"
Jul 29 19:18:54.400269 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:54.400230    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-978bbdbf5-kp5lc"]
Jul 29 19:18:54.414393 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:54.414351    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-978bbdbf5-kp5lc"]
Jul 29 19:18:54.414546 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:54.414460    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-978bbdbf5-kp5lc"
Jul 29 19:18:54.417397 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:54.417372    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 29 19:18:54.417397 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:54.417394    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 29 19:18:54.417587 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:54.417443    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 29 19:18:54.417756 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:54.417727    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 29 19:18:54.418406 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:54.418382    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-v8zkg\""
Jul 29 19:18:54.418406 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:54.418396    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 29 19:18:54.422599 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:54.422580    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 29 19:18:54.467474 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:54.467442    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/692df9a1-9ac3-414c-89e3-2e414f8b0599-oauth-serving-cert\") pod \"console-978bbdbf5-kp5lc\" (UID: \"692df9a1-9ac3-414c-89e3-2e414f8b0599\") " pod="openshift-console/console-978bbdbf5-kp5lc"
Jul 29 19:18:54.467605 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:54.467495    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/692df9a1-9ac3-414c-89e3-2e414f8b0599-console-serving-cert\") pod \"console-978bbdbf5-kp5lc\" (UID: \"692df9a1-9ac3-414c-89e3-2e414f8b0599\") " pod="openshift-console/console-978bbdbf5-kp5lc"
Jul 29 19:18:54.467605 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:54.467523    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/692df9a1-9ac3-414c-89e3-2e414f8b0599-service-ca\") pod \"console-978bbdbf5-kp5lc\" (UID: \"692df9a1-9ac3-414c-89e3-2e414f8b0599\") " pod="openshift-console/console-978bbdbf5-kp5lc"
Jul 29 19:18:54.467704 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:54.467598    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-274sd\" (UniqueName: \"kubernetes.io/projected/692df9a1-9ac3-414c-89e3-2e414f8b0599-kube-api-access-274sd\") pod \"console-978bbdbf5-kp5lc\" (UID: \"692df9a1-9ac3-414c-89e3-2e414f8b0599\") " pod="openshift-console/console-978bbdbf5-kp5lc"
Jul 29 19:18:54.467704 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:54.467649    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/692df9a1-9ac3-414c-89e3-2e414f8b0599-trusted-ca-bundle\") pod \"console-978bbdbf5-kp5lc\" (UID: \"692df9a1-9ac3-414c-89e3-2e414f8b0599\") " pod="openshift-console/console-978bbdbf5-kp5lc"
Jul 29 19:18:54.467801 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:54.467715    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/692df9a1-9ac3-414c-89e3-2e414f8b0599-console-config\") pod \"console-978bbdbf5-kp5lc\" (UID: \"692df9a1-9ac3-414c-89e3-2e414f8b0599\") " pod="openshift-console/console-978bbdbf5-kp5lc"
Jul 29 19:18:54.467801 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:54.467775    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/692df9a1-9ac3-414c-89e3-2e414f8b0599-console-oauth-config\") pod \"console-978bbdbf5-kp5lc\" (UID: \"692df9a1-9ac3-414c-89e3-2e414f8b0599\") " pod="openshift-console/console-978bbdbf5-kp5lc"
Jul 29 19:18:54.568509 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:54.568470    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/692df9a1-9ac3-414c-89e3-2e414f8b0599-console-oauth-config\") pod \"console-978bbdbf5-kp5lc\" (UID: \"692df9a1-9ac3-414c-89e3-2e414f8b0599\") " pod="openshift-console/console-978bbdbf5-kp5lc"
Jul 29 19:18:54.568686 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:54.568531    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/692df9a1-9ac3-414c-89e3-2e414f8b0599-oauth-serving-cert\") pod \"console-978bbdbf5-kp5lc\" (UID: \"692df9a1-9ac3-414c-89e3-2e414f8b0599\") " pod="openshift-console/console-978bbdbf5-kp5lc"
Jul 29 19:18:54.568686 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:54.568568    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/692df9a1-9ac3-414c-89e3-2e414f8b0599-console-serving-cert\") pod \"console-978bbdbf5-kp5lc\" (UID: \"692df9a1-9ac3-414c-89e3-2e414f8b0599\") " pod="openshift-console/console-978bbdbf5-kp5lc"
Jul 29 19:18:54.568686 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:54.568593    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/692df9a1-9ac3-414c-89e3-2e414f8b0599-service-ca\") pod \"console-978bbdbf5-kp5lc\" (UID: \"692df9a1-9ac3-414c-89e3-2e414f8b0599\") " pod="openshift-console/console-978bbdbf5-kp5lc"
Jul 29 19:18:54.568686 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:54.568629    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-274sd\" (UniqueName: \"kubernetes.io/projected/692df9a1-9ac3-414c-89e3-2e414f8b0599-kube-api-access-274sd\") pod \"console-978bbdbf5-kp5lc\" (UID: \"692df9a1-9ac3-414c-89e3-2e414f8b0599\") " pod="openshift-console/console-978bbdbf5-kp5lc"
Jul 29 19:18:54.568686 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:54.568672    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/692df9a1-9ac3-414c-89e3-2e414f8b0599-trusted-ca-bundle\") pod \"console-978bbdbf5-kp5lc\" (UID: \"692df9a1-9ac3-414c-89e3-2e414f8b0599\") " pod="openshift-console/console-978bbdbf5-kp5lc"
Jul 29 19:18:54.568947 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:54.568703    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/692df9a1-9ac3-414c-89e3-2e414f8b0599-console-config\") pod \"console-978bbdbf5-kp5lc\" (UID: \"692df9a1-9ac3-414c-89e3-2e414f8b0599\") " pod="openshift-console/console-978bbdbf5-kp5lc"
Jul 29 19:18:54.569396 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:54.569343    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/692df9a1-9ac3-414c-89e3-2e414f8b0599-service-ca\") pod \"console-978bbdbf5-kp5lc\" (UID: \"692df9a1-9ac3-414c-89e3-2e414f8b0599\") " pod="openshift-console/console-978bbdbf5-kp5lc"
Jul 29 19:18:54.569396 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:54.569360    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/692df9a1-9ac3-414c-89e3-2e414f8b0599-oauth-serving-cert\") pod \"console-978bbdbf5-kp5lc\" (UID: \"692df9a1-9ac3-414c-89e3-2e414f8b0599\") " pod="openshift-console/console-978bbdbf5-kp5lc"
Jul 29 19:18:54.569578 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:54.569441    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/692df9a1-9ac3-414c-89e3-2e414f8b0599-console-config\") pod \"console-978bbdbf5-kp5lc\" (UID: \"692df9a1-9ac3-414c-89e3-2e414f8b0599\") " pod="openshift-console/console-978bbdbf5-kp5lc"
Jul 29 19:18:54.569719 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:54.569697    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/692df9a1-9ac3-414c-89e3-2e414f8b0599-trusted-ca-bundle\") pod \"console-978bbdbf5-kp5lc\" (UID: \"692df9a1-9ac3-414c-89e3-2e414f8b0599\") " pod="openshift-console/console-978bbdbf5-kp5lc"
Jul 29 19:18:54.571456 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:54.571407    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/692df9a1-9ac3-414c-89e3-2e414f8b0599-console-serving-cert\") pod \"console-978bbdbf5-kp5lc\" (UID: \"692df9a1-9ac3-414c-89e3-2e414f8b0599\") " pod="openshift-console/console-978bbdbf5-kp5lc"
Jul 29 19:18:54.571620 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:54.571601    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/692df9a1-9ac3-414c-89e3-2e414f8b0599-console-oauth-config\") pod \"console-978bbdbf5-kp5lc\" (UID: \"692df9a1-9ac3-414c-89e3-2e414f8b0599\") " pod="openshift-console/console-978bbdbf5-kp5lc"
Jul 29 19:18:54.577051 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:54.577008    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-274sd\" (UniqueName: \"kubernetes.io/projected/692df9a1-9ac3-414c-89e3-2e414f8b0599-kube-api-access-274sd\") pod \"console-978bbdbf5-kp5lc\" (UID: \"692df9a1-9ac3-414c-89e3-2e414f8b0599\") " pod="openshift-console/console-978bbdbf5-kp5lc"
Jul 29 19:18:54.726159 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:54.726074    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-978bbdbf5-kp5lc"
Jul 29 19:18:58.579844 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:58.579762    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kq4zj"]
Jul 29 19:18:58.584784 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:18:58.584753    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffafbd2c_ca03_47e9_ab0d_cd6125697e9e.slice/crio-42034a3ff3ad581d4901ab89646716ce770a83180029eac93ad2c4a698df13a0 WatchSource:0}: Error finding container 42034a3ff3ad581d4901ab89646716ce770a83180029eac93ad2c4a698df13a0: Status 404 returned error can't find the container with id 42034a3ff3ad581d4901ab89646716ce770a83180029eac93ad2c4a698df13a0
Jul 29 19:18:58.598733 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:58.598701    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-978bbdbf5-kp5lc"]
Jul 29 19:18:58.601737 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:18:58.601712    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod692df9a1_9ac3_414c_89e3_2e414f8b0599.slice/crio-514be55c8b78415aacf0b8730cb83716d9222f41fe389a6029a44edb32a3729e WatchSource:0}: Error finding container 514be55c8b78415aacf0b8730cb83716d9222f41fe389a6029a44edb32a3729e: Status 404 returned error can't find the container with id 514be55c8b78415aacf0b8730cb83716d9222f41fe389a6029a44edb32a3729e
Jul 29 19:18:58.611921 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:58.611902    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-frhw9"]
Jul 29 19:18:58.614011 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:18:58.613990    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7bc168b_7b7c_4242_b505_92500fd9a3a3.slice/crio-cb964e2debf3eb0bef73a45b5695556d5e9bbf72331b9a7975ab67ed2d3696a0 WatchSource:0}: Error finding container cb964e2debf3eb0bef73a45b5695556d5e9bbf72331b9a7975ab67ed2d3696a0: Status 404 returned error can't find the container with id cb964e2debf3eb0bef73a45b5695556d5e9bbf72331b9a7975ab67ed2d3696a0
Jul 29 19:18:59.264263 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:59.264177    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kq4zj" event={"ID":"ffafbd2c-ca03-47e9-ab0d-cd6125697e9e","Type":"ContainerStarted","Data":"42034a3ff3ad581d4901ab89646716ce770a83180029eac93ad2c4a698df13a0"}
Jul 29 19:18:59.269577 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:59.269499    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bpq7l" event={"ID":"6e8a45f8-a6cf-4d48-b1cb-92a91e190c50","Type":"ContainerStarted","Data":"68e1c5e85e0d106405f8dfcc2f37d72180b266527ab12e58ed52c74438b85fa5"}
Jul 29 19:18:59.275375 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:59.274916    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-84drj" event={"ID":"b92baea5-f190-4e2d-9e6e-2fcfc1a61176","Type":"ContainerStarted","Data":"bc3f431b46b6369e89ddbe74a01e6157f47a57128cc4a9d72e70b89401bcfe0e"}
Jul 29 19:18:59.276471 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:59.276417    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-84drj"
Jul 29 19:18:59.278856 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:59.278781    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-frhw9" event={"ID":"d7bc168b-7b7c-4242-b505-92500fd9a3a3","Type":"ContainerStarted","Data":"cb964e2debf3eb0bef73a45b5695556d5e9bbf72331b9a7975ab67ed2d3696a0"}
Jul 29 19:18:59.281681 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:59.281656    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-978bbdbf5-kp5lc" event={"ID":"692df9a1-9ac3-414c-89e3-2e414f8b0599","Type":"ContainerStarted","Data":"514be55c8b78415aacf0b8730cb83716d9222f41fe389a6029a44edb32a3729e"}
Jul 29 19:18:59.288075 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:59.288030    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-84drj"
Jul 29 19:18:59.300203 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:18:59.299995    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-84drj" podStartSLOduration=1.94923957 podStartE2EDuration="20.299959999s" totalImagesPullingTime="18.350720429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:18:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:18:59.294564879 +0000 UTC m=+110.122063978" watchObservedRunningTime="2026-07-29 19:18:59.299959999 +0000 UTC m=+110.127459072"
Jul 29 19:19:07.325865 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:07.325824    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kq4zj" event={"ID":"ffafbd2c-ca03-47e9-ab0d-cd6125697e9e","Type":"ContainerStarted","Data":"8a0e8b64c3c03a4565a0e507cce735d7c2f41baecd68b7e2411609932492f6a6"}
Jul 29 19:19:07.326387 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:07.325891    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kq4zj" event={"ID":"ffafbd2c-ca03-47e9-ab0d-cd6125697e9e","Type":"ContainerStarted","Data":"fa9db0f20219730bfe2b62ea6f4c4e9f9f2e974b1748b0a7c456627cef9f6e1c"}
Jul 29 19:19:07.326387 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:07.326117    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kq4zj"
Jul 29 19:19:07.328131 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:07.328096    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bpq7l" event={"ID":"6e8a45f8-a6cf-4d48-b1cb-92a91e190c50","Type":"ContainerStarted","Data":"48c8cd5b7d925640df92d8e3b9b14392559236f9f0076c5d440e4bc8723d51a8"}
Jul 29 19:19:07.331160 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:07.331132    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-frhw9" event={"ID":"d7bc168b-7b7c-4242-b505-92500fd9a3a3","Type":"ContainerStarted","Data":"c7bea794aec91b5794325d01da6be259668f0de1f02549b03819a2b23d5381af"}
Jul 29 19:19:07.331286 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:07.331194    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-frhw9" event={"ID":"d7bc168b-7b7c-4242-b505-92500fd9a3a3","Type":"ContainerStarted","Data":"75c124c94c3c424165583892d0cf69e1a4ca41a1ed666a4cdffe6999c3733d37"}
Jul 29 19:19:07.333221 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:07.333191    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-978bbdbf5-kp5lc" event={"ID":"692df9a1-9ac3-414c-89e3-2e414f8b0599","Type":"ContainerStarted","Data":"b7cd076fbcf4d33bd35e634e072d1beeb94b9d752c2cfc9479fc471682920ce8"}
Jul 29 19:19:07.343913 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:07.343861    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kq4zj" podStartSLOduration=74.119361129 podStartE2EDuration="1m22.343846532s" totalImagesPullingTime="8.224485403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:19:07.343256675 +0000 UTC m=+118.170755754" watchObservedRunningTime="2026-07-29 19:19:07.343846532 +0000 UTC m=+118.171345593"
Jul 29 19:19:07.361083 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:07.361026    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-frhw9" podStartSLOduration=9.192192686 podStartE2EDuration="17.361008725s" totalImagesPullingTime="8.168816039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:18:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:19:07.359842769 +0000 UTC m=+118.187341829" watchObservedRunningTime="2026-07-29 19:19:07.361008725 +0000 UTC m=+118.188507785"
Jul 29 19:19:07.380673 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:07.380613    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-978bbdbf5-kp5lc" podStartSLOduration=5.164568962 podStartE2EDuration="13.380589447s" totalImagesPullingTime="8.216020485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:18:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:19:07.379735943 +0000 UTC m=+118.207235001" watchObservedRunningTime="2026-07-29 19:19:07.380589447 +0000 UTC m=+118.208088512"
Jul 29 19:19:09.103644 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:09.102108    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-v8jvg"]
Jul 29 19:19:09.131543 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:09.131504    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v8jvg"
Jul 29 19:19:09.135124 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:09.135099    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2pdcd\""
Jul 29 19:19:09.136122 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:09.136098    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 29 19:19:09.136492 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:09.136468    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 29 19:19:09.136608 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:09.136476    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 29 19:19:09.214618 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:09.214588    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e58f89f-c42e-4fe3-9d4f-14e805228f11-sys\") pod \"node-exporter-v8jvg\" (UID: \"5e58f89f-c42e-4fe3-9d4f-14e805228f11\") " pod="openshift-monitoring/node-exporter-v8jvg"
Jul 29 19:19:09.214805 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:09.214630    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e58f89f-c42e-4fe3-9d4f-14e805228f11-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v8jvg\" (UID: \"5e58f89f-c42e-4fe3-9d4f-14e805228f11\") " pod="openshift-monitoring/node-exporter-v8jvg"
Jul 29 19:19:09.214805 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:09.214663    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmc76\" (UniqueName: \"kubernetes.io/projected/5e58f89f-c42e-4fe3-9d4f-14e805228f11-kube-api-access-wmc76\") pod \"node-exporter-v8jvg\" (UID: \"5e58f89f-c42e-4fe3-9d4f-14e805228f11\") " pod="openshift-monitoring/node-exporter-v8jvg"
Jul 29 19:19:09.214805 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:09.214736    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5e58f89f-c42e-4fe3-9d4f-14e805228f11-node-exporter-tls\") pod \"node-exporter-v8jvg\" (UID: \"5e58f89f-c42e-4fe3-9d4f-14e805228f11\") " pod="openshift-monitoring/node-exporter-v8jvg"
Jul 29 19:19:09.214998 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:09.214803    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5e58f89f-c42e-4fe3-9d4f-14e805228f11-node-exporter-wtmp\") pod \"node-exporter-v8jvg\" (UID: \"5e58f89f-c42e-4fe3-9d4f-14e805228f11\") " pod="openshift-monitoring/node-exporter-v8jvg"
Jul 29 19:19:09.214998 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:09.214859    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5e58f89f-c42e-4fe3-9d4f-14e805228f11-root\") pod \"node-exporter-v8jvg\" (UID: \"5e58f89f-c42e-4fe3-9d4f-14e805228f11\") " pod="openshift-monitoring/node-exporter-v8jvg"
Jul 29 19:19:09.214998 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:09.214906    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5e58f89f-c42e-4fe3-9d4f-14e805228f11-node-exporter-textfile\") pod \"node-exporter-v8jvg\" (UID: \"5e58f89f-c42e-4fe3-9d4f-14e805228f11\") " pod="openshift-monitoring/node-exporter-v8jvg"
Jul 29 19:19:09.214998 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:09.214926    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e58f89f-c42e-4fe3-9d4f-14e805228f11-metrics-client-ca\") pod \"node-exporter-v8jvg\" (UID: \"5e58f89f-c42e-4fe3-9d4f-14e805228f11\") " pod="openshift-monitoring/node-exporter-v8jvg"
Jul 29 19:19:09.215169 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:09.215039    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5e58f89f-c42e-4fe3-9d4f-14e805228f11-node-exporter-accelerators-collector-config\") pod \"node-exporter-v8jvg\" (UID: \"5e58f89f-c42e-4fe3-9d4f-14e805228f11\") " pod="openshift-monitoring/node-exporter-v8jvg"
Jul 29 19:19:09.315885 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:09.315844    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e58f89f-c42e-4fe3-9d4f-14e805228f11-sys\") pod \"node-exporter-v8jvg\" (UID: \"5e58f89f-c42e-4fe3-9d4f-14e805228f11\") " pod="openshift-monitoring/node-exporter-v8jvg"
Jul 29 19:19:09.316066 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:09.315892    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e58f89f-c42e-4fe3-9d4f-14e805228f11-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v8jvg\" (UID: \"5e58f89f-c42e-4fe3-9d4f-14e805228f11\") " pod="openshift-monitoring/node-exporter-v8jvg"
Jul 29 19:19:09.316066 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:09.315924    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmc76\" (UniqueName: \"kubernetes.io/projected/5e58f89f-c42e-4fe3-9d4f-14e805228f11-kube-api-access-wmc76\") pod \"node-exporter-v8jvg\" (UID: \"5e58f89f-c42e-4fe3-9d4f-14e805228f11\") " pod="openshift-monitoring/node-exporter-v8jvg"
Jul 29 19:19:09.316066 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:09.315953    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5e58f89f-c42e-4fe3-9d4f-14e805228f11-node-exporter-tls\") pod \"node-exporter-v8jvg\" (UID: \"5e58f89f-c42e-4fe3-9d4f-14e805228f11\") " pod="openshift-monitoring/node-exporter-v8jvg"
Jul 29 19:19:09.316066 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:09.315980    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e58f89f-c42e-4fe3-9d4f-14e805228f11-sys\") pod \"node-exporter-v8jvg\" (UID: \"5e58f89f-c42e-4fe3-9d4f-14e805228f11\") " pod="openshift-monitoring/node-exporter-v8jvg"
Jul 29 19:19:09.316066 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:09.316009    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5e58f89f-c42e-4fe3-9d4f-14e805228f11-node-exporter-wtmp\") pod \"node-exporter-v8jvg\" (UID: \"5e58f89f-c42e-4fe3-9d4f-14e805228f11\") " pod="openshift-monitoring/node-exporter-v8jvg"
Jul 29 19:19:09.316066 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:09.316065    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5e58f89f-c42e-4fe3-9d4f-14e805228f11-root\") pod \"node-exporter-v8jvg\" (UID: \"5e58f89f-c42e-4fe3-9d4f-14e805228f11\") " pod="openshift-monitoring/node-exporter-v8jvg"
Jul 29 19:19:09.316387 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:09.316121    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5e58f89f-c42e-4fe3-9d4f-14e805228f11-node-exporter-wtmp\") pod \"node-exporter-v8jvg\" (UID: \"5e58f89f-c42e-4fe3-9d4f-14e805228f11\") " pod="openshift-monitoring/node-exporter-v8jvg"
Jul 29 19:19:09.316387 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:09.316162    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5e58f89f-c42e-4fe3-9d4f-14e805228f11-root\") pod \"node-exporter-v8jvg\" (UID: \"5e58f89f-c42e-4fe3-9d4f-14e805228f11\") " pod="openshift-monitoring/node-exporter-v8jvg"
Jul 29 19:19:09.316494 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:09.316117    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5e58f89f-c42e-4fe3-9d4f-14e805228f11-node-exporter-textfile\") pod \"node-exporter-v8jvg\" (UID: \"5e58f89f-c42e-4fe3-9d4f-14e805228f11\") " pod="openshift-monitoring/node-exporter-v8jvg"
Jul 29 19:19:09.316494 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:09.316418    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5e58f89f-c42e-4fe3-9d4f-14e805228f11-node-exporter-textfile\") pod \"node-exporter-v8jvg\" (UID: \"5e58f89f-c42e-4fe3-9d4f-14e805228f11\") " pod="openshift-monitoring/node-exporter-v8jvg"
Jul 29 19:19:09.316494 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:09.316461    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e58f89f-c42e-4fe3-9d4f-14e805228f11-metrics-client-ca\") pod \"node-exporter-v8jvg\" (UID: \"5e58f89f-c42e-4fe3-9d4f-14e805228f11\") " pod="openshift-monitoring/node-exporter-v8jvg"
Jul 29 19:19:09.316655 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:09.316514    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5e58f89f-c42e-4fe3-9d4f-14e805228f11-node-exporter-accelerators-collector-config\") pod \"node-exporter-v8jvg\" (UID: \"5e58f89f-c42e-4fe3-9d4f-14e805228f11\") " pod="openshift-monitoring/node-exporter-v8jvg"
Jul 29 19:19:09.317143 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:09.317068    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5e58f89f-c42e-4fe3-9d4f-14e805228f11-node-exporter-accelerators-collector-config\") pod \"node-exporter-v8jvg\" (UID: \"5e58f89f-c42e-4fe3-9d4f-14e805228f11\") " pod="openshift-monitoring/node-exporter-v8jvg"
Jul 29 19:19:09.317143 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:09.317127    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e58f89f-c42e-4fe3-9d4f-14e805228f11-metrics-client-ca\") pod \"node-exporter-v8jvg\" (UID: \"5e58f89f-c42e-4fe3-9d4f-14e805228f11\") " pod="openshift-monitoring/node-exporter-v8jvg"
Jul 29 19:19:09.318562 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:09.318540    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e58f89f-c42e-4fe3-9d4f-14e805228f11-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v8jvg\" (UID: \"5e58f89f-c42e-4fe3-9d4f-14e805228f11\") " pod="openshift-monitoring/node-exporter-v8jvg"
Jul 29 19:19:09.318875 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:09.318849    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5e58f89f-c42e-4fe3-9d4f-14e805228f11-node-exporter-tls\") pod \"node-exporter-v8jvg\" (UID: \"5e58f89f-c42e-4fe3-9d4f-14e805228f11\") " pod="openshift-monitoring/node-exporter-v8jvg"
Jul 29 19:19:09.326626 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:09.326595    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmc76\" (UniqueName: \"kubernetes.io/projected/5e58f89f-c42e-4fe3-9d4f-14e805228f11-kube-api-access-wmc76\") pod \"node-exporter-v8jvg\" (UID: \"5e58f89f-c42e-4fe3-9d4f-14e805228f11\") " pod="openshift-monitoring/node-exporter-v8jvg"
Jul 29 19:19:09.444753 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:09.444715    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v8jvg"
Jul 29 19:19:09.476420 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:19:09.476382    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e58f89f_c42e_4fe3_9d4f_14e805228f11.slice/crio-b733178c7139752262db551869892cfd0287ad2c03a8188f6d1a2c5ee768f542 WatchSource:0}: Error finding container b733178c7139752262db551869892cfd0287ad2c03a8188f6d1a2c5ee768f542: Status 404 returned error can't find the container with id b733178c7139752262db551869892cfd0287ad2c03a8188f6d1a2c5ee768f542
Jul 29 19:19:10.346378 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:10.346340    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v8jvg" event={"ID":"5e58f89f-c42e-4fe3-9d4f-14e805228f11","Type":"ContainerStarted","Data":"b733178c7139752262db551869892cfd0287ad2c03a8188f6d1a2c5ee768f542"}
Jul 29 19:19:13.361894 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:13.361850    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bpq7l" event={"ID":"6e8a45f8-a6cf-4d48-b1cb-92a91e190c50","Type":"ContainerStarted","Data":"0c4a4fb739817f29a492930eb83e5542a01ccda8f1ac801a40c381be1097cae9"}
Jul 29 19:19:13.363365 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:13.363337    2557 generic.go:356] "Generic (PLEG): container finished" podID="5e58f89f-c42e-4fe3-9d4f-14e805228f11" containerID="e862ce65055de8550e5000390ac2df941455e30c01efcda0d2eb918b65c1559e" exitCode=0
Jul 29 19:19:13.363476 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:13.363395    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v8jvg" event={"ID":"5e58f89f-c42e-4fe3-9d4f-14e805228f11","Type":"ContainerDied","Data":"e862ce65055de8550e5000390ac2df941455e30c01efcda0d2eb918b65c1559e"}
Jul 29 19:19:13.381760 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:13.381718    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bpq7l" podStartSLOduration=34.315670649 podStartE2EDuration="1m0.381704559s" totalImagesPullingTime="26.06603391s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:18:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:19:13.380143854 +0000 UTC m=+124.207642911" watchObservedRunningTime="2026-07-29 19:19:13.381704559 +0000 UTC m=+124.209203617"
Jul 29 19:19:13.853613 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:13.853579    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-s7m8g"]
Jul 29 19:19:13.868751 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:13.868718    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-978bbdbf5-kp5lc"]
Jul 29 19:19:13.868952 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:13.868830    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-s7m8g"
Jul 29 19:19:13.871229 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:13.871203    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-s7m8g"]
Jul 29 19:19:13.872483 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:13.872459    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 29 19:19:13.872792 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:13.872774    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-5d6l5\""
Jul 29 19:19:13.959372 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:13.959331    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8329219a-75ca-4323-a2e1-42d46d14b6d8-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-s7m8g\" (UID: \"8329219a-75ca-4323-a2e1-42d46d14b6d8\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-s7m8g"
Jul 29 19:19:13.959546 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:13.959400    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pcld\" (UniqueName: \"kubernetes.io/projected/8329219a-75ca-4323-a2e1-42d46d14b6d8-kube-api-access-6pcld\") pod \"monitoring-plugin-68b7fc9574-s7m8g\" (UID: \"8329219a-75ca-4323-a2e1-42d46d14b6d8\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-s7m8g"
Jul 29 19:19:14.060126 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:14.060092    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8329219a-75ca-4323-a2e1-42d46d14b6d8-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-s7m8g\" (UID: \"8329219a-75ca-4323-a2e1-42d46d14b6d8\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-s7m8g"
Jul 29 19:19:14.060126 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:14.060138    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pcld\" (UniqueName: \"kubernetes.io/projected/8329219a-75ca-4323-a2e1-42d46d14b6d8-kube-api-access-6pcld\") pod \"monitoring-plugin-68b7fc9574-s7m8g\" (UID: \"8329219a-75ca-4323-a2e1-42d46d14b6d8\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-s7m8g"
Jul 29 19:19:14.062602 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:14.062571    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8329219a-75ca-4323-a2e1-42d46d14b6d8-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-s7m8g\" (UID: \"8329219a-75ca-4323-a2e1-42d46d14b6d8\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-s7m8g"
Jul 29 19:19:14.068939 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:14.068912    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pcld\" (UniqueName: \"kubernetes.io/projected/8329219a-75ca-4323-a2e1-42d46d14b6d8-kube-api-access-6pcld\") pod \"monitoring-plugin-68b7fc9574-s7m8g\" (UID: \"8329219a-75ca-4323-a2e1-42d46d14b6d8\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-s7m8g"
Jul 29 19:19:14.181776 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:14.181743    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-s7m8g"
Jul 29 19:19:14.320976 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:14.320936    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-s7m8g"]
Jul 29 19:19:14.324756 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:19:14.324723    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8329219a_75ca_4323_a2e1_42d46d14b6d8.slice/crio-aa43eddaef00a41cc98558834eccddb638b5d7f9ed31fdab6ebe7ba1bc13dcbc WatchSource:0}: Error finding container aa43eddaef00a41cc98558834eccddb638b5d7f9ed31fdab6ebe7ba1bc13dcbc: Status 404 returned error can't find the container with id aa43eddaef00a41cc98558834eccddb638b5d7f9ed31fdab6ebe7ba1bc13dcbc
Jul 29 19:19:14.370209 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:14.370176    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v8jvg" event={"ID":"5e58f89f-c42e-4fe3-9d4f-14e805228f11","Type":"ContainerStarted","Data":"4cd464cd3d0046bfaac48ed02929e5d3bf3cc9e025fb692b3b4cf833175108f3"}
Jul 29 19:19:14.370819 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:14.370241    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v8jvg" event={"ID":"5e58f89f-c42e-4fe3-9d4f-14e805228f11","Type":"ContainerStarted","Data":"fc6baa82deca515794cc1c57a13f682efb6143567064f8175a786ca8889641e7"}
Jul 29 19:19:14.371447 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:14.371416    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-s7m8g" event={"ID":"8329219a-75ca-4323-a2e1-42d46d14b6d8","Type":"ContainerStarted","Data":"aa43eddaef00a41cc98558834eccddb638b5d7f9ed31fdab6ebe7ba1bc13dcbc"}
Jul 29 19:19:14.390105 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:14.390053    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-v8jvg" podStartSLOduration=1.9222515150000001 podStartE2EDuration="5.390037964s" totalImagesPullingTime="3.378013186s" totalInitContainerRuntime="89.773263ms" isStatefulPod=true podCreationTimestamp="2026-07-29 19:19:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:19:14.389054674 +0000 UTC m=+125.216553732" watchObservedRunningTime="2026-07-29 19:19:14.390037964 +0000 UTC m=+125.217537023"
Jul 29 19:19:14.726793 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:14.726760    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-978bbdbf5-kp5lc"
Jul 29 19:19:15.290732 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.289717    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 19:19:15.303508 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.303481    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.306365 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.306336    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 29 19:19:15.306365 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.306336    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 29 19:19:15.306563 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.306435    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 29 19:19:15.306801 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.306779    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jwqcw\""
Jul 29 19:19:15.306801 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.306779    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 29 19:19:15.309256 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.309232    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 29 19:19:15.309748 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.309721    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 29 19:19:15.309988 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.309948    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 29 19:19:15.310711 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.310279    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 29 19:19:15.310711 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.310556    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 29 19:19:15.310848 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.310767    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 29 19:19:15.310985 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.310929    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 29 19:19:15.311075 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.310926    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 29 19:19:15.311383 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.311170    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 29 19:19:15.311383 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.311182    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 29 19:19:15.311773 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.311658    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9ivhvpis64tfb\""
Jul 29 19:19:15.312699 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.312678    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 19:19:15.313024 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.312859    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 29 19:19:15.471994 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.470913    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eba3dd68-ff9d-472d-b498-a0d7af3ae175-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.471994 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.470960    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/eba3dd68-ff9d-472d-b498-a0d7af3ae175-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.471994 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.471046    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eba3dd68-ff9d-472d-b498-a0d7af3ae175-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.471994 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.471086    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eba3dd68-ff9d-472d-b498-a0d7af3ae175-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.471994 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.471108    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eba3dd68-ff9d-472d-b498-a0d7af3ae175-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.471994 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.471135    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/eba3dd68-ff9d-472d-b498-a0d7af3ae175-config\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.471994 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.471155    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eba3dd68-ff9d-472d-b498-a0d7af3ae175-config-out\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.471994 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.471178    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eba3dd68-ff9d-472d-b498-a0d7af3ae175-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.471994 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.471215    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/eba3dd68-ff9d-472d-b498-a0d7af3ae175-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.471994 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.471236    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eba3dd68-ff9d-472d-b498-a0d7af3ae175-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.471994 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.471277    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eba3dd68-ff9d-472d-b498-a0d7af3ae175-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.471994 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.471309    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eba3dd68-ff9d-472d-b498-a0d7af3ae175-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.471994 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.471342    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eba3dd68-ff9d-472d-b498-a0d7af3ae175-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.471994 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.471384    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/eba3dd68-ff9d-472d-b498-a0d7af3ae175-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.471994 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.471430    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eba3dd68-ff9d-472d-b498-a0d7af3ae175-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.471994 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.471467    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eba3dd68-ff9d-472d-b498-a0d7af3ae175-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.473166 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.471491    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f6hs\" (UniqueName: \"kubernetes.io/projected/eba3dd68-ff9d-472d-b498-a0d7af3ae175-kube-api-access-8f6hs\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.473166 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.471539    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eba3dd68-ff9d-472d-b498-a0d7af3ae175-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.473166 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.471601    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/eba3dd68-ff9d-472d-b498-a0d7af3ae175-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.473166 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.471676    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eba3dd68-ff9d-472d-b498-a0d7af3ae175-web-config\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.572906 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.572806    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f6hs\" (UniqueName: \"kubernetes.io/projected/eba3dd68-ff9d-472d-b498-a0d7af3ae175-kube-api-access-8f6hs\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.572906 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.572869    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eba3dd68-ff9d-472d-b498-a0d7af3ae175-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.572906 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.572896    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/eba3dd68-ff9d-472d-b498-a0d7af3ae175-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.573177 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.572923    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eba3dd68-ff9d-472d-b498-a0d7af3ae175-web-config\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.573177 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.572952    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eba3dd68-ff9d-472d-b498-a0d7af3ae175-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.573177 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.572992    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/eba3dd68-ff9d-472d-b498-a0d7af3ae175-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.573177 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.573033    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eba3dd68-ff9d-472d-b498-a0d7af3ae175-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.573177 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.573062    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eba3dd68-ff9d-472d-b498-a0d7af3ae175-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.573177 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.573086    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eba3dd68-ff9d-472d-b498-a0d7af3ae175-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.573177 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.573113    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/eba3dd68-ff9d-472d-b498-a0d7af3ae175-config\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.573177 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.573134    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eba3dd68-ff9d-472d-b498-a0d7af3ae175-config-out\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.573177 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.573160    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eba3dd68-ff9d-472d-b498-a0d7af3ae175-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.573606 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.573244    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/eba3dd68-ff9d-472d-b498-a0d7af3ae175-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.573606 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.573280    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eba3dd68-ff9d-472d-b498-a0d7af3ae175-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.573606 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.573324    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eba3dd68-ff9d-472d-b498-a0d7af3ae175-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.573606 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.573351    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eba3dd68-ff9d-472d-b498-a0d7af3ae175-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.573606 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.573386    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eba3dd68-ff9d-472d-b498-a0d7af3ae175-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.573606 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.573432    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/eba3dd68-ff9d-472d-b498-a0d7af3ae175-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.573606 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.573480    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eba3dd68-ff9d-472d-b498-a0d7af3ae175-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.573606 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.573517    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eba3dd68-ff9d-472d-b498-a0d7af3ae175-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.573943 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.573924    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/eba3dd68-ff9d-472d-b498-a0d7af3ae175-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.574229 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.574203    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eba3dd68-ff9d-472d-b498-a0d7af3ae175-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.574334 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.574244    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eba3dd68-ff9d-472d-b498-a0d7af3ae175-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.574514 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.574490    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eba3dd68-ff9d-472d-b498-a0d7af3ae175-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.576995 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.576668    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eba3dd68-ff9d-472d-b498-a0d7af3ae175-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.577107 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.577066    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/eba3dd68-ff9d-472d-b498-a0d7af3ae175-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.579424 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.577617    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eba3dd68-ff9d-472d-b498-a0d7af3ae175-config-out\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.579424 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.578394    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/eba3dd68-ff9d-472d-b498-a0d7af3ae175-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.579424 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.574244    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eba3dd68-ff9d-472d-b498-a0d7af3ae175-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.579424 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.578852    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eba3dd68-ff9d-472d-b498-a0d7af3ae175-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.579799 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.579766    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eba3dd68-ff9d-472d-b498-a0d7af3ae175-web-config\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.580567 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.580523    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eba3dd68-ff9d-472d-b498-a0d7af3ae175-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.581096 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.581058    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/eba3dd68-ff9d-472d-b498-a0d7af3ae175-config\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.582651 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.582629    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eba3dd68-ff9d-472d-b498-a0d7af3ae175-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.583948 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.583441    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eba3dd68-ff9d-472d-b498-a0d7af3ae175-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.583948 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.583515    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eba3dd68-ff9d-472d-b498-a0d7af3ae175-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.583948 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.583881    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eba3dd68-ff9d-472d-b498-a0d7af3ae175-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.584203 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.584017    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/eba3dd68-ff9d-472d-b498-a0d7af3ae175-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.584203 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.584021    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f6hs\" (UniqueName: \"kubernetes.io/projected/eba3dd68-ff9d-472d-b498-a0d7af3ae175-kube-api-access-8f6hs\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.585491 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.585463    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eba3dd68-ff9d-472d-b498-a0d7af3ae175-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"eba3dd68-ff9d-472d-b498-a0d7af3ae175\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.617934 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.617692    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:15.978024 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:19:15.977990    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba3dd68_ff9d_472d_b498_a0d7af3ae175.slice/crio-e268258a76f6011dde0f4696ff0349e4747adcd6b43ea6c01982eaa264da35b7 WatchSource:0}: Error finding container e268258a76f6011dde0f4696ff0349e4747adcd6b43ea6c01982eaa264da35b7: Status 404 returned error can't find the container with id e268258a76f6011dde0f4696ff0349e4747adcd6b43ea6c01982eaa264da35b7
Jul 29 19:19:15.984341 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:15.984309    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 19:19:16.379198 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:16.379164    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eba3dd68-ff9d-472d-b498-a0d7af3ae175","Type":"ContainerStarted","Data":"e268258a76f6011dde0f4696ff0349e4747adcd6b43ea6c01982eaa264da35b7"}
Jul 29 19:19:16.380495 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:16.380468    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-s7m8g" event={"ID":"8329219a-75ca-4323-a2e1-42d46d14b6d8","Type":"ContainerStarted","Data":"d9a60de688ba31f387d770165794356bddbb6190043eda59bb6aa870cc2ab52e"}
Jul 29 19:19:16.380699 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:16.380681    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-s7m8g"
Jul 29 19:19:16.385897 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:16.385878    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-s7m8g"
Jul 29 19:19:16.410594 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:16.410541    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-s7m8g" podStartSLOduration=1.56967306 podStartE2EDuration="3.410523704s" totalImagesPullingTime="1.840850644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:19:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:19:16.409323375 +0000 UTC m=+127.236822430" watchObservedRunningTime="2026-07-29 19:19:16.410523704 +0000 UTC m=+127.238022763"
Jul 29 19:19:17.338799 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:17.338768    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kq4zj"
Jul 29 19:19:18.391167 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:18.391128    2557 generic.go:356] "Generic (PLEG): container finished" podID="eba3dd68-ff9d-472d-b498-a0d7af3ae175" containerID="83aa767d0a2a8f32421ae2e71d7337c755b077ce02c111b0d9063f3358124039" exitCode=0
Jul 29 19:19:18.391633 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:18.391219    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eba3dd68-ff9d-472d-b498-a0d7af3ae175","Type":"ContainerDied","Data":"83aa767d0a2a8f32421ae2e71d7337c755b077ce02c111b0d9063f3358124039"}
Jul 29 19:19:19.407257 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:19.407216    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4f202a3-5dc7-416f-8793-e857e6086174-metrics-certs\") pod \"network-metrics-daemon-cw75m\" (UID: \"a4f202a3-5dc7-416f-8793-e857e6086174\") " pod="openshift-multus/network-metrics-daemon-cw75m"
Jul 29 19:19:19.409645 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:19.409619    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4f202a3-5dc7-416f-8793-e857e6086174-metrics-certs\") pod \"network-metrics-daemon-cw75m\" (UID: \"a4f202a3-5dc7-416f-8793-e857e6086174\") " pod="openshift-multus/network-metrics-daemon-cw75m"
Jul 29 19:19:19.585692 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:19.585659    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tgbwc\""
Jul 29 19:19:19.593309 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:19.593275    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cw75m"
Jul 29 19:19:19.765521 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:19.763720    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cw75m"]
Jul 29 19:19:20.404263 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:20.404215    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cw75m" event={"ID":"a4f202a3-5dc7-416f-8793-e857e6086174","Type":"ContainerStarted","Data":"03245a642db679dc06d3cf97e793b11fa9d53c70494d3d42bb3da9c1596ec90c"}
Jul 29 19:19:22.412788 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:22.412761    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cw75m" event={"ID":"a4f202a3-5dc7-416f-8793-e857e6086174","Type":"ContainerStarted","Data":"08d70dc3610467b562aace986c76323da4032c6672b11e750236a052d988b6ea"}
Jul 29 19:19:22.414501 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:22.414478    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eba3dd68-ff9d-472d-b498-a0d7af3ae175","Type":"ContainerStarted","Data":"61e222b25844ea58f4c6d06f13164828c53ff58d070ba2828b5bc28696f30a23"}
Jul 29 19:19:23.421876 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:23.421439    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cw75m" event={"ID":"a4f202a3-5dc7-416f-8793-e857e6086174","Type":"ContainerStarted","Data":"554f236ae0b3a8d4d1b532e544f35ead75603f11a2043a4ebe3403307332e2f9"}
Jul 29 19:19:23.426604 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:23.426549    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eba3dd68-ff9d-472d-b498-a0d7af3ae175","Type":"ContainerStarted","Data":"6b66016265f5c4e17f4c671791efb2f90e82bd0c52e5b1d3860485b19976c63e"}
Jul 29 19:19:23.441734 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:23.441679    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cw75m" podStartSLOduration=131.98726774 podStartE2EDuration="2m14.441661057s" totalImagesPullingTime="2.454393317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:17:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:19:23.440347449 +0000 UTC m=+134.267846510" watchObservedRunningTime="2026-07-29 19:19:23.441661057 +0000 UTC m=+134.269160119"
Jul 29 19:19:25.438874 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:25.438836    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eba3dd68-ff9d-472d-b498-a0d7af3ae175","Type":"ContainerStarted","Data":"83518718c3dc773511cab9303c1457520c4f5e4cde66257bb7d3910bf0e4e5d0"}
Jul 29 19:19:25.439266 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:25.438933    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eba3dd68-ff9d-472d-b498-a0d7af3ae175","Type":"ContainerStarted","Data":"7231ab4c210c5d9d3afdee08561dae7749b3027b26f09b48e68453599f0d28dc"}
Jul 29 19:19:25.439266 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:25.438980    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eba3dd68-ff9d-472d-b498-a0d7af3ae175","Type":"ContainerStarted","Data":"48ce4943979832d532624fb67a0712959548dd3ad5292427397d5dd6faf70237"}
Jul 29 19:19:25.439266 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:25.439005    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eba3dd68-ff9d-472d-b498-a0d7af3ae175","Type":"ContainerStarted","Data":"d58d224af9b540cf905005b4293ba4b72799307f22d509d2423243cc331015fb"}
Jul 29 19:19:25.471278 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:25.471227    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.705762608 podStartE2EDuration="10.471210897s" totalImagesPullingTime="7.751770561s" totalInitContainerRuntime="13.677728ms" isStatefulPod=false podCreationTimestamp="2026-07-29 19:19:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:19:25.469299855 +0000 UTC m=+136.296798953" watchObservedRunningTime="2026-07-29 19:19:25.471210897 +0000 UTC m=+136.298709955"
Jul 29 19:19:25.619159 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:25.619128    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:19:38.489560 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:38.489525    2557 generic.go:356] "Generic (PLEG): container finished" podID="2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac" containerID="b0add6812fd061fd47bf81d64630c543bdd9a97668fcd8b6915f30d376a7aaac" exitCode=0
Jul 29 19:19:38.490003 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:38.489598    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-cfmh8" event={"ID":"2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac","Type":"ContainerDied","Data":"b0add6812fd061fd47bf81d64630c543bdd9a97668fcd8b6915f30d376a7aaac"}
Jul 29 19:19:38.490003 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:38.489924    2557 scope.go:117] "RemoveContainer" containerID="b0add6812fd061fd47bf81d64630c543bdd9a97668fcd8b6915f30d376a7aaac"
Jul 29 19:19:38.889858 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:38.889792    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-978bbdbf5-kp5lc" podUID="692df9a1-9ac3-414c-89e3-2e414f8b0599" containerName="console" containerID="cri-o://b7cd076fbcf4d33bd35e634e072d1beeb94b9d752c2cfc9479fc471682920ce8" gracePeriod=15
Jul 29 19:19:39.165348 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:39.165283    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-978bbdbf5-kp5lc"
Jul 29 19:19:39.270364 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:39.270334    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/692df9a1-9ac3-414c-89e3-2e414f8b0599-trusted-ca-bundle\") pod \"692df9a1-9ac3-414c-89e3-2e414f8b0599\" (UID: \"692df9a1-9ac3-414c-89e3-2e414f8b0599\") "
Jul 29 19:19:39.270364 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:39.270369    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/692df9a1-9ac3-414c-89e3-2e414f8b0599-oauth-serving-cert\") pod \"692df9a1-9ac3-414c-89e3-2e414f8b0599\" (UID: \"692df9a1-9ac3-414c-89e3-2e414f8b0599\") "
Jul 29 19:19:39.270576 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:39.270399    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/692df9a1-9ac3-414c-89e3-2e414f8b0599-console-serving-cert\") pod \"692df9a1-9ac3-414c-89e3-2e414f8b0599\" (UID: \"692df9a1-9ac3-414c-89e3-2e414f8b0599\") "
Jul 29 19:19:39.270576 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:39.270418    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-274sd\" (UniqueName: \"kubernetes.io/projected/692df9a1-9ac3-414c-89e3-2e414f8b0599-kube-api-access-274sd\") pod \"692df9a1-9ac3-414c-89e3-2e414f8b0599\" (UID: \"692df9a1-9ac3-414c-89e3-2e414f8b0599\") "
Jul 29 19:19:39.270576 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:39.270544    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/692df9a1-9ac3-414c-89e3-2e414f8b0599-console-config\") pod \"692df9a1-9ac3-414c-89e3-2e414f8b0599\" (UID: \"692df9a1-9ac3-414c-89e3-2e414f8b0599\") "
Jul 29 19:19:39.270759 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:39.270597    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/692df9a1-9ac3-414c-89e3-2e414f8b0599-service-ca\") pod \"692df9a1-9ac3-414c-89e3-2e414f8b0599\" (UID: \"692df9a1-9ac3-414c-89e3-2e414f8b0599\") "
Jul 29 19:19:39.270759 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:39.270638    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/692df9a1-9ac3-414c-89e3-2e414f8b0599-console-oauth-config\") pod \"692df9a1-9ac3-414c-89e3-2e414f8b0599\" (UID: \"692df9a1-9ac3-414c-89e3-2e414f8b0599\") "
Jul 29 19:19:39.270854 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:39.270779    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/692df9a1-9ac3-414c-89e3-2e414f8b0599-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "692df9a1-9ac3-414c-89e3-2e414f8b0599" (UID: "692df9a1-9ac3-414c-89e3-2e414f8b0599"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:19:39.270908 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:39.270871    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/692df9a1-9ac3-414c-89e3-2e414f8b0599-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "692df9a1-9ac3-414c-89e3-2e414f8b0599" (UID: "692df9a1-9ac3-414c-89e3-2e414f8b0599"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:19:39.271589 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:39.271522    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/692df9a1-9ac3-414c-89e3-2e414f8b0599-service-ca" (OuterVolumeSpecName: "service-ca") pod "692df9a1-9ac3-414c-89e3-2e414f8b0599" (UID: "692df9a1-9ac3-414c-89e3-2e414f8b0599"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:19:39.271589 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:39.271551    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/692df9a1-9ac3-414c-89e3-2e414f8b0599-console-config" (OuterVolumeSpecName: "console-config") pod "692df9a1-9ac3-414c-89e3-2e414f8b0599" (UID: "692df9a1-9ac3-414c-89e3-2e414f8b0599"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:19:39.272047 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:39.272022    2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/692df9a1-9ac3-414c-89e3-2e414f8b0599-trusted-ca-bundle\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:19:39.272126 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:39.272052    2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/692df9a1-9ac3-414c-89e3-2e414f8b0599-oauth-serving-cert\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:19:39.278616 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:39.278526    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/692df9a1-9ac3-414c-89e3-2e414f8b0599-kube-api-access-274sd" (OuterVolumeSpecName: "kube-api-access-274sd") pod "692df9a1-9ac3-414c-89e3-2e414f8b0599" (UID: "692df9a1-9ac3-414c-89e3-2e414f8b0599"). InnerVolumeSpecName "kube-api-access-274sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:19:39.279250 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:39.279198    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/692df9a1-9ac3-414c-89e3-2e414f8b0599-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "692df9a1-9ac3-414c-89e3-2e414f8b0599" (UID: "692df9a1-9ac3-414c-89e3-2e414f8b0599"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:19:39.279693 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:39.279669    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/692df9a1-9ac3-414c-89e3-2e414f8b0599-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "692df9a1-9ac3-414c-89e3-2e414f8b0599" (UID: "692df9a1-9ac3-414c-89e3-2e414f8b0599"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:19:39.373376 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:39.373340    2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/692df9a1-9ac3-414c-89e3-2e414f8b0599-console-serving-cert\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:19:39.373376 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:39.373368    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-274sd\" (UniqueName: \"kubernetes.io/projected/692df9a1-9ac3-414c-89e3-2e414f8b0599-kube-api-access-274sd\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:19:39.373376 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:39.373379    2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/692df9a1-9ac3-414c-89e3-2e414f8b0599-console-config\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:19:39.373647 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:39.373387    2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/692df9a1-9ac3-414c-89e3-2e414f8b0599-service-ca\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:19:39.373647 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:39.373395    2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/692df9a1-9ac3-414c-89e3-2e414f8b0599-console-oauth-config\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:19:39.495815 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:39.495723    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-cfmh8" event={"ID":"2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac","Type":"ContainerStarted","Data":"2d37871bbd2f077a35198146b3e6f966b85b0b4f7d69719c77a8a43afc93f98a"}
Jul 29 19:19:39.497122 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:39.497095    2557 generic.go:356] "Generic (PLEG): container finished" podID="692df9a1-9ac3-414c-89e3-2e414f8b0599" containerID="b7cd076fbcf4d33bd35e634e072d1beeb94b9d752c2cfc9479fc471682920ce8" exitCode=0
Jul 29 19:19:39.497210 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:39.497139    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-978bbdbf5-kp5lc" event={"ID":"692df9a1-9ac3-414c-89e3-2e414f8b0599","Type":"ContainerDied","Data":"b7cd076fbcf4d33bd35e634e072d1beeb94b9d752c2cfc9479fc471682920ce8"}
Jul 29 19:19:39.497210 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:39.497174    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-978bbdbf5-kp5lc" event={"ID":"692df9a1-9ac3-414c-89e3-2e414f8b0599","Type":"ContainerDied","Data":"514be55c8b78415aacf0b8730cb83716d9222f41fe389a6029a44edb32a3729e"}
Jul 29 19:19:39.497210 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:39.497193    2557 scope.go:117] "RemoveContainer" containerID="b7cd076fbcf4d33bd35e634e072d1beeb94b9d752c2cfc9479fc471682920ce8"
Jul 29 19:19:39.497366 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:39.497141    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-978bbdbf5-kp5lc"
Jul 29 19:19:39.505745 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:39.505718    2557 scope.go:117] "RemoveContainer" containerID="b7cd076fbcf4d33bd35e634e072d1beeb94b9d752c2cfc9479fc471682920ce8"
Jul 29 19:19:39.506045 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:19:39.506023    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7cd076fbcf4d33bd35e634e072d1beeb94b9d752c2cfc9479fc471682920ce8\": container with ID starting with b7cd076fbcf4d33bd35e634e072d1beeb94b9d752c2cfc9479fc471682920ce8 not found: ID does not exist" containerID="b7cd076fbcf4d33bd35e634e072d1beeb94b9d752c2cfc9479fc471682920ce8"
Jul 29 19:19:39.506136 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:39.506051    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7cd076fbcf4d33bd35e634e072d1beeb94b9d752c2cfc9479fc471682920ce8"} err="failed to get container status \"b7cd076fbcf4d33bd35e634e072d1beeb94b9d752c2cfc9479fc471682920ce8\": rpc error: code = NotFound desc = could not find container \"b7cd076fbcf4d33bd35e634e072d1beeb94b9d752c2cfc9479fc471682920ce8\": container with ID starting with b7cd076fbcf4d33bd35e634e072d1beeb94b9d752c2cfc9479fc471682920ce8 not found: ID does not exist"
Jul 29 19:19:39.532106 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:39.532075    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-978bbdbf5-kp5lc"]
Jul 29 19:19:39.534088 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:39.534066    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-978bbdbf5-kp5lc"]
Jul 29 19:19:39.678257 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:19:39.678225    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="692df9a1-9ac3-414c-89e3-2e414f8b0599" path="/var/lib/kubelet/pods/692df9a1-9ac3-414c-89e3-2e414f8b0599/volumes"
Jul 29 19:20:15.618749 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:20:15.618704    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:20:15.634163 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:20:15.634135    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:20:15.648453 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:20:15.648426    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:22:09.537937 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:09.537906    2557 kubelet.go:1648] "Image garbage collection succeeded"
Jul 29 19:22:51.513737 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:51.513698    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-ffdbfbbc8-nd2q7"]
Jul 29 19:22:51.514178 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:51.514038    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="692df9a1-9ac3-414c-89e3-2e414f8b0599" containerName="console"
Jul 29 19:22:51.514178 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:51.514050    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="692df9a1-9ac3-414c-89e3-2e414f8b0599" containerName="console"
Jul 29 19:22:51.514178 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:51.514107    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="692df9a1-9ac3-414c-89e3-2e414f8b0599" containerName="console"
Jul 29 19:22:51.515923 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:51.515906    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-ffdbfbbc8-nd2q7"
Jul 29 19:22:51.520757 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:51.520730    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 29 19:22:51.530572 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:51.530552    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-ffdbfbbc8-nd2q7"]
Jul 29 19:22:51.535201 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:51.535181    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 29 19:22:51.535317 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:51.535298    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 29 19:22:51.535370 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:51.535314    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 29 19:22:51.539131 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:51.539118    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 29 19:22:51.539820 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:51.539805    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-pj965\""
Jul 29 19:22:51.600189 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:51.600156    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/189375cf-ed59-4af0-8b17-0813e2a39795-cert\") pod \"lws-controller-manager-ffdbfbbc8-nd2q7\" (UID: \"189375cf-ed59-4af0-8b17-0813e2a39795\") " pod="openshift-lws-operator/lws-controller-manager-ffdbfbbc8-nd2q7"
Jul 29 19:22:51.600189 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:51.600194    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/189375cf-ed59-4af0-8b17-0813e2a39795-manager-config\") pod \"lws-controller-manager-ffdbfbbc8-nd2q7\" (UID: \"189375cf-ed59-4af0-8b17-0813e2a39795\") " pod="openshift-lws-operator/lws-controller-manager-ffdbfbbc8-nd2q7"
Jul 29 19:22:51.600385 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:51.600224    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwljf\" (UniqueName: \"kubernetes.io/projected/189375cf-ed59-4af0-8b17-0813e2a39795-kube-api-access-cwljf\") pod \"lws-controller-manager-ffdbfbbc8-nd2q7\" (UID: \"189375cf-ed59-4af0-8b17-0813e2a39795\") " pod="openshift-lws-operator/lws-controller-manager-ffdbfbbc8-nd2q7"
Jul 29 19:22:51.600385 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:51.600245    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/189375cf-ed59-4af0-8b17-0813e2a39795-metrics-cert\") pod \"lws-controller-manager-ffdbfbbc8-nd2q7\" (UID: \"189375cf-ed59-4af0-8b17-0813e2a39795\") " pod="openshift-lws-operator/lws-controller-manager-ffdbfbbc8-nd2q7"
Jul 29 19:22:51.701541 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:51.701514    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/189375cf-ed59-4af0-8b17-0813e2a39795-cert\") pod \"lws-controller-manager-ffdbfbbc8-nd2q7\" (UID: \"189375cf-ed59-4af0-8b17-0813e2a39795\") " pod="openshift-lws-operator/lws-controller-manager-ffdbfbbc8-nd2q7"
Jul 29 19:22:51.701698 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:51.701548    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/189375cf-ed59-4af0-8b17-0813e2a39795-manager-config\") pod \"lws-controller-manager-ffdbfbbc8-nd2q7\" (UID: \"189375cf-ed59-4af0-8b17-0813e2a39795\") " pod="openshift-lws-operator/lws-controller-manager-ffdbfbbc8-nd2q7"
Jul 29 19:22:51.701698 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:51.701575    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwljf\" (UniqueName: \"kubernetes.io/projected/189375cf-ed59-4af0-8b17-0813e2a39795-kube-api-access-cwljf\") pod \"lws-controller-manager-ffdbfbbc8-nd2q7\" (UID: \"189375cf-ed59-4af0-8b17-0813e2a39795\") " pod="openshift-lws-operator/lws-controller-manager-ffdbfbbc8-nd2q7"
Jul 29 19:22:51.701698 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:51.701595    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/189375cf-ed59-4af0-8b17-0813e2a39795-metrics-cert\") pod \"lws-controller-manager-ffdbfbbc8-nd2q7\" (UID: \"189375cf-ed59-4af0-8b17-0813e2a39795\") " pod="openshift-lws-operator/lws-controller-manager-ffdbfbbc8-nd2q7"
Jul 29 19:22:51.702284 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:51.702256    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/189375cf-ed59-4af0-8b17-0813e2a39795-manager-config\") pod \"lws-controller-manager-ffdbfbbc8-nd2q7\" (UID: \"189375cf-ed59-4af0-8b17-0813e2a39795\") " pod="openshift-lws-operator/lws-controller-manager-ffdbfbbc8-nd2q7"
Jul 29 19:22:51.704006 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:51.703985    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/189375cf-ed59-4af0-8b17-0813e2a39795-cert\") pod \"lws-controller-manager-ffdbfbbc8-nd2q7\" (UID: \"189375cf-ed59-4af0-8b17-0813e2a39795\") " pod="openshift-lws-operator/lws-controller-manager-ffdbfbbc8-nd2q7"
Jul 29 19:22:51.704214 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:51.704194    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/189375cf-ed59-4af0-8b17-0813e2a39795-metrics-cert\") pod \"lws-controller-manager-ffdbfbbc8-nd2q7\" (UID: \"189375cf-ed59-4af0-8b17-0813e2a39795\") " pod="openshift-lws-operator/lws-controller-manager-ffdbfbbc8-nd2q7"
Jul 29 19:22:51.714246 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:51.714220    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwljf\" (UniqueName: \"kubernetes.io/projected/189375cf-ed59-4af0-8b17-0813e2a39795-kube-api-access-cwljf\") pod \"lws-controller-manager-ffdbfbbc8-nd2q7\" (UID: \"189375cf-ed59-4af0-8b17-0813e2a39795\") " pod="openshift-lws-operator/lws-controller-manager-ffdbfbbc8-nd2q7"
Jul 29 19:22:51.824677 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:51.824598    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-ffdbfbbc8-nd2q7"
Jul 29 19:22:51.977269 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:51.977238    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-ffdbfbbc8-nd2q7"]
Jul 29 19:22:51.983653 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:51.983631    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 19:22:52.207020 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:52.206981    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-ffdbfbbc8-nd2q7" event={"ID":"189375cf-ed59-4af0-8b17-0813e2a39795","Type":"ContainerStarted","Data":"f030154a41e6554729cb4bcf85a8b643c0c6753c1d217dfaf0dd34a4542c1b82"}
Jul 29 19:22:55.220276 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:55.220239    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-ffdbfbbc8-nd2q7" event={"ID":"189375cf-ed59-4af0-8b17-0813e2a39795","Type":"ContainerStarted","Data":"118564b75ec0378045e4effecc1bc6d703dbfebc2f59583a27be4345cb30a4fa"}
Jul 29 19:22:55.220626 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:55.220365    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-ffdbfbbc8-nd2q7"
Jul 29 19:22:55.241234 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:55.241191    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-ffdbfbbc8-nd2q7" podStartSLOduration=1.219936874 podStartE2EDuration="4.241178145s" totalImagesPullingTime="3.021241271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:22:55.240207333 +0000 UTC m=+346.067706392" watchObservedRunningTime="2026-07-29 19:22:55.241178145 +0000 UTC m=+346.068677269"
Jul 29 19:22:55.549376 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:55.549340    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d84699fcf-kk7qz"]
Jul 29 19:22:55.551621 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:55.551604    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d84699fcf-kk7qz"
Jul 29 19:22:55.554776 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:55.554754    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 29 19:22:55.554898 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:55.554854    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-72wzr\""
Jul 29 19:22:55.554976 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:55.554906    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 29 19:22:55.555137 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:55.555122    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 29 19:22:55.555181 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:55.555149    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 29 19:22:55.568486 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:55.568465    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d84699fcf-kk7qz"]
Jul 29 19:22:55.635959 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:55.635924    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f034f84d-5306-43a8-914d-0a91c0d60b4e-manifests\") pod \"opendatahub-operator-controller-manager-d84699fcf-kk7qz\" (UID: \"f034f84d-5306-43a8-914d-0a91c0d60b4e\") " pod="opendatahub/opendatahub-operator-controller-manager-d84699fcf-kk7qz"
Jul 29 19:22:55.636114 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:55.636042    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f034f84d-5306-43a8-914d-0a91c0d60b4e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d84699fcf-kk7qz\" (UID: \"f034f84d-5306-43a8-914d-0a91c0d60b4e\") " pod="opendatahub/opendatahub-operator-controller-manager-d84699fcf-kk7qz"
Jul 29 19:22:55.636114 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:55.636067    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f034f84d-5306-43a8-914d-0a91c0d60b4e-webhook-cert\") pod \"opendatahub-operator-controller-manager-d84699fcf-kk7qz\" (UID: \"f034f84d-5306-43a8-914d-0a91c0d60b4e\") " pod="opendatahub/opendatahub-operator-controller-manager-d84699fcf-kk7qz"
Jul 29 19:22:55.636114 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:55.636094    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxv86\" (UniqueName: \"kubernetes.io/projected/f034f84d-5306-43a8-914d-0a91c0d60b4e-kube-api-access-qxv86\") pod \"opendatahub-operator-controller-manager-d84699fcf-kk7qz\" (UID: \"f034f84d-5306-43a8-914d-0a91c0d60b4e\") " pod="opendatahub/opendatahub-operator-controller-manager-d84699fcf-kk7qz"
Jul 29 19:22:55.636217 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:55.636143    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f034f84d-5306-43a8-914d-0a91c0d60b4e-tmp\") pod \"opendatahub-operator-controller-manager-d84699fcf-kk7qz\" (UID: \"f034f84d-5306-43a8-914d-0a91c0d60b4e\") " pod="opendatahub/opendatahub-operator-controller-manager-d84699fcf-kk7qz"
Jul 29 19:22:55.737155 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:55.737124    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f034f84d-5306-43a8-914d-0a91c0d60b4e-tmp\") pod \"opendatahub-operator-controller-manager-d84699fcf-kk7qz\" (UID: \"f034f84d-5306-43a8-914d-0a91c0d60b4e\") " pod="opendatahub/opendatahub-operator-controller-manager-d84699fcf-kk7qz"
Jul 29 19:22:55.737320 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:55.737181    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f034f84d-5306-43a8-914d-0a91c0d60b4e-manifests\") pod \"opendatahub-operator-controller-manager-d84699fcf-kk7qz\" (UID: \"f034f84d-5306-43a8-914d-0a91c0d60b4e\") " pod="opendatahub/opendatahub-operator-controller-manager-d84699fcf-kk7qz"
Jul 29 19:22:55.737320 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:55.737211    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f034f84d-5306-43a8-914d-0a91c0d60b4e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d84699fcf-kk7qz\" (UID: \"f034f84d-5306-43a8-914d-0a91c0d60b4e\") " pod="opendatahub/opendatahub-operator-controller-manager-d84699fcf-kk7qz"
Jul 29 19:22:55.737320 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:55.737239    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f034f84d-5306-43a8-914d-0a91c0d60b4e-webhook-cert\") pod \"opendatahub-operator-controller-manager-d84699fcf-kk7qz\" (UID: \"f034f84d-5306-43a8-914d-0a91c0d60b4e\") " pod="opendatahub/opendatahub-operator-controller-manager-d84699fcf-kk7qz"
Jul 29 19:22:55.737320 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:55.737257    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxv86\" (UniqueName: \"kubernetes.io/projected/f034f84d-5306-43a8-914d-0a91c0d60b4e-kube-api-access-qxv86\") pod \"opendatahub-operator-controller-manager-d84699fcf-kk7qz\" (UID: \"f034f84d-5306-43a8-914d-0a91c0d60b4e\") " pod="opendatahub/opendatahub-operator-controller-manager-d84699fcf-kk7qz"
Jul 29 19:22:55.737694 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:55.737668    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f034f84d-5306-43a8-914d-0a91c0d60b4e-manifests\") pod \"opendatahub-operator-controller-manager-d84699fcf-kk7qz\" (UID: \"f034f84d-5306-43a8-914d-0a91c0d60b4e\") " pod="opendatahub/opendatahub-operator-controller-manager-d84699fcf-kk7qz"
Jul 29 19:22:55.739673 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:55.739651    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f034f84d-5306-43a8-914d-0a91c0d60b4e-webhook-cert\") pod \"opendatahub-operator-controller-manager-d84699fcf-kk7qz\" (UID: \"f034f84d-5306-43a8-914d-0a91c0d60b4e\") " pod="opendatahub/opendatahub-operator-controller-manager-d84699fcf-kk7qz"
Jul 29 19:22:55.739756 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:55.739631    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f034f84d-5306-43a8-914d-0a91c0d60b4e-tmp\") pod \"opendatahub-operator-controller-manager-d84699fcf-kk7qz\" (UID: \"f034f84d-5306-43a8-914d-0a91c0d60b4e\") " pod="opendatahub/opendatahub-operator-controller-manager-d84699fcf-kk7qz"
Jul 29 19:22:55.739756 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:55.739689    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f034f84d-5306-43a8-914d-0a91c0d60b4e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d84699fcf-kk7qz\" (UID: \"f034f84d-5306-43a8-914d-0a91c0d60b4e\") " pod="opendatahub/opendatahub-operator-controller-manager-d84699fcf-kk7qz"
Jul 29 19:22:55.746450 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:55.746426    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxv86\" (UniqueName: \"kubernetes.io/projected/f034f84d-5306-43a8-914d-0a91c0d60b4e-kube-api-access-qxv86\") pod \"opendatahub-operator-controller-manager-d84699fcf-kk7qz\" (UID: \"f034f84d-5306-43a8-914d-0a91c0d60b4e\") " pod="opendatahub/opendatahub-operator-controller-manager-d84699fcf-kk7qz"
Jul 29 19:22:55.861307 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:55.861234    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d84699fcf-kk7qz"
Jul 29 19:22:55.997830 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:55.997800    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d84699fcf-kk7qz"]
Jul 29 19:22:56.000258 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:22:56.000232    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf034f84d_5306_43a8_914d_0a91c0d60b4e.slice/crio-fd6a9f6b059cf2b32617f5c5476b559c1c9d40b3086cadf19e8663cf5b4f10c4 WatchSource:0}: Error finding container fd6a9f6b059cf2b32617f5c5476b559c1c9d40b3086cadf19e8663cf5b4f10c4: Status 404 returned error can't find the container with id fd6a9f6b059cf2b32617f5c5476b559c1c9d40b3086cadf19e8663cf5b4f10c4
Jul 29 19:22:56.225731 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:56.225697    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d84699fcf-kk7qz" event={"ID":"f034f84d-5306-43a8-914d-0a91c0d60b4e","Type":"ContainerStarted","Data":"fd6a9f6b059cf2b32617f5c5476b559c1c9d40b3086cadf19e8663cf5b4f10c4"}
Jul 29 19:22:59.241185 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:22:59.241151    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d84699fcf-kk7qz" event={"ID":"f034f84d-5306-43a8-914d-0a91c0d60b4e","Type":"ContainerStarted","Data":"46025b22ba22022cccd37cf64de51e55c19aa73225a1fb7b9d26f8e6b9667f8e"}
Jul 29 19:23:00.246639 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:23:00.246549    2557 generic.go:356] "Generic (PLEG): container finished" podID="f034f84d-5306-43a8-914d-0a91c0d60b4e" containerID="46025b22ba22022cccd37cf64de51e55c19aa73225a1fb7b9d26f8e6b9667f8e" exitCode=0
Jul 29 19:23:00.247017 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:23:00.246639    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d84699fcf-kk7qz" event={"ID":"f034f84d-5306-43a8-914d-0a91c0d60b4e","Type":"ContainerDied","Data":"46025b22ba22022cccd37cf64de51e55c19aa73225a1fb7b9d26f8e6b9667f8e"}
Jul 29 19:23:01.253771 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:23:01.253737    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d84699fcf-kk7qz" event={"ID":"f034f84d-5306-43a8-914d-0a91c0d60b4e","Type":"ContainerStarted","Data":"d73314180764be35acfecc812588fc92c6310f58432899a1a4199b93c42137c1"}
Jul 29 19:23:01.254180 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:23:01.253886    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-d84699fcf-kk7qz"
Jul 29 19:23:06.227901 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:23:06.227870    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-ffdbfbbc8-nd2q7"
Jul 29 19:23:06.249095 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:23:06.249047    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-d84699fcf-kk7qz" podStartSLOduration=6.961205542 podStartE2EDuration="11.249029506s" totalImagesPullingTime="3.261279789s" totalInitContainerRuntime="1.026544175s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:22:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:23:01.291728759 +0000 UTC m=+352.119227818" watchObservedRunningTime="2026-07-29 19:23:06.249029506 +0000 UTC m=+357.076528564"
Jul 29 19:23:12.261601 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:23:12.261568    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-d84699fcf-kk7qz"
Jul 29 19:23:21.710520 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:23:21.710484    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6d594574db-l29np"]
Jul 29 19:23:21.720936 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:23:21.720912    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6d594574db-l29np"
Jul 29 19:23:21.737221 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:23:21.737199    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 29 19:23:21.737728 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:23:21.737705    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-lgfbs\""
Jul 29 19:23:21.737841 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:23:21.737736    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 29 19:23:21.748552 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:23:21.748522    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6d594574db-l29np"]
Jul 29 19:23:21.852917 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:23:21.852869    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3518470c-c9c6-4fce-b7ee-399de7938787-tmp\") pod \"kube-auth-proxy-6d594574db-l29np\" (UID: \"3518470c-c9c6-4fce-b7ee-399de7938787\") " pod="openshift-ingress/kube-auth-proxy-6d594574db-l29np"
Jul 29 19:23:21.853111 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:23:21.853012    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3518470c-c9c6-4fce-b7ee-399de7938787-tls-certs\") pod \"kube-auth-proxy-6d594574db-l29np\" (UID: \"3518470c-c9c6-4fce-b7ee-399de7938787\") " pod="openshift-ingress/kube-auth-proxy-6d594574db-l29np"
Jul 29 19:23:21.853111 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:23:21.853041    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzw59\" (UniqueName: \"kubernetes.io/projected/3518470c-c9c6-4fce-b7ee-399de7938787-kube-api-access-nzw59\") pod \"kube-auth-proxy-6d594574db-l29np\" (UID: \"3518470c-c9c6-4fce-b7ee-399de7938787\") " pod="openshift-ingress/kube-auth-proxy-6d594574db-l29np"
Jul 29 19:23:21.954413 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:23:21.954372    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3518470c-c9c6-4fce-b7ee-399de7938787-tmp\") pod \"kube-auth-proxy-6d594574db-l29np\" (UID: \"3518470c-c9c6-4fce-b7ee-399de7938787\") " pod="openshift-ingress/kube-auth-proxy-6d594574db-l29np"
Jul 29 19:23:21.954610 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:23:21.954538    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3518470c-c9c6-4fce-b7ee-399de7938787-tls-certs\") pod \"kube-auth-proxy-6d594574db-l29np\" (UID: \"3518470c-c9c6-4fce-b7ee-399de7938787\") " pod="openshift-ingress/kube-auth-proxy-6d594574db-l29np"
Jul 29 19:23:21.954610 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:23:21.954562    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzw59\" (UniqueName: \"kubernetes.io/projected/3518470c-c9c6-4fce-b7ee-399de7938787-kube-api-access-nzw59\") pod \"kube-auth-proxy-6d594574db-l29np\" (UID: \"3518470c-c9c6-4fce-b7ee-399de7938787\") " pod="openshift-ingress/kube-auth-proxy-6d594574db-l29np"
Jul 29 19:23:21.956690 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:23:21.956658    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3518470c-c9c6-4fce-b7ee-399de7938787-tmp\") pod \"kube-auth-proxy-6d594574db-l29np\" (UID: \"3518470c-c9c6-4fce-b7ee-399de7938787\") " pod="openshift-ingress/kube-auth-proxy-6d594574db-l29np"
Jul 29 19:23:21.956884 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:23:21.956865    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3518470c-c9c6-4fce-b7ee-399de7938787-tls-certs\") pod \"kube-auth-proxy-6d594574db-l29np\" (UID: \"3518470c-c9c6-4fce-b7ee-399de7938787\") " pod="openshift-ingress/kube-auth-proxy-6d594574db-l29np"
Jul 29 19:23:21.972038 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:23:21.971959    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzw59\" (UniqueName: \"kubernetes.io/projected/3518470c-c9c6-4fce-b7ee-399de7938787-kube-api-access-nzw59\") pod \"kube-auth-proxy-6d594574db-l29np\" (UID: \"3518470c-c9c6-4fce-b7ee-399de7938787\") " pod="openshift-ingress/kube-auth-proxy-6d594574db-l29np"
Jul 29 19:23:22.030534 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:23:22.030498    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6d594574db-l29np"
Jul 29 19:23:22.185688 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:23:22.185664    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6d594574db-l29np"]
Jul 29 19:23:22.188286 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:23:22.188261    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3518470c_c9c6_4fce_b7ee_399de7938787.slice/crio-85b7bbc36c70a9cb7e7ceda000d5733edd092e61f10a4ace29a8aab2c4ce92b4 WatchSource:0}: Error finding container 85b7bbc36c70a9cb7e7ceda000d5733edd092e61f10a4ace29a8aab2c4ce92b4: Status 404 returned error can't find the container with id 85b7bbc36c70a9cb7e7ceda000d5733edd092e61f10a4ace29a8aab2c4ce92b4
Jul 29 19:23:22.341512 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:23:22.341427    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6d594574db-l29np" event={"ID":"3518470c-c9c6-4fce-b7ee-399de7938787","Type":"ContainerStarted","Data":"85b7bbc36c70a9cb7e7ceda000d5733edd092e61f10a4ace29a8aab2c4ce92b4"}
Jul 29 19:23:25.357074 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:23:25.357035    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6d594574db-l29np" event={"ID":"3518470c-c9c6-4fce-b7ee-399de7938787","Type":"ContainerStarted","Data":"80768fb6d095a9cc1102e356865e6634ae6c6b9d78ea87c8afe283b6723c1a50"}
Jul 29 19:23:25.382638 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:23:25.382587    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6d594574db-l29np" podStartSLOduration=2.105487777 podStartE2EDuration="4.382569467s" totalImagesPullingTime="2.27708169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:23:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:23:25.380399867 +0000 UTC m=+376.207898926" watchObservedRunningTime="2026-07-29 19:23:25.382569467 +0000 UTC m=+376.210068525"
Jul 29 19:24:58.819475 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:24:58.819433    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9bw6q"]
Jul 29 19:24:58.821879 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:24:58.821862    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9bw6q"
Jul 29 19:24:58.826204 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:24:58.826181    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 29 19:24:58.826529 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:24:58.826511    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 29 19:24:58.826639 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:24:58.826600    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 29 19:24:58.826639 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:24:58.826601    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-w9774\""
Jul 29 19:24:58.826765 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:24:58.826671    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 29 19:24:58.835174 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:24:58.835150    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9bw6q"]
Jul 29 19:24:58.974776 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:24:58.974737    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f43acfbd-67be-4d90-9985-e3b3382274c1-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9bw6q\" (UID: \"f43acfbd-67be-4d90-9985-e3b3382274c1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9bw6q"
Jul 29 19:24:58.974954 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:24:58.974792    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8mf5\" (UniqueName: \"kubernetes.io/projected/f43acfbd-67be-4d90-9985-e3b3382274c1-kube-api-access-v8mf5\") pod \"kuadrant-console-plugin-66b5774985-9bw6q\" (UID: \"f43acfbd-67be-4d90-9985-e3b3382274c1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9bw6q"
Jul 29 19:24:58.974954 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:24:58.974905    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f43acfbd-67be-4d90-9985-e3b3382274c1-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9bw6q\" (UID: \"f43acfbd-67be-4d90-9985-e3b3382274c1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9bw6q"
Jul 29 19:24:59.076302 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:24:59.076208    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f43acfbd-67be-4d90-9985-e3b3382274c1-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9bw6q\" (UID: \"f43acfbd-67be-4d90-9985-e3b3382274c1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9bw6q"
Jul 29 19:24:59.076302 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:24:59.076258    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f43acfbd-67be-4d90-9985-e3b3382274c1-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9bw6q\" (UID: \"f43acfbd-67be-4d90-9985-e3b3382274c1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9bw6q"
Jul 29 19:24:59.076302 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:24:59.076301    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8mf5\" (UniqueName: \"kubernetes.io/projected/f43acfbd-67be-4d90-9985-e3b3382274c1-kube-api-access-v8mf5\") pod \"kuadrant-console-plugin-66b5774985-9bw6q\" (UID: \"f43acfbd-67be-4d90-9985-e3b3382274c1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9bw6q"
Jul 29 19:24:59.077002 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:24:59.076980    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f43acfbd-67be-4d90-9985-e3b3382274c1-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9bw6q\" (UID: \"f43acfbd-67be-4d90-9985-e3b3382274c1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9bw6q"
Jul 29 19:24:59.078586 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:24:59.078562    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f43acfbd-67be-4d90-9985-e3b3382274c1-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9bw6q\" (UID: \"f43acfbd-67be-4d90-9985-e3b3382274c1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9bw6q"
Jul 29 19:24:59.088837 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:24:59.088806    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8mf5\" (UniqueName: \"kubernetes.io/projected/f43acfbd-67be-4d90-9985-e3b3382274c1-kube-api-access-v8mf5\") pod \"kuadrant-console-plugin-66b5774985-9bw6q\" (UID: \"f43acfbd-67be-4d90-9985-e3b3382274c1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9bw6q"
Jul 29 19:24:59.131835 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:24:59.131805    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9bw6q"
Jul 29 19:24:59.257284 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:24:59.257260    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9bw6q"]
Jul 29 19:24:59.259272 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:24:59.259249    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf43acfbd_67be_4d90_9985_e3b3382274c1.slice/crio-dd582e555f185e8d8efae1e6adc4e0e85e91a9d2969e77bf3e4a45d9c7776362 WatchSource:0}: Error finding container dd582e555f185e8d8efae1e6adc4e0e85e91a9d2969e77bf3e4a45d9c7776362: Status 404 returned error can't find the container with id dd582e555f185e8d8efae1e6adc4e0e85e91a9d2969e77bf3e4a45d9c7776362
Jul 29 19:24:59.750176 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:24:59.750137    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9bw6q" event={"ID":"f43acfbd-67be-4d90-9985-e3b3382274c1","Type":"ContainerStarted","Data":"dd582e555f185e8d8efae1e6adc4e0e85e91a9d2969e77bf3e4a45d9c7776362"}
Jul 29 19:25:23.863134 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:23.863089    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9bw6q" event={"ID":"f43acfbd-67be-4d90-9985-e3b3382274c1","Type":"ContainerStarted","Data":"f64ae747ff1f568e657d333d2bb99bc3e12947351364f0688e987e4ed48c7453"}
Jul 29 19:25:23.887723 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:23.887678    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9bw6q" podStartSLOduration=2.388741858 podStartE2EDuration="25.887663366s" totalImagesPullingTime="23.498921508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:24:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:25:23.88709442 +0000 UTC m=+494.714593480" watchObservedRunningTime="2026-07-29 19:25:23.887663366 +0000 UTC m=+494.715162449"
Jul 29 19:25:41.730537 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:41.730505    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b57854b96-s5rfx"]
Jul 29 19:25:41.733259 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:41.733242    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b57854b96-s5rfx"
Jul 29 19:25:41.737349 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:41.737319    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 29 19:25:41.737479 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:41.737378    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 29 19:25:41.737576 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:41.737562    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-v8zkg\""
Jul 29 19:25:41.737668 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:41.737602    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 29 19:25:41.737724 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:41.737659    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 29 19:25:41.737724 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:41.737706    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 29 19:25:41.746142 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:41.746000    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 29 19:25:41.752318 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:41.752293    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b57854b96-s5rfx"]
Jul 29 19:25:41.856916 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:41.856882    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec-console-config\") pod \"console-5b57854b96-s5rfx\" (UID: \"ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec\") " pod="openshift-console/console-5b57854b96-s5rfx"
Jul 29 19:25:41.857088 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:41.856922    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec-trusted-ca-bundle\") pod \"console-5b57854b96-s5rfx\" (UID: \"ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec\") " pod="openshift-console/console-5b57854b96-s5rfx"
Jul 29 19:25:41.857088 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:41.856955    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec-console-oauth-config\") pod \"console-5b57854b96-s5rfx\" (UID: \"ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec\") " pod="openshift-console/console-5b57854b96-s5rfx"
Jul 29 19:25:41.857088 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:41.857026    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec-service-ca\") pod \"console-5b57854b96-s5rfx\" (UID: \"ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec\") " pod="openshift-console/console-5b57854b96-s5rfx"
Jul 29 19:25:41.857088 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:41.857082    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vnjt\" (UniqueName: \"kubernetes.io/projected/ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec-kube-api-access-4vnjt\") pod \"console-5b57854b96-s5rfx\" (UID: \"ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec\") " pod="openshift-console/console-5b57854b96-s5rfx"
Jul 29 19:25:41.857233 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:41.857132    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec-console-serving-cert\") pod \"console-5b57854b96-s5rfx\" (UID: \"ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec\") " pod="openshift-console/console-5b57854b96-s5rfx"
Jul 29 19:25:41.857233 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:41.857150    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec-oauth-serving-cert\") pod \"console-5b57854b96-s5rfx\" (UID: \"ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec\") " pod="openshift-console/console-5b57854b96-s5rfx"
Jul 29 19:25:41.958338 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:41.958306    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec-console-config\") pod \"console-5b57854b96-s5rfx\" (UID: \"ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec\") " pod="openshift-console/console-5b57854b96-s5rfx"
Jul 29 19:25:41.958338 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:41.958342    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec-trusted-ca-bundle\") pod \"console-5b57854b96-s5rfx\" (UID: \"ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec\") " pod="openshift-console/console-5b57854b96-s5rfx"
Jul 29 19:25:41.958610 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:41.958365    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec-console-oauth-config\") pod \"console-5b57854b96-s5rfx\" (UID: \"ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec\") " pod="openshift-console/console-5b57854b96-s5rfx"
Jul 29 19:25:41.958610 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:41.958383    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec-service-ca\") pod \"console-5b57854b96-s5rfx\" (UID: \"ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec\") " pod="openshift-console/console-5b57854b96-s5rfx"
Jul 29 19:25:41.958610 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:41.958401    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vnjt\" (UniqueName: \"kubernetes.io/projected/ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec-kube-api-access-4vnjt\") pod \"console-5b57854b96-s5rfx\" (UID: \"ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec\") " pod="openshift-console/console-5b57854b96-s5rfx"
Jul 29 19:25:41.958610 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:41.958447    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec-console-serving-cert\") pod \"console-5b57854b96-s5rfx\" (UID: \"ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec\") " pod="openshift-console/console-5b57854b96-s5rfx"
Jul 29 19:25:41.958610 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:41.958465    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec-oauth-serving-cert\") pod \"console-5b57854b96-s5rfx\" (UID: \"ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec\") " pod="openshift-console/console-5b57854b96-s5rfx"
Jul 29 19:25:41.959222 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:41.959194    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec-console-config\") pod \"console-5b57854b96-s5rfx\" (UID: \"ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec\") " pod="openshift-console/console-5b57854b96-s5rfx"
Jul 29 19:25:41.959343 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:41.959226    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec-service-ca\") pod \"console-5b57854b96-s5rfx\" (UID: \"ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec\") " pod="openshift-console/console-5b57854b96-s5rfx"
Jul 29 19:25:41.959343 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:41.959250    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec-oauth-serving-cert\") pod \"console-5b57854b96-s5rfx\" (UID: \"ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec\") " pod="openshift-console/console-5b57854b96-s5rfx"
Jul 29 19:25:41.959343 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:41.959328    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec-trusted-ca-bundle\") pod \"console-5b57854b96-s5rfx\" (UID: \"ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec\") " pod="openshift-console/console-5b57854b96-s5rfx"
Jul 29 19:25:41.960815 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:41.960787    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec-console-oauth-config\") pod \"console-5b57854b96-s5rfx\" (UID: \"ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec\") " pod="openshift-console/console-5b57854b96-s5rfx"
Jul 29 19:25:41.960920 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:41.960879    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec-console-serving-cert\") pod \"console-5b57854b96-s5rfx\" (UID: \"ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec\") " pod="openshift-console/console-5b57854b96-s5rfx"
Jul 29 19:25:41.968473 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:41.968447    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vnjt\" (UniqueName: \"kubernetes.io/projected/ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec-kube-api-access-4vnjt\") pod \"console-5b57854b96-s5rfx\" (UID: \"ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec\") " pod="openshift-console/console-5b57854b96-s5rfx"
Jul 29 19:25:42.047520 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:42.047441    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b57854b96-s5rfx"
Jul 29 19:25:42.175416 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:42.175384    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b57854b96-s5rfx"]
Jul 29 19:25:42.178245 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:25:42.178219    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef5d5ef6_6f90_4b28_afeb_1487cfa7a4ec.slice/crio-39cc3f9bc025b83ed9545592ed29ee82942ede265a1402c21223ed7912347a4e WatchSource:0}: Error finding container 39cc3f9bc025b83ed9545592ed29ee82942ede265a1402c21223ed7912347a4e: Status 404 returned error can't find the container with id 39cc3f9bc025b83ed9545592ed29ee82942ede265a1402c21223ed7912347a4e
Jul 29 19:25:42.938895 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:42.938848    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b57854b96-s5rfx" event={"ID":"ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec","Type":"ContainerStarted","Data":"c72c992537c6917937444ea1aa1f8f3ec910bc28e67ff35f55117fb4fd0fe7b2"}
Jul 29 19:25:42.939336 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:42.938913    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b57854b96-s5rfx" event={"ID":"ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec","Type":"ContainerStarted","Data":"39cc3f9bc025b83ed9545592ed29ee82942ede265a1402c21223ed7912347a4e"}
Jul 29 19:25:42.961073 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:42.961024    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b57854b96-s5rfx" podStartSLOduration=1.961008075 podStartE2EDuration="1.961008075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:25:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:25:42.959784545 +0000 UTC m=+513.787283604" watchObservedRunningTime="2026-07-29 19:25:42.961008075 +0000 UTC m=+513.788507133"
Jul 29 19:25:45.247538 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:45.247505    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2zhvh"]
Jul 29 19:25:45.250850 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:45.250832    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-2zhvh"
Jul 29 19:25:45.253717 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:45.253690    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 29 19:25:45.261583 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:45.261556    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2zhvh"]
Jul 29 19:25:45.333643 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:45.333611    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2zhvh"]
Jul 29 19:25:45.386026 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:45.385701    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9f7a1529-02ee-4cd8-af4c-43724bd6c72b-config-file\") pod \"limitador-limitador-64c94fd64f-2zhvh\" (UID: \"9f7a1529-02ee-4cd8-af4c-43724bd6c72b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2zhvh"
Jul 29 19:25:45.386026 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:45.385787    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdz72\" (UniqueName: \"kubernetes.io/projected/9f7a1529-02ee-4cd8-af4c-43724bd6c72b-kube-api-access-pdz72\") pod \"limitador-limitador-64c94fd64f-2zhvh\" (UID: \"9f7a1529-02ee-4cd8-af4c-43724bd6c72b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2zhvh"
Jul 29 19:25:45.486639 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:45.486602    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9f7a1529-02ee-4cd8-af4c-43724bd6c72b-config-file\") pod \"limitador-limitador-64c94fd64f-2zhvh\" (UID: \"9f7a1529-02ee-4cd8-af4c-43724bd6c72b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2zhvh"
Jul 29 19:25:45.486797 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:45.486763    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdz72\" (UniqueName: \"kubernetes.io/projected/9f7a1529-02ee-4cd8-af4c-43724bd6c72b-kube-api-access-pdz72\") pod \"limitador-limitador-64c94fd64f-2zhvh\" (UID: \"9f7a1529-02ee-4cd8-af4c-43724bd6c72b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2zhvh"
Jul 29 19:25:45.487244 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:45.487223    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9f7a1529-02ee-4cd8-af4c-43724bd6c72b-config-file\") pod \"limitador-limitador-64c94fd64f-2zhvh\" (UID: \"9f7a1529-02ee-4cd8-af4c-43724bd6c72b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2zhvh"
Jul 29 19:25:45.495670 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:45.495647    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdz72\" (UniqueName: \"kubernetes.io/projected/9f7a1529-02ee-4cd8-af4c-43724bd6c72b-kube-api-access-pdz72\") pod \"limitador-limitador-64c94fd64f-2zhvh\" (UID: \"9f7a1529-02ee-4cd8-af4c-43724bd6c72b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2zhvh"
Jul 29 19:25:45.562117 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:45.562044    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-2zhvh"
Jul 29 19:25:45.691596 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:45.691572    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2zhvh"]
Jul 29 19:25:45.693993 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:25:45.693952    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f7a1529_02ee_4cd8_af4c_43724bd6c72b.slice/crio-ef1a062890a972f1a106491f375deefe7be7bca6c40e5b0830ed6df2fddc8f9b WatchSource:0}: Error finding container ef1a062890a972f1a106491f375deefe7be7bca6c40e5b0830ed6df2fddc8f9b: Status 404 returned error can't find the container with id ef1a062890a972f1a106491f375deefe7be7bca6c40e5b0830ed6df2fddc8f9b
Jul 29 19:25:45.952088 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:45.952052    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-2zhvh" event={"ID":"9f7a1529-02ee-4cd8-af4c-43724bd6c72b","Type":"ContainerStarted","Data":"ef1a062890a972f1a106491f375deefe7be7bca6c40e5b0830ed6df2fddc8f9b"}
Jul 29 19:25:46.160371 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:46.160337    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-wpk2s"]
Jul 29 19:25:46.164672 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:46.164650    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-wpk2s"
Jul 29 19:25:46.167730 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:46.167709    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-fl666\""
Jul 29 19:25:46.170925 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:46.170894    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wpk2s"]
Jul 29 19:25:46.294431 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:46.294351    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btjsf\" (UniqueName: \"kubernetes.io/projected/077d7ce0-5531-4444-8c32-c7aa24088759-kube-api-access-btjsf\") pod \"authorino-75d556bd77-wpk2s\" (UID: \"077d7ce0-5531-4444-8c32-c7aa24088759\") " pod="kuadrant-system/authorino-75d556bd77-wpk2s"
Jul 29 19:25:46.395911 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:46.395877    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btjsf\" (UniqueName: \"kubernetes.io/projected/077d7ce0-5531-4444-8c32-c7aa24088759-kube-api-access-btjsf\") pod \"authorino-75d556bd77-wpk2s\" (UID: \"077d7ce0-5531-4444-8c32-c7aa24088759\") " pod="kuadrant-system/authorino-75d556bd77-wpk2s"
Jul 29 19:25:46.406375 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:46.406345    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btjsf\" (UniqueName: \"kubernetes.io/projected/077d7ce0-5531-4444-8c32-c7aa24088759-kube-api-access-btjsf\") pod \"authorino-75d556bd77-wpk2s\" (UID: \"077d7ce0-5531-4444-8c32-c7aa24088759\") " pod="kuadrant-system/authorino-75d556bd77-wpk2s"
Jul 29 19:25:46.474570 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:46.474534    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-wpk2s"
Jul 29 19:25:46.657028 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:46.656999    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wpk2s"]
Jul 29 19:25:46.958537 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:46.958489    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-wpk2s" event={"ID":"077d7ce0-5531-4444-8c32-c7aa24088759","Type":"ContainerStarted","Data":"b756098f75552174fbf3a870c8f395a10291dbca7fdd6902ac03a2dc3e913c9b"}
Jul 29 19:25:48.971862 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:48.971822    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-2zhvh" event={"ID":"9f7a1529-02ee-4cd8-af4c-43724bd6c72b","Type":"ContainerStarted","Data":"f351fe7c77ba8cc7a370f788587818d2a6d5445e9149b6419cfeb8b6412c6fdc"}
Jul 29 19:25:48.972364 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:48.971918    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-2zhvh"
Jul 29 19:25:51.988783 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:51.988744    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-wpk2s" event={"ID":"077d7ce0-5531-4444-8c32-c7aa24088759","Type":"ContainerStarted","Data":"15155f484bc1fa94361d678be79fe6acc90734ece462b09aa6e372619b7db338"}
Jul 29 19:25:52.004633 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:52.004589    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-wpk2s" podStartSLOduration=1.38749295 podStartE2EDuration="6.00457455s" totalImagesPullingTime="4.6170816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:25:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:25:52.004409011 +0000 UTC m=+522.831908091" watchObservedRunningTime="2026-07-29 19:25:52.00457455 +0000 UTC m=+522.832073607"
Jul 29 19:25:52.005121 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:52.005095    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-2zhvh" podStartSLOduration=4.196534606 podStartE2EDuration="7.005087441s" totalImagesPullingTime="2.808552835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:25:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:25:48.995002777 +0000 UTC m=+519.822501836" watchObservedRunningTime="2026-07-29 19:25:52.005087441 +0000 UTC m=+522.832586535"
Jul 29 19:25:52.047637 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:52.047603    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b57854b96-s5rfx"
Jul 29 19:25:52.047784 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:52.047667    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b57854b96-s5rfx"
Jul 29 19:25:52.052298 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:52.052271    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b57854b96-s5rfx"
Jul 29 19:25:52.998172 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:52.998133    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b57854b96-s5rfx"
Jul 29 19:25:59.978555 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:25:59.978525    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-2zhvh"
Jul 29 19:26:02.306616 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:02.306536    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2zhvh"]
Jul 29 19:26:02.307023 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:02.306728    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-2zhvh" podUID="9f7a1529-02ee-4cd8-af4c-43724bd6c72b" containerName="limitador" containerID="cri-o://f351fe7c77ba8cc7a370f788587818d2a6d5445e9149b6419cfeb8b6412c6fdc" gracePeriod=30
Jul 29 19:26:02.851859 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:02.851835    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-2zhvh"
Jul 29 19:26:03.040244 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:03.040203    2557 generic.go:356] "Generic (PLEG): container finished" podID="9f7a1529-02ee-4cd8-af4c-43724bd6c72b" containerID="f351fe7c77ba8cc7a370f788587818d2a6d5445e9149b6419cfeb8b6412c6fdc" exitCode=0
Jul 29 19:26:03.040244 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:03.040252    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-2zhvh"
Jul 29 19:26:03.040449 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:03.040293    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-2zhvh" event={"ID":"9f7a1529-02ee-4cd8-af4c-43724bd6c72b","Type":"ContainerDied","Data":"f351fe7c77ba8cc7a370f788587818d2a6d5445e9149b6419cfeb8b6412c6fdc"}
Jul 29 19:26:03.040449 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:03.040357    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-2zhvh" event={"ID":"9f7a1529-02ee-4cd8-af4c-43724bd6c72b","Type":"ContainerDied","Data":"ef1a062890a972f1a106491f375deefe7be7bca6c40e5b0830ed6df2fddc8f9b"}
Jul 29 19:26:03.040449 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:03.040379    2557 scope.go:117] "RemoveContainer" containerID="f351fe7c77ba8cc7a370f788587818d2a6d5445e9149b6419cfeb8b6412c6fdc"
Jul 29 19:26:03.042537 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:03.042512    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdz72\" (UniqueName: \"kubernetes.io/projected/9f7a1529-02ee-4cd8-af4c-43724bd6c72b-kube-api-access-pdz72\") pod \"9f7a1529-02ee-4cd8-af4c-43724bd6c72b\" (UID: \"9f7a1529-02ee-4cd8-af4c-43724bd6c72b\") "
Jul 29 19:26:03.042667 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:03.042576    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9f7a1529-02ee-4cd8-af4c-43724bd6c72b-config-file\") pod \"9f7a1529-02ee-4cd8-af4c-43724bd6c72b\" (UID: \"9f7a1529-02ee-4cd8-af4c-43724bd6c72b\") "
Jul 29 19:26:03.042916 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:03.042894    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f7a1529-02ee-4cd8-af4c-43724bd6c72b-config-file" (OuterVolumeSpecName: "config-file") pod "9f7a1529-02ee-4cd8-af4c-43724bd6c72b" (UID: "9f7a1529-02ee-4cd8-af4c-43724bd6c72b"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:26:03.044647 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:03.044616    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f7a1529-02ee-4cd8-af4c-43724bd6c72b-kube-api-access-pdz72" (OuterVolumeSpecName: "kube-api-access-pdz72") pod "9f7a1529-02ee-4cd8-af4c-43724bd6c72b" (UID: "9f7a1529-02ee-4cd8-af4c-43724bd6c72b"). InnerVolumeSpecName "kube-api-access-pdz72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:26:03.049017 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:03.048995    2557 scope.go:117] "RemoveContainer" containerID="f351fe7c77ba8cc7a370f788587818d2a6d5445e9149b6419cfeb8b6412c6fdc"
Jul 29 19:26:03.049283 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:26:03.049260    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f351fe7c77ba8cc7a370f788587818d2a6d5445e9149b6419cfeb8b6412c6fdc\": container with ID starting with f351fe7c77ba8cc7a370f788587818d2a6d5445e9149b6419cfeb8b6412c6fdc not found: ID does not exist" containerID="f351fe7c77ba8cc7a370f788587818d2a6d5445e9149b6419cfeb8b6412c6fdc"
Jul 29 19:26:03.049332 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:03.049292    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f351fe7c77ba8cc7a370f788587818d2a6d5445e9149b6419cfeb8b6412c6fdc"} err="failed to get container status \"f351fe7c77ba8cc7a370f788587818d2a6d5445e9149b6419cfeb8b6412c6fdc\": rpc error: code = NotFound desc = could not find container \"f351fe7c77ba8cc7a370f788587818d2a6d5445e9149b6419cfeb8b6412c6fdc\": container with ID starting with f351fe7c77ba8cc7a370f788587818d2a6d5445e9149b6419cfeb8b6412c6fdc not found: ID does not exist"
Jul 29 19:26:03.144115 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:03.144085    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdz72\" (UniqueName: \"kubernetes.io/projected/9f7a1529-02ee-4cd8-af4c-43724bd6c72b-kube-api-access-pdz72\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:26:03.144115 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:03.144110    2557 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9f7a1529-02ee-4cd8-af4c-43724bd6c72b-config-file\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:26:03.362445 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:03.362412    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2zhvh"]
Jul 29 19:26:03.366455 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:03.366431    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2zhvh"]
Jul 29 19:26:03.678089 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:03.678059    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f7a1529-02ee-4cd8-af4c-43724bd6c72b" path="/var/lib/kubelet/pods/9f7a1529-02ee-4cd8-af4c-43724bd6c72b/volumes"
Jul 29 19:26:06.948729 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:06.948691    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-cwr5x"]
Jul 29 19:26:06.949107 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:06.949026    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f7a1529-02ee-4cd8-af4c-43724bd6c72b" containerName="limitador"
Jul 29 19:26:06.949107 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:06.949040    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f7a1529-02ee-4cd8-af4c-43724bd6c72b" containerName="limitador"
Jul 29 19:26:06.949107 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:06.949102    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f7a1529-02ee-4cd8-af4c-43724bd6c72b" containerName="limitador"
Jul 29 19:26:06.953097 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:06.953074    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cwr5x"
Jul 29 19:26:06.956033 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:06.956009    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 19:26:06.956166 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:06.956045    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 29 19:26:06.956166 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:06.956096    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 19:26:06.957250 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:06.957234    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-n8vkx\""
Jul 29 19:26:06.961559 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:06.961535    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-cwr5x"]
Jul 29 19:26:06.975068 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:06.975040    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2aa53dc2-24c4-4801-bbca-b84ac909f19e-data\") pod \"postgres-69fb4594cf-cwr5x\" (UID: \"2aa53dc2-24c4-4801-bbca-b84ac909f19e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-cwr5x"
Jul 29 19:26:06.975177 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:06.975120    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46k4p\" (UniqueName: \"kubernetes.io/projected/2aa53dc2-24c4-4801-bbca-b84ac909f19e-kube-api-access-46k4p\") pod \"postgres-69fb4594cf-cwr5x\" (UID: \"2aa53dc2-24c4-4801-bbca-b84ac909f19e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-cwr5x"
Jul 29 19:26:07.075818 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:07.075781    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2aa53dc2-24c4-4801-bbca-b84ac909f19e-data\") pod \"postgres-69fb4594cf-cwr5x\" (UID: \"2aa53dc2-24c4-4801-bbca-b84ac909f19e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-cwr5x"
Jul 29 19:26:07.076032 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:07.075845    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46k4p\" (UniqueName: \"kubernetes.io/projected/2aa53dc2-24c4-4801-bbca-b84ac909f19e-kube-api-access-46k4p\") pod \"postgres-69fb4594cf-cwr5x\" (UID: \"2aa53dc2-24c4-4801-bbca-b84ac909f19e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-cwr5x"
Jul 29 19:26:07.076237 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:07.076216    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2aa53dc2-24c4-4801-bbca-b84ac909f19e-data\") pod \"postgres-69fb4594cf-cwr5x\" (UID: \"2aa53dc2-24c4-4801-bbca-b84ac909f19e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-cwr5x"
Jul 29 19:26:07.084119 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:07.084097    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46k4p\" (UniqueName: \"kubernetes.io/projected/2aa53dc2-24c4-4801-bbca-b84ac909f19e-kube-api-access-46k4p\") pod \"postgres-69fb4594cf-cwr5x\" (UID: \"2aa53dc2-24c4-4801-bbca-b84ac909f19e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-cwr5x"
Jul 29 19:26:07.263483 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:07.263384    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cwr5x"
Jul 29 19:26:07.393875 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:07.393847    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-cwr5x"]
Jul 29 19:26:07.395891 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:26:07.395860    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aa53dc2_24c4_4801_bbca_b84ac909f19e.slice/crio-c821876691ed5d14f23589fed6b069d3d15d07239e8a95eee0a2cd3a39757e34 WatchSource:0}: Error finding container c821876691ed5d14f23589fed6b069d3d15d07239e8a95eee0a2cd3a39757e34: Status 404 returned error can't find the container with id c821876691ed5d14f23589fed6b069d3d15d07239e8a95eee0a2cd3a39757e34
Jul 29 19:26:08.065323 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:08.065274    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cwr5x" event={"ID":"2aa53dc2-24c4-4801-bbca-b84ac909f19e","Type":"ContainerStarted","Data":"c821876691ed5d14f23589fed6b069d3d15d07239e8a95eee0a2cd3a39757e34"}
Jul 29 19:26:12.098120 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:12.098098    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 29 19:26:13.091861 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:13.091818    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cwr5x" event={"ID":"2aa53dc2-24c4-4801-bbca-b84ac909f19e","Type":"ContainerStarted","Data":"220c109c935bce8b42a144d2a935745bca1ccb41bf9cc5d249db50fb2ec5e402"}
Jul 29 19:26:13.092044 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:13.091956    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cwr5x"
Jul 29 19:26:13.109453 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:13.109405    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cwr5x" podStartSLOduration=2.411205659 podStartE2EDuration="7.109392674s" totalImagesPullingTime="4.698187015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:26:13.108790502 +0000 UTC m=+543.936289560" watchObservedRunningTime="2026-07-29 19:26:13.109392674 +0000 UTC m=+543.936891731"
Jul 29 19:26:19.124528 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:19.124489    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cwr5x"
Jul 29 19:26:19.948720 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:19.948685    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6c7d7877c9-n85rd"]
Jul 29 19:26:19.956572 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:19.956547    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c7d7877c9-n85rd"
Jul 29 19:26:19.959322 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:19.959289    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 29 19:26:19.960832 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:19.960811    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c7d7877c9-n85rd"]
Jul 29 19:26:19.992630 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:19.992604    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/84787379-cec5-430f-8152-28147d010880-tls-cert\") pod \"authorino-6c7d7877c9-n85rd\" (UID: \"84787379-cec5-430f-8152-28147d010880\") " pod="kuadrant-system/authorino-6c7d7877c9-n85rd"
Jul 29 19:26:19.992756 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:19.992646    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjd6q\" (UniqueName: \"kubernetes.io/projected/84787379-cec5-430f-8152-28147d010880-kube-api-access-jjd6q\") pod \"authorino-6c7d7877c9-n85rd\" (UID: \"84787379-cec5-430f-8152-28147d010880\") " pod="kuadrant-system/authorino-6c7d7877c9-n85rd"
Jul 29 19:26:20.093286 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:20.093251    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/84787379-cec5-430f-8152-28147d010880-tls-cert\") pod \"authorino-6c7d7877c9-n85rd\" (UID: \"84787379-cec5-430f-8152-28147d010880\") " pod="kuadrant-system/authorino-6c7d7877c9-n85rd"
Jul 29 19:26:20.093444 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:20.093291    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjd6q\" (UniqueName: \"kubernetes.io/projected/84787379-cec5-430f-8152-28147d010880-kube-api-access-jjd6q\") pod \"authorino-6c7d7877c9-n85rd\" (UID: \"84787379-cec5-430f-8152-28147d010880\") " pod="kuadrant-system/authorino-6c7d7877c9-n85rd"
Jul 29 19:26:20.095677 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:20.095657    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/84787379-cec5-430f-8152-28147d010880-tls-cert\") pod \"authorino-6c7d7877c9-n85rd\" (UID: \"84787379-cec5-430f-8152-28147d010880\") " pod="kuadrant-system/authorino-6c7d7877c9-n85rd"
Jul 29 19:26:20.102342 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:20.102315    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjd6q\" (UniqueName: \"kubernetes.io/projected/84787379-cec5-430f-8152-28147d010880-kube-api-access-jjd6q\") pod \"authorino-6c7d7877c9-n85rd\" (UID: \"84787379-cec5-430f-8152-28147d010880\") " pod="kuadrant-system/authorino-6c7d7877c9-n85rd"
Jul 29 19:26:20.266790 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:20.266714    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c7d7877c9-n85rd"
Jul 29 19:26:20.392801 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:20.392776    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c7d7877c9-n85rd"]
Jul 29 19:26:20.395432 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:26:20.395403    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84787379_cec5_430f_8152_28147d010880.slice/crio-52635a1da1a5a7fb5f35ce0b396141e10e5f71bcc054962380e78c133f80a749 WatchSource:0}: Error finding container 52635a1da1a5a7fb5f35ce0b396141e10e5f71bcc054962380e78c133f80a749: Status 404 returned error can't find the container with id 52635a1da1a5a7fb5f35ce0b396141e10e5f71bcc054962380e78c133f80a749
Jul 29 19:26:21.125098 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:21.125069    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c7d7877c9-n85rd" event={"ID":"84787379-cec5-430f-8152-28147d010880","Type":"ContainerStarted","Data":"52635a1da1a5a7fb5f35ce0b396141e10e5f71bcc054962380e78c133f80a749"}
Jul 29 19:26:22.130419 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:22.130384    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c7d7877c9-n85rd" event={"ID":"84787379-cec5-430f-8152-28147d010880","Type":"ContainerStarted","Data":"40770f591347511ee9593b46c5dfaa88a853d6e13dbb2120e92a19f26d5e605e"}
Jul 29 19:26:22.146479 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:22.146428    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6c7d7877c9-n85rd" podStartSLOduration=2.448906014 podStartE2EDuration="3.146413002s" totalImagesPullingTime="697.506988ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:26:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:26:22.145491589 +0000 UTC m=+552.972990646" watchObservedRunningTime="2026-07-29 19:26:22.146413002 +0000 UTC m=+552.973912060"
Jul 29 19:26:22.172988 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:22.172927    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wpk2s"]
Jul 29 19:26:22.173150 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:22.173123    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-wpk2s" podUID="077d7ce0-5531-4444-8c32-c7aa24088759" containerName="authorino" containerID="cri-o://15155f484bc1fa94361d678be79fe6acc90734ece462b09aa6e372619b7db338" gracePeriod=30
Jul 29 19:26:22.426553 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:22.426526    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-wpk2s"
Jul 29 19:26:22.514844 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:22.514811    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btjsf\" (UniqueName: \"kubernetes.io/projected/077d7ce0-5531-4444-8c32-c7aa24088759-kube-api-access-btjsf\") pod \"077d7ce0-5531-4444-8c32-c7aa24088759\" (UID: \"077d7ce0-5531-4444-8c32-c7aa24088759\") "
Jul 29 19:26:22.516869 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:22.516840    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/077d7ce0-5531-4444-8c32-c7aa24088759-kube-api-access-btjsf" (OuterVolumeSpecName: "kube-api-access-btjsf") pod "077d7ce0-5531-4444-8c32-c7aa24088759" (UID: "077d7ce0-5531-4444-8c32-c7aa24088759"). InnerVolumeSpecName "kube-api-access-btjsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:26:22.616435 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:22.616401    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btjsf\" (UniqueName: \"kubernetes.io/projected/077d7ce0-5531-4444-8c32-c7aa24088759-kube-api-access-btjsf\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:26:23.136139 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:23.136106    2557 generic.go:356] "Generic (PLEG): container finished" podID="077d7ce0-5531-4444-8c32-c7aa24088759" containerID="15155f484bc1fa94361d678be79fe6acc90734ece462b09aa6e372619b7db338" exitCode=0
Jul 29 19:26:23.136634 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:23.136178    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-wpk2s"
Jul 29 19:26:23.136634 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:23.136192    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-wpk2s" event={"ID":"077d7ce0-5531-4444-8c32-c7aa24088759","Type":"ContainerDied","Data":"15155f484bc1fa94361d678be79fe6acc90734ece462b09aa6e372619b7db338"}
Jul 29 19:26:23.136634 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:23.136238    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-wpk2s" event={"ID":"077d7ce0-5531-4444-8c32-c7aa24088759","Type":"ContainerDied","Data":"b756098f75552174fbf3a870c8f395a10291dbca7fdd6902ac03a2dc3e913c9b"}
Jul 29 19:26:23.136634 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:23.136254    2557 scope.go:117] "RemoveContainer" containerID="15155f484bc1fa94361d678be79fe6acc90734ece462b09aa6e372619b7db338"
Jul 29 19:26:23.145032 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:23.145014    2557 scope.go:117] "RemoveContainer" containerID="15155f484bc1fa94361d678be79fe6acc90734ece462b09aa6e372619b7db338"
Jul 29 19:26:23.145278 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:26:23.145260    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15155f484bc1fa94361d678be79fe6acc90734ece462b09aa6e372619b7db338\": container with ID starting with 15155f484bc1fa94361d678be79fe6acc90734ece462b09aa6e372619b7db338 not found: ID does not exist" containerID="15155f484bc1fa94361d678be79fe6acc90734ece462b09aa6e372619b7db338"
Jul 29 19:26:23.145331 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:23.145287    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15155f484bc1fa94361d678be79fe6acc90734ece462b09aa6e372619b7db338"} err="failed to get container status \"15155f484bc1fa94361d678be79fe6acc90734ece462b09aa6e372619b7db338\": rpc error: code = NotFound desc = could not find container \"15155f484bc1fa94361d678be79fe6acc90734ece462b09aa6e372619b7db338\": container with ID starting with 15155f484bc1fa94361d678be79fe6acc90734ece462b09aa6e372619b7db338 not found: ID does not exist"
Jul 29 19:26:23.158853 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:23.158827    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wpk2s"]
Jul 29 19:26:23.162566 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:23.162546    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wpk2s"]
Jul 29 19:26:23.679198 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:26:23.679166    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="077d7ce0-5531-4444-8c32-c7aa24088759" path="/var/lib/kubelet/pods/077d7ce0-5531-4444-8c32-c7aa24088759/volumes"
Jul 29 19:27:02.561495 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:02.561459    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-tj96m"]
Jul 29 19:27:02.561979 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:02.561770    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="077d7ce0-5531-4444-8c32-c7aa24088759" containerName="authorino"
Jul 29 19:27:02.561979 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:02.561781    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="077d7ce0-5531-4444-8c32-c7aa24088759" containerName="authorino"
Jul 29 19:27:02.561979 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:02.561841    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="077d7ce0-5531-4444-8c32-c7aa24088759" containerName="authorino"
Jul 29 19:27:02.583139 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:02.583112    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-tj96m"]
Jul 29 19:27:02.583307 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:02.583221    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tj96m"
Jul 29 19:27:02.586005 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:02.585983    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-ddj85\""
Jul 29 19:27:02.586129 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:02.585989    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 29 19:27:02.670324 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:02.670295    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cnth\" (UniqueName: \"kubernetes.io/projected/31e35ba1-a9ef-4cb3-b1a2-aa983501d6bf-kube-api-access-4cnth\") pod \"payload-pre-processing-687dbb6cf6-tj96m\" (UID: \"31e35ba1-a9ef-4cb3-b1a2-aa983501d6bf\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tj96m"
Jul 29 19:27:02.670485 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:02.670342    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31e35ba1-a9ef-4cb3-b1a2-aa983501d6bf-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-tj96m\" (UID: \"31e35ba1-a9ef-4cb3-b1a2-aa983501d6bf\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tj96m"
Jul 29 19:27:02.771732 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:02.771694    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cnth\" (UniqueName: \"kubernetes.io/projected/31e35ba1-a9ef-4cb3-b1a2-aa983501d6bf-kube-api-access-4cnth\") pod \"payload-pre-processing-687dbb6cf6-tj96m\" (UID: \"31e35ba1-a9ef-4cb3-b1a2-aa983501d6bf\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tj96m"
Jul 29 19:27:02.771920 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:02.771758    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31e35ba1-a9ef-4cb3-b1a2-aa983501d6bf-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-tj96m\" (UID: \"31e35ba1-a9ef-4cb3-b1a2-aa983501d6bf\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tj96m"
Jul 29 19:27:02.772462 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:02.772440    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31e35ba1-a9ef-4cb3-b1a2-aa983501d6bf-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-tj96m\" (UID: \"31e35ba1-a9ef-4cb3-b1a2-aa983501d6bf\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tj96m"
Jul 29 19:27:02.780411 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:02.780388    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cnth\" (UniqueName: \"kubernetes.io/projected/31e35ba1-a9ef-4cb3-b1a2-aa983501d6bf-kube-api-access-4cnth\") pod \"payload-pre-processing-687dbb6cf6-tj96m\" (UID: \"31e35ba1-a9ef-4cb3-b1a2-aa983501d6bf\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tj96m"
Jul 29 19:27:02.893646 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:02.893610    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tj96m"
Jul 29 19:27:03.038664 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:03.038640    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-tj96m"]
Jul 29 19:27:03.040802 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:27:03.040770    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31e35ba1_a9ef_4cb3_b1a2_aa983501d6bf.slice/crio-8d1a50333c37a85e85a6bac5f6e6d0cfa4be57f610fbc661991213b0549f9b69 WatchSource:0}: Error finding container 8d1a50333c37a85e85a6bac5f6e6d0cfa4be57f610fbc661991213b0549f9b69: Status 404 returned error can't find the container with id 8d1a50333c37a85e85a6bac5f6e6d0cfa4be57f610fbc661991213b0549f9b69
Jul 29 19:27:03.308695 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:03.308614    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tj96m" event={"ID":"31e35ba1-a9ef-4cb3-b1a2-aa983501d6bf","Type":"ContainerStarted","Data":"8d1a50333c37a85e85a6bac5f6e6d0cfa4be57f610fbc661991213b0549f9b69"}
Jul 29 19:27:05.322424 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:05.322388    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tj96m" event={"ID":"31e35ba1-a9ef-4cb3-b1a2-aa983501d6bf","Type":"ContainerStarted","Data":"ee6c0f740d031d81b6d120e860588be516531de38775fd553b53672508516acd"}
Jul 29 19:27:05.322837 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:05.322516    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tj96m"
Jul 29 19:27:05.340786 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:05.340736    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tj96m" podStartSLOduration=1.254881983 podStartE2EDuration="3.34072002s" totalImagesPullingTime="2.085838037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:27:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:27:05.339471974 +0000 UTC m=+596.166971025" watchObservedRunningTime="2026-07-29 19:27:05.34072002 +0000 UTC m=+596.168219078"
Jul 29 19:27:16.328901 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:16.328871    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-tj96m"
Jul 29 19:27:19.050933 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:19.050899    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s"]
Jul 29 19:27:19.055518 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:19.055499    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s"
Jul 29 19:27:19.059593 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:19.059566    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 29 19:27:19.059715 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:19.059614    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 29 19:27:19.059715 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:19.059622    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 29 19:27:19.059715 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:19.059622    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-qm9st\""
Jul 29 19:27:19.064476 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:19.064429    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s"]
Jul 29 19:27:19.122392 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:19.122367    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6c448d8-f996-4179-b52a-e637a7622d14-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s\" (UID: \"f6c448d8-f996-4179-b52a-e637a7622d14\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s"
Jul 29 19:27:19.122542 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:19.122463    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6c448d8-f996-4179-b52a-e637a7622d14-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s\" (UID: \"f6c448d8-f996-4179-b52a-e637a7622d14\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s"
Jul 29 19:27:19.122542 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:19.122502    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgmgc\" (UniqueName: \"kubernetes.io/projected/f6c448d8-f996-4179-b52a-e637a7622d14-kube-api-access-tgmgc\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s\" (UID: \"f6c448d8-f996-4179-b52a-e637a7622d14\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s"
Jul 29 19:27:19.122542 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:19.122524    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6c448d8-f996-4179-b52a-e637a7622d14-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s\" (UID: \"f6c448d8-f996-4179-b52a-e637a7622d14\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s"
Jul 29 19:27:19.122542 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:19.122542    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6c448d8-f996-4179-b52a-e637a7622d14-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s\" (UID: \"f6c448d8-f996-4179-b52a-e637a7622d14\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s"
Jul 29 19:27:19.122717 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:19.122564    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6c448d8-f996-4179-b52a-e637a7622d14-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s\" (UID: \"f6c448d8-f996-4179-b52a-e637a7622d14\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s"
Jul 29 19:27:19.223985 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:19.223929    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6c448d8-f996-4179-b52a-e637a7622d14-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s\" (UID: \"f6c448d8-f996-4179-b52a-e637a7622d14\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s"
Jul 29 19:27:19.224166 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:19.224007    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6c448d8-f996-4179-b52a-e637a7622d14-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s\" (UID: \"f6c448d8-f996-4179-b52a-e637a7622d14\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s"
Jul 29 19:27:19.224166 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:19.224031    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgmgc\" (UniqueName: \"kubernetes.io/projected/f6c448d8-f996-4179-b52a-e637a7622d14-kube-api-access-tgmgc\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s\" (UID: \"f6c448d8-f996-4179-b52a-e637a7622d14\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s"
Jul 29 19:27:19.224166 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:19.224052    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6c448d8-f996-4179-b52a-e637a7622d14-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s\" (UID: \"f6c448d8-f996-4179-b52a-e637a7622d14\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s"
Jul 29 19:27:19.224166 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:19.224069    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6c448d8-f996-4179-b52a-e637a7622d14-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s\" (UID: \"f6c448d8-f996-4179-b52a-e637a7622d14\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s"
Jul 29 19:27:19.224166 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:19.224087    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6c448d8-f996-4179-b52a-e637a7622d14-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s\" (UID: \"f6c448d8-f996-4179-b52a-e637a7622d14\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s"
Jul 29 19:27:19.224422 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:19.224388    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6c448d8-f996-4179-b52a-e637a7622d14-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s\" (UID: \"f6c448d8-f996-4179-b52a-e637a7622d14\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s"
Jul 29 19:27:19.224464 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:19.224444    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6c448d8-f996-4179-b52a-e637a7622d14-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s\" (UID: \"f6c448d8-f996-4179-b52a-e637a7622d14\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s"
Jul 29 19:27:19.224499 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:19.224484    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6c448d8-f996-4179-b52a-e637a7622d14-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s\" (UID: \"f6c448d8-f996-4179-b52a-e637a7622d14\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s"
Jul 29 19:27:19.226278 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:19.226254    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6c448d8-f996-4179-b52a-e637a7622d14-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s\" (UID: \"f6c448d8-f996-4179-b52a-e637a7622d14\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s"
Jul 29 19:27:19.226538 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:19.226522    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6c448d8-f996-4179-b52a-e637a7622d14-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s\" (UID: \"f6c448d8-f996-4179-b52a-e637a7622d14\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s"
Jul 29 19:27:19.232500 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:19.232476    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgmgc\" (UniqueName: \"kubernetes.io/projected/f6c448d8-f996-4179-b52a-e637a7622d14-kube-api-access-tgmgc\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s\" (UID: \"f6c448d8-f996-4179-b52a-e637a7622d14\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s"
Jul 29 19:27:19.366625 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:19.366545    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s"
Jul 29 19:27:19.499772 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:19.499742    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s"]
Jul 29 19:27:19.501889 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:27:19.501860    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c448d8_f996_4179_b52a_e637a7622d14.slice/crio-78a1d29f88e5bb5acd8b92d12fcf7270969f8322c2172d545a18c4ed4cce32ad WatchSource:0}: Error finding container 78a1d29f88e5bb5acd8b92d12fcf7270969f8322c2172d545a18c4ed4cce32ad: Status 404 returned error can't find the container with id 78a1d29f88e5bb5acd8b92d12fcf7270969f8322c2172d545a18c4ed4cce32ad
Jul 29 19:27:20.391903 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:20.391846    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s" event={"ID":"f6c448d8-f996-4179-b52a-e637a7622d14","Type":"ContainerStarted","Data":"78a1d29f88e5bb5acd8b92d12fcf7270969f8322c2172d545a18c4ed4cce32ad"}
Jul 29 19:27:21.398226 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:21.398184    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s" event={"ID":"f6c448d8-f996-4179-b52a-e637a7622d14","Type":"ContainerStarted","Data":"3d3c64ee2ffc5f87434534c78db474733d78c4508e515b78409065ed40094a1c"}
Jul 29 19:27:21.416518 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:21.416473    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s" podStartSLOduration=1.3666204770000001 podStartE2EDuration="2.416460327s" totalImagesPullingTime="1.04983985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:27:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:27:21.416039611 +0000 UTC m=+612.243538690" watchObservedRunningTime="2026-07-29 19:27:21.416460327 +0000 UTC m=+612.243959384"
Jul 29 19:27:29.367231 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:29.367191    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s"
Jul 29 19:27:29.367603 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:29.367262    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s"
Jul 29 19:27:29.379542 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:29.379519    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s"
Jul 29 19:27:29.453832 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:27:29.453799    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s"
Jul 29 19:28:11.941820 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:28:11.941786    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq"]
Jul 29 19:28:11.945359 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:28:11.945336    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq"
Jul 29 19:28:11.948512 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:28:11.948491    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 29 19:28:11.958625 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:28:11.958602    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq"]
Jul 29 19:28:12.080057 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:28:12.080023    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df575b63-388f-45b7-aa75-a2833c64c645-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq\" (UID: \"df575b63-388f-45b7-aa75-a2833c64c645\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq"
Jul 29 19:28:12.080219 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:28:12.080080    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5mqq\" (UniqueName: \"kubernetes.io/projected/df575b63-388f-45b7-aa75-a2833c64c645-kube-api-access-n5mqq\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq\" (UID: \"df575b63-388f-45b7-aa75-a2833c64c645\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq"
Jul 29 19:28:12.080219 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:28:12.080145    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df575b63-388f-45b7-aa75-a2833c64c645-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq\" (UID: \"df575b63-388f-45b7-aa75-a2833c64c645\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq"
Jul 29 19:28:12.080219 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:28:12.080177    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df575b63-388f-45b7-aa75-a2833c64c645-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq\" (UID: \"df575b63-388f-45b7-aa75-a2833c64c645\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq"
Jul 29 19:28:12.080219 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:28:12.080199    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df575b63-388f-45b7-aa75-a2833c64c645-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq\" (UID: \"df575b63-388f-45b7-aa75-a2833c64c645\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq"
Jul 29 19:28:12.080219 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:28:12.080218    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df575b63-388f-45b7-aa75-a2833c64c645-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq\" (UID: \"df575b63-388f-45b7-aa75-a2833c64c645\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq"
Jul 29 19:28:12.181466 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:28:12.181435    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df575b63-388f-45b7-aa75-a2833c64c645-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq\" (UID: \"df575b63-388f-45b7-aa75-a2833c64c645\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq"
Jul 29 19:28:12.181645 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:28:12.181509    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5mqq\" (UniqueName: \"kubernetes.io/projected/df575b63-388f-45b7-aa75-a2833c64c645-kube-api-access-n5mqq\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq\" (UID: \"df575b63-388f-45b7-aa75-a2833c64c645\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq"
Jul 29 19:28:12.181645 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:28:12.181540    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df575b63-388f-45b7-aa75-a2833c64c645-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq\" (UID: \"df575b63-388f-45b7-aa75-a2833c64c645\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq"
Jul 29 19:28:12.181645 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:28:12.181567    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df575b63-388f-45b7-aa75-a2833c64c645-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq\" (UID: \"df575b63-388f-45b7-aa75-a2833c64c645\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq"
Jul 29 19:28:12.181645 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:28:12.181594    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df575b63-388f-45b7-aa75-a2833c64c645-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq\" (UID: \"df575b63-388f-45b7-aa75-a2833c64c645\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq"
Jul 29 19:28:12.181645 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:28:12.181621    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df575b63-388f-45b7-aa75-a2833c64c645-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq\" (UID: \"df575b63-388f-45b7-aa75-a2833c64c645\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq"
Jul 29 19:28:12.181914 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:28:12.181887    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df575b63-388f-45b7-aa75-a2833c64c645-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq\" (UID: \"df575b63-388f-45b7-aa75-a2833c64c645\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq"
Jul 29 19:28:12.181991 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:28:12.181914    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df575b63-388f-45b7-aa75-a2833c64c645-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq\" (UID: \"df575b63-388f-45b7-aa75-a2833c64c645\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq"
Jul 29 19:28:12.182068 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:28:12.182026    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df575b63-388f-45b7-aa75-a2833c64c645-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq\" (UID: \"df575b63-388f-45b7-aa75-a2833c64c645\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq"
Jul 29 19:28:12.183859 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:28:12.183838    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df575b63-388f-45b7-aa75-a2833c64c645-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq\" (UID: \"df575b63-388f-45b7-aa75-a2833c64c645\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq"
Jul 29 19:28:12.184159 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:28:12.184137    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df575b63-388f-45b7-aa75-a2833c64c645-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq\" (UID: \"df575b63-388f-45b7-aa75-a2833c64c645\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq"
Jul 29 19:28:12.190529 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:28:12.190509    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5mqq\" (UniqueName: \"kubernetes.io/projected/df575b63-388f-45b7-aa75-a2833c64c645-kube-api-access-n5mqq\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq\" (UID: \"df575b63-388f-45b7-aa75-a2833c64c645\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq"
Jul 29 19:28:12.256602 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:28:12.256533    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq"
Jul 29 19:28:12.388769 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:28:12.388743    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq"]
Jul 29 19:28:12.391109 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:28:12.391085    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf575b63_388f_45b7_aa75_a2833c64c645.slice/crio-b65b09c20e2d41d8e07c2ee00ac97c22dc0c452320bf72c9b7b941e2a9cc85bb WatchSource:0}: Error finding container b65b09c20e2d41d8e07c2ee00ac97c22dc0c452320bf72c9b7b941e2a9cc85bb: Status 404 returned error can't find the container with id b65b09c20e2d41d8e07c2ee00ac97c22dc0c452320bf72c9b7b941e2a9cc85bb
Jul 29 19:28:12.392429 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:28:12.392412    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 19:28:12.642296 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:28:12.642264    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq" event={"ID":"df575b63-388f-45b7-aa75-a2833c64c645","Type":"ContainerStarted","Data":"362261b59df9edb3f3ddf7bb256e65a6a949a804075916cb379f42016f8317f6"}
Jul 29 19:28:12.642417 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:28:12.642302    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq" event={"ID":"df575b63-388f-45b7-aa75-a2833c64c645","Type":"ContainerStarted","Data":"b65b09c20e2d41d8e07c2ee00ac97c22dc0c452320bf72c9b7b941e2a9cc85bb"}
Jul 29 19:28:12.663251 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:28:12.663198    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq" podStartSLOduration=1.491422929 podStartE2EDuration="1.66318438s" totalImagesPullingTime="171.761451ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:28:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:28:12.663116572 +0000 UTC m=+663.490615640" watchObservedRunningTime="2026-07-29 19:28:12.66318438 +0000 UTC m=+663.490683435"
Jul 29 19:28:22.257149 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:28:22.257101    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq"
Jul 29 19:28:22.257641 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:28:22.257185    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq"
Jul 29 19:28:22.269720 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:28:22.269693    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq"
Jul 29 19:28:22.695065 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:28:22.695031    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq"
Jul 29 19:29:16.405401 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:29:16.405362    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c7d7877c9-n85rd"]
Jul 29 19:29:16.405957 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:29:16.405584    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6c7d7877c9-n85rd" podUID="84787379-cec5-430f-8152-28147d010880" containerName="authorino" containerID="cri-o://40770f591347511ee9593b46c5dfaa88a853d6e13dbb2120e92a19f26d5e605e" gracePeriod=30
Jul 29 19:29:16.653197 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:29:16.653173    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c7d7877c9-n85rd"
Jul 29 19:29:16.760662 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:29:16.760582    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjd6q\" (UniqueName: \"kubernetes.io/projected/84787379-cec5-430f-8152-28147d010880-kube-api-access-jjd6q\") pod \"84787379-cec5-430f-8152-28147d010880\" (UID: \"84787379-cec5-430f-8152-28147d010880\") "
Jul 29 19:29:16.760662 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:29:16.760630    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/84787379-cec5-430f-8152-28147d010880-tls-cert\") pod \"84787379-cec5-430f-8152-28147d010880\" (UID: \"84787379-cec5-430f-8152-28147d010880\") "
Jul 29 19:29:16.762723 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:29:16.762687    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84787379-cec5-430f-8152-28147d010880-kube-api-access-jjd6q" (OuterVolumeSpecName: "kube-api-access-jjd6q") pod "84787379-cec5-430f-8152-28147d010880" (UID: "84787379-cec5-430f-8152-28147d010880"). InnerVolumeSpecName "kube-api-access-jjd6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:29:16.772037 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:29:16.772011    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84787379-cec5-430f-8152-28147d010880-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "84787379-cec5-430f-8152-28147d010880" (UID: "84787379-cec5-430f-8152-28147d010880"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:29:16.861606 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:29:16.861561    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjd6q\" (UniqueName: \"kubernetes.io/projected/84787379-cec5-430f-8152-28147d010880-kube-api-access-jjd6q\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:29:16.861606 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:29:16.861602    2557 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/84787379-cec5-430f-8152-28147d010880-tls-cert\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:29:16.913318 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:29:16.913285    2557 generic.go:356] "Generic (PLEG): container finished" podID="84787379-cec5-430f-8152-28147d010880" containerID="40770f591347511ee9593b46c5dfaa88a853d6e13dbb2120e92a19f26d5e605e" exitCode=0
Jul 29 19:29:16.913454 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:29:16.913325    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c7d7877c9-n85rd"
Jul 29 19:29:16.913454 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:29:16.913370    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c7d7877c9-n85rd" event={"ID":"84787379-cec5-430f-8152-28147d010880","Type":"ContainerDied","Data":"40770f591347511ee9593b46c5dfaa88a853d6e13dbb2120e92a19f26d5e605e"}
Jul 29 19:29:16.913454 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:29:16.913408    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c7d7877c9-n85rd" event={"ID":"84787379-cec5-430f-8152-28147d010880","Type":"ContainerDied","Data":"52635a1da1a5a7fb5f35ce0b396141e10e5f71bcc054962380e78c133f80a749"}
Jul 29 19:29:16.913454 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:29:16.913426    2557 scope.go:117] "RemoveContainer" containerID="40770f591347511ee9593b46c5dfaa88a853d6e13dbb2120e92a19f26d5e605e"
Jul 29 19:29:16.922046 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:29:16.922025    2557 scope.go:117] "RemoveContainer" containerID="40770f591347511ee9593b46c5dfaa88a853d6e13dbb2120e92a19f26d5e605e"
Jul 29 19:29:16.922302 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:29:16.922283    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40770f591347511ee9593b46c5dfaa88a853d6e13dbb2120e92a19f26d5e605e\": container with ID starting with 40770f591347511ee9593b46c5dfaa88a853d6e13dbb2120e92a19f26d5e605e not found: ID does not exist" containerID="40770f591347511ee9593b46c5dfaa88a853d6e13dbb2120e92a19f26d5e605e"
Jul 29 19:29:16.922352 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:29:16.922310    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40770f591347511ee9593b46c5dfaa88a853d6e13dbb2120e92a19f26d5e605e"} err="failed to get container status \"40770f591347511ee9593b46c5dfaa88a853d6e13dbb2120e92a19f26d5e605e\": rpc error: code = NotFound desc = could not find container \"40770f591347511ee9593b46c5dfaa88a853d6e13dbb2120e92a19f26d5e605e\": container with ID starting with 40770f591347511ee9593b46c5dfaa88a853d6e13dbb2120e92a19f26d5e605e not found: ID does not exist"
Jul 29 19:29:16.937542 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:29:16.937517    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c7d7877c9-n85rd"]
Jul 29 19:29:16.944021 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:29:16.944001    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6c7d7877c9-n85rd"]
Jul 29 19:29:17.678477 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:29:17.678443    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84787379-cec5-430f-8152-28147d010880" path="/var/lib/kubelet/pods/84787379-cec5-430f-8152-28147d010880/volumes"
Jul 29 19:30:00.156725 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:30:00.156680    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755890-6hnnk"]
Jul 29 19:30:00.157279 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:30:00.157170    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84787379-cec5-430f-8152-28147d010880" containerName="authorino"
Jul 29 19:30:00.157279 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:30:00.157191    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="84787379-cec5-430f-8152-28147d010880" containerName="authorino"
Jul 29 19:30:00.157391 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:30:00.157301    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="84787379-cec5-430f-8152-28147d010880" containerName="authorino"
Jul 29 19:30:00.160470 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:30:00.160449    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755890-6hnnk"
Jul 29 19:30:00.163995 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:30:00.163945    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jsq2r\""
Jul 29 19:30:00.170279 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:30:00.170249    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755890-6hnnk"]
Jul 29 19:30:00.171369 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:30:00.171351    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755890-6hnnk"
Jul 29 19:30:00.304703 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:30:00.304677    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755890-6hnnk"]
Jul 29 19:30:00.307021 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:30:00.306995    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab14f273_7fc8_4074_854e_de0fb06a8737.slice/crio-793a80f926b88e5ba9de544c48708709fb649e0dafb96f50ada55f981b493499 WatchSource:0}: Error finding container 793a80f926b88e5ba9de544c48708709fb649e0dafb96f50ada55f981b493499: Status 404 returned error can't find the container with id 793a80f926b88e5ba9de544c48708709fb649e0dafb96f50ada55f981b493499
Jul 29 19:30:01.107092 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:30:01.107054    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755890-6hnnk" event={"ID":"ab14f273-7fc8-4074-854e-de0fb06a8737","Type":"ContainerStarted","Data":"793a80f926b88e5ba9de544c48708709fb649e0dafb96f50ada55f981b493499"}
Jul 29 19:30:04.126320 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:30:04.126288    2557 generic.go:356] "Generic (PLEG): container finished" podID="ab14f273-7fc8-4074-854e-de0fb06a8737" containerID="9d86872bbb76061eadf3f020ec706e08ee1bdc094d374a1606f3686d3049dd1d" exitCode=0
Jul 29 19:30:04.126674 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:30:04.126342    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755890-6hnnk" event={"ID":"ab14f273-7fc8-4074-854e-de0fb06a8737","Type":"ContainerDied","Data":"9d86872bbb76061eadf3f020ec706e08ee1bdc094d374a1606f3686d3049dd1d"}
Jul 29 19:30:05.259736 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:30:05.259708    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755890-6hnnk"
Jul 29 19:30:06.135906 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:30:06.135882    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755890-6hnnk"
Jul 29 19:30:06.136081 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:30:06.135877    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755890-6hnnk" event={"ID":"ab14f273-7fc8-4074-854e-de0fb06a8737","Type":"ContainerDied","Data":"793a80f926b88e5ba9de544c48708709fb649e0dafb96f50ada55f981b493499"}
Jul 29 19:30:06.136081 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:30:06.136046    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="793a80f926b88e5ba9de544c48708709fb649e0dafb96f50ada55f981b493499"
Jul 29 19:40:46.951939 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:46.951902    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg"]
Jul 29 19:40:46.954356 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:46.952388    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab14f273-7fc8-4074-854e-de0fb06a8737" containerName="cleanup"
Jul 29 19:40:46.954356 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:46.952407    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab14f273-7fc8-4074-854e-de0fb06a8737" containerName="cleanup"
Jul 29 19:40:46.954356 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:46.952528    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab14f273-7fc8-4074-854e-de0fb06a8737" containerName="cleanup"
Jul 29 19:40:46.955595 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:46.955580    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg"
Jul 29 19:40:46.958411 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:46.958389    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-0a3fb2f7-kserve-self-signed-certs\""
Jul 29 19:40:46.974428 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:46.974406    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg"]
Jul 29 19:40:47.123236 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:47.123200    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b401364f-427d-4c26-a4ca-4421c3e4a8bd-dshm\") pod \"e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg\" (UID: \"b401364f-427d-4c26-a4ca-4421c3e4a8bd\") " pod="llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg"
Jul 29 19:40:47.123408 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:47.123263    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b401364f-427d-4c26-a4ca-4421c3e4a8bd-model-cache\") pod \"e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg\" (UID: \"b401364f-427d-4c26-a4ca-4421c3e4a8bd\") " pod="llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg"
Jul 29 19:40:47.123408 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:47.123297    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b401364f-427d-4c26-a4ca-4421c3e4a8bd-home\") pod \"e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg\" (UID: \"b401364f-427d-4c26-a4ca-4421c3e4a8bd\") " pod="llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg"
Jul 29 19:40:47.123408 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:47.123339    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b401364f-427d-4c26-a4ca-4421c3e4a8bd-tls-certs\") pod \"e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg\" (UID: \"b401364f-427d-4c26-a4ca-4421c3e4a8bd\") " pod="llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg"
Jul 29 19:40:47.123542 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:47.123409    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5d9r\" (UniqueName: \"kubernetes.io/projected/b401364f-427d-4c26-a4ca-4421c3e4a8bd-kube-api-access-v5d9r\") pod \"e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg\" (UID: \"b401364f-427d-4c26-a4ca-4421c3e4a8bd\") " pod="llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg"
Jul 29 19:40:47.123542 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:47.123448    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b401364f-427d-4c26-a4ca-4421c3e4a8bd-tmp-dir\") pod \"e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg\" (UID: \"b401364f-427d-4c26-a4ca-4421c3e4a8bd\") " pod="llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg"
Jul 29 19:40:47.224531 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:47.224441    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b401364f-427d-4c26-a4ca-4421c3e4a8bd-dshm\") pod \"e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg\" (UID: \"b401364f-427d-4c26-a4ca-4421c3e4a8bd\") " pod="llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg"
Jul 29 19:40:47.224531 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:47.224492    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b401364f-427d-4c26-a4ca-4421c3e4a8bd-model-cache\") pod \"e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg\" (UID: \"b401364f-427d-4c26-a4ca-4421c3e4a8bd\") " pod="llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg"
Jul 29 19:40:47.224531 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:47.224518    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b401364f-427d-4c26-a4ca-4421c3e4a8bd-home\") pod \"e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg\" (UID: \"b401364f-427d-4c26-a4ca-4421c3e4a8bd\") " pod="llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg"
Jul 29 19:40:47.224786 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:47.224554    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b401364f-427d-4c26-a4ca-4421c3e4a8bd-tls-certs\") pod \"e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg\" (UID: \"b401364f-427d-4c26-a4ca-4421c3e4a8bd\") " pod="llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg"
Jul 29 19:40:47.224786 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:47.224612    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5d9r\" (UniqueName: \"kubernetes.io/projected/b401364f-427d-4c26-a4ca-4421c3e4a8bd-kube-api-access-v5d9r\") pod \"e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg\" (UID: \"b401364f-427d-4c26-a4ca-4421c3e4a8bd\") " pod="llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg"
Jul 29 19:40:47.224786 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:47.224642    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b401364f-427d-4c26-a4ca-4421c3e4a8bd-tmp-dir\") pod \"e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg\" (UID: \"b401364f-427d-4c26-a4ca-4421c3e4a8bd\") " pod="llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg"
Jul 29 19:40:47.224941 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:47.224877    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b401364f-427d-4c26-a4ca-4421c3e4a8bd-model-cache\") pod \"e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg\" (UID: \"b401364f-427d-4c26-a4ca-4421c3e4a8bd\") " pod="llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg"
Jul 29 19:40:47.224941 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:47.224924    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b401364f-427d-4c26-a4ca-4421c3e4a8bd-home\") pod \"e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg\" (UID: \"b401364f-427d-4c26-a4ca-4421c3e4a8bd\") " pod="llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg"
Jul 29 19:40:47.225108 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:47.225089    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b401364f-427d-4c26-a4ca-4421c3e4a8bd-tmp-dir\") pod \"e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg\" (UID: \"b401364f-427d-4c26-a4ca-4421c3e4a8bd\") " pod="llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg"
Jul 29 19:40:47.226690 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:47.226668    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b401364f-427d-4c26-a4ca-4421c3e4a8bd-dshm\") pod \"e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg\" (UID: \"b401364f-427d-4c26-a4ca-4421c3e4a8bd\") " pod="llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg"
Jul 29 19:40:47.226944 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:47.226925    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b401364f-427d-4c26-a4ca-4421c3e4a8bd-tls-certs\") pod \"e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg\" (UID: \"b401364f-427d-4c26-a4ca-4421c3e4a8bd\") " pod="llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg"
Jul 29 19:40:47.235193 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:47.235163    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5d9r\" (UniqueName: \"kubernetes.io/projected/b401364f-427d-4c26-a4ca-4421c3e4a8bd-kube-api-access-v5d9r\") pod \"e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg\" (UID: \"b401364f-427d-4c26-a4ca-4421c3e4a8bd\") " pod="llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg"
Jul 29 19:40:47.264139 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:47.264113    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg"
Jul 29 19:40:47.413919 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:47.413890    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg"]
Jul 29 19:40:47.415480 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:40:47.415439    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb401364f_427d_4c26_a4ca_4421c3e4a8bd.slice/crio-22926f171bd8bbc1dded9dd13b2b171abbf033bffdbfaa4a62af5cc0a1c67d9f WatchSource:0}: Error finding container 22926f171bd8bbc1dded9dd13b2b171abbf033bffdbfaa4a62af5cc0a1c67d9f: Status 404 returned error can't find the container with id 22926f171bd8bbc1dded9dd13b2b171abbf033bffdbfaa4a62af5cc0a1c67d9f
Jul 29 19:40:47.960203 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:47.960166    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg" event={"ID":"b401364f-427d-4c26-a4ca-4421c3e4a8bd","Type":"ContainerStarted","Data":"e5f787ef76e351243d1c046d28f009bdbe9b1aa84c4e824b1e20e37f79425057"}
Jul 29 19:40:47.960203 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:47.960210    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg" event={"ID":"b401364f-427d-4c26-a4ca-4421c3e4a8bd","Type":"ContainerStarted","Data":"22926f171bd8bbc1dded9dd13b2b171abbf033bffdbfaa4a62af5cc0a1c67d9f"}
Jul 29 19:40:47.984152 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:47.984107    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg" podStartSLOduration=1.984092815 podStartE2EDuration="1.984092815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:40:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:40:47.98265138 +0000 UTC m=+1418.810150449" watchObservedRunningTime="2026-07-29 19:40:47.984092815 +0000 UTC m=+1418.811591872"
Jul 29 19:40:56.640063 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:56.640031    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r"]
Jul 29 19:40:56.643623 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:56.643605    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r"
Jul 29 19:40:56.646674 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:56.646654    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-0a3fb2f7-kserve-self-signed-certs\""
Jul 29 19:40:56.654890 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:56.654870    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r"]
Jul 29 19:40:56.702543 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:56.702514    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d065a9f-a461-4881-9177-7e36300f7e52-model-cache\") pod \"e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r\" (UID: \"8d065a9f-a461-4881-9177-7e36300f7e52\") " pod="llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r"
Jul 29 19:40:56.702661 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:56.702556    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d065a9f-a461-4881-9177-7e36300f7e52-tmp-dir\") pod \"e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r\" (UID: \"8d065a9f-a461-4881-9177-7e36300f7e52\") " pod="llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r"
Jul 29 19:40:56.702661 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:56.702574    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg2kw\" (UniqueName: \"kubernetes.io/projected/8d065a9f-a461-4881-9177-7e36300f7e52-kube-api-access-zg2kw\") pod \"e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r\" (UID: \"8d065a9f-a461-4881-9177-7e36300f7e52\") " pod="llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r"
Jul 29 19:40:56.702734 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:56.702654    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d065a9f-a461-4881-9177-7e36300f7e52-home\") pod \"e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r\" (UID: \"8d065a9f-a461-4881-9177-7e36300f7e52\") " pod="llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r"
Jul 29 19:40:56.702734 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:56.702685    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d065a9f-a461-4881-9177-7e36300f7e52-tls-certs\") pod \"e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r\" (UID: \"8d065a9f-a461-4881-9177-7e36300f7e52\") " pod="llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r"
Jul 29 19:40:56.702796 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:56.702753    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d065a9f-a461-4881-9177-7e36300f7e52-dshm\") pod \"e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r\" (UID: \"8d065a9f-a461-4881-9177-7e36300f7e52\") " pod="llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r"
Jul 29 19:40:56.803274 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:56.803239    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d065a9f-a461-4881-9177-7e36300f7e52-dshm\") pod \"e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r\" (UID: \"8d065a9f-a461-4881-9177-7e36300f7e52\") " pod="llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r"
Jul 29 19:40:56.803403 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:56.803284    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d065a9f-a461-4881-9177-7e36300f7e52-model-cache\") pod \"e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r\" (UID: \"8d065a9f-a461-4881-9177-7e36300f7e52\") " pod="llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r"
Jul 29 19:40:56.803403 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:56.803331    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d065a9f-a461-4881-9177-7e36300f7e52-tmp-dir\") pod \"e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r\" (UID: \"8d065a9f-a461-4881-9177-7e36300f7e52\") " pod="llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r"
Jul 29 19:40:56.803403 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:56.803381    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg2kw\" (UniqueName: \"kubernetes.io/projected/8d065a9f-a461-4881-9177-7e36300f7e52-kube-api-access-zg2kw\") pod \"e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r\" (UID: \"8d065a9f-a461-4881-9177-7e36300f7e52\") " pod="llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r"
Jul 29 19:40:56.803511 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:56.803483    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d065a9f-a461-4881-9177-7e36300f7e52-home\") pod \"e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r\" (UID: \"8d065a9f-a461-4881-9177-7e36300f7e52\") " pod="llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r"
Jul 29 19:40:56.803551 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:56.803521    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d065a9f-a461-4881-9177-7e36300f7e52-tls-certs\") pod \"e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r\" (UID: \"8d065a9f-a461-4881-9177-7e36300f7e52\") " pod="llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r"
Jul 29 19:40:56.803792 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:56.803762    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d065a9f-a461-4881-9177-7e36300f7e52-model-cache\") pod \"e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r\" (UID: \"8d065a9f-a461-4881-9177-7e36300f7e52\") " pod="llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r"
Jul 29 19:40:56.803898 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:56.803795    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d065a9f-a461-4881-9177-7e36300f7e52-tmp-dir\") pod \"e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r\" (UID: \"8d065a9f-a461-4881-9177-7e36300f7e52\") " pod="llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r"
Jul 29 19:40:56.803898 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:56.803883    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d065a9f-a461-4881-9177-7e36300f7e52-home\") pod \"e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r\" (UID: \"8d065a9f-a461-4881-9177-7e36300f7e52\") " pod="llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r"
Jul 29 19:40:56.805545 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:56.805521    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d065a9f-a461-4881-9177-7e36300f7e52-dshm\") pod \"e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r\" (UID: \"8d065a9f-a461-4881-9177-7e36300f7e52\") " pod="llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r"
Jul 29 19:40:56.805873 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:56.805855    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d065a9f-a461-4881-9177-7e36300f7e52-tls-certs\") pod \"e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r\" (UID: \"8d065a9f-a461-4881-9177-7e36300f7e52\") " pod="llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r"
Jul 29 19:40:56.811840 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:56.811822    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg2kw\" (UniqueName: \"kubernetes.io/projected/8d065a9f-a461-4881-9177-7e36300f7e52-kube-api-access-zg2kw\") pod \"e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r\" (UID: \"8d065a9f-a461-4881-9177-7e36300f7e52\") " pod="llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r"
Jul 29 19:40:56.955588 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:56.955554    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r"
Jul 29 19:40:57.089132 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:57.089098    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r"]
Jul 29 19:40:57.091011 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:40:57.090986    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d065a9f_a461_4881_9177_7e36300f7e52.slice/crio-d54c6a303ab10909be735461ba283c559b3624f8c852fbb98524da4e670893e0 WatchSource:0}: Error finding container d54c6a303ab10909be735461ba283c559b3624f8c852fbb98524da4e670893e0: Status 404 returned error can't find the container with id d54c6a303ab10909be735461ba283c559b3624f8c852fbb98524da4e670893e0
Jul 29 19:40:57.265086 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:57.265002    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg"
Jul 29 19:40:57.265234 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:57.265157    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg"
Jul 29 19:40:57.277577 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:57.277545    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg"
Jul 29 19:40:58.004793 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:58.004760    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r" event={"ID":"8d065a9f-a461-4881-9177-7e36300f7e52","Type":"ContainerStarted","Data":"d769fbb5e3fb5554a462b4939d0a7b5be698f24bf767393a7d14c2ba3a09a34a"}
Jul 29 19:40:58.004793 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:58.004798    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r" event={"ID":"8d065a9f-a461-4881-9177-7e36300f7e52","Type":"ContainerStarted","Data":"d54c6a303ab10909be735461ba283c559b3624f8c852fbb98524da4e670893e0"}
Jul 29 19:40:58.016094 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:58.016070    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg"
Jul 29 19:40:58.023703 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:40:58.023661    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r" podStartSLOduration=2.02364733 podStartE2EDuration="2.02364733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:40:58.023044317 +0000 UTC m=+1428.850543399" watchObservedRunningTime="2026-07-29 19:40:58.02364733 +0000 UTC m=+1428.851146387"
Jul 29 19:41:06.956075 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:06.955992    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r"
Jul 29 19:41:06.956075 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:06.956034    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r"
Jul 29 19:41:06.968615 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:06.968587    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r"
Jul 29 19:41:07.053827 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:07.053798    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r"
Jul 29 19:41:27.303118 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:27.303083    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg"]
Jul 29 19:41:28.405218 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:28.405180    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r"]
Jul 29 19:41:42.321717 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:42.321660    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg" podUID="b401364f-427d-4c26-a4ca-4421c3e4a8bd" containerName="main" containerID="cri-o://e5f787ef76e351243d1c046d28f009bdbe9b1aa84c4e824b1e20e37f79425057" gracePeriod=45
Jul 29 19:41:42.577096 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:42.577037    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg"
Jul 29 19:41:42.700999 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:42.700948    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b401364f-427d-4c26-a4ca-4421c3e4a8bd-dshm\") pod \"b401364f-427d-4c26-a4ca-4421c3e4a8bd\" (UID: \"b401364f-427d-4c26-a4ca-4421c3e4a8bd\") "
Jul 29 19:41:42.701177 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:42.701023    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5d9r\" (UniqueName: \"kubernetes.io/projected/b401364f-427d-4c26-a4ca-4421c3e4a8bd-kube-api-access-v5d9r\") pod \"b401364f-427d-4c26-a4ca-4421c3e4a8bd\" (UID: \"b401364f-427d-4c26-a4ca-4421c3e4a8bd\") "
Jul 29 19:41:42.701177 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:42.701050    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b401364f-427d-4c26-a4ca-4421c3e4a8bd-tmp-dir\") pod \"b401364f-427d-4c26-a4ca-4421c3e4a8bd\" (UID: \"b401364f-427d-4c26-a4ca-4421c3e4a8bd\") "
Jul 29 19:41:42.701177 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:42.701084    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b401364f-427d-4c26-a4ca-4421c3e4a8bd-tls-certs\") pod \"b401364f-427d-4c26-a4ca-4421c3e4a8bd\" (UID: \"b401364f-427d-4c26-a4ca-4421c3e4a8bd\") "
Jul 29 19:41:42.701335 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:42.701209    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b401364f-427d-4c26-a4ca-4421c3e4a8bd-model-cache\") pod \"b401364f-427d-4c26-a4ca-4421c3e4a8bd\" (UID: \"b401364f-427d-4c26-a4ca-4421c3e4a8bd\") "
Jul 29 19:41:42.701335 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:42.701235    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b401364f-427d-4c26-a4ca-4421c3e4a8bd-home\") pod \"b401364f-427d-4c26-a4ca-4421c3e4a8bd\" (UID: \"b401364f-427d-4c26-a4ca-4421c3e4a8bd\") "
Jul 29 19:41:42.701435 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:42.701334    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b401364f-427d-4c26-a4ca-4421c3e4a8bd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b401364f-427d-4c26-a4ca-4421c3e4a8bd" (UID: "b401364f-427d-4c26-a4ca-4421c3e4a8bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:41:42.701486 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:42.701440    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b401364f-427d-4c26-a4ca-4421c3e4a8bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "b401364f-427d-4c26-a4ca-4421c3e4a8bd" (UID: "b401364f-427d-4c26-a4ca-4421c3e4a8bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:41:42.701557 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:42.701536    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b401364f-427d-4c26-a4ca-4421c3e4a8bd-home" (OuterVolumeSpecName: "home") pod "b401364f-427d-4c26-a4ca-4421c3e4a8bd" (UID: "b401364f-427d-4c26-a4ca-4421c3e4a8bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:41:42.701613 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:42.701606    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b401364f-427d-4c26-a4ca-4421c3e4a8bd-model-cache\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:41:42.701668 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:42.701621    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b401364f-427d-4c26-a4ca-4421c3e4a8bd-home\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:41:42.701668 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:42.701656    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b401364f-427d-4c26-a4ca-4421c3e4a8bd-tmp-dir\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:41:42.703231 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:42.703196    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b401364f-427d-4c26-a4ca-4421c3e4a8bd-dshm" (OuterVolumeSpecName: "dshm") pod "b401364f-427d-4c26-a4ca-4421c3e4a8bd" (UID: "b401364f-427d-4c26-a4ca-4421c3e4a8bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:41:42.703530 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:42.703503    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b401364f-427d-4c26-a4ca-4421c3e4a8bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b401364f-427d-4c26-a4ca-4421c3e4a8bd" (UID: "b401364f-427d-4c26-a4ca-4421c3e4a8bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:41:42.703530 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:42.703515    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b401364f-427d-4c26-a4ca-4421c3e4a8bd-kube-api-access-v5d9r" (OuterVolumeSpecName: "kube-api-access-v5d9r") pod "b401364f-427d-4c26-a4ca-4421c3e4a8bd" (UID: "b401364f-427d-4c26-a4ca-4421c3e4a8bd"). InnerVolumeSpecName "kube-api-access-v5d9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:41:42.802230 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:42.802196    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b401364f-427d-4c26-a4ca-4421c3e4a8bd-dshm\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:41:42.802230 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:42.802222    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5d9r\" (UniqueName: \"kubernetes.io/projected/b401364f-427d-4c26-a4ca-4421c3e4a8bd-kube-api-access-v5d9r\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:41:42.802230 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:42.802232    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b401364f-427d-4c26-a4ca-4421c3e4a8bd-tls-certs\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:41:43.206887 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:43.206855    2557 generic.go:356] "Generic (PLEG): container finished" podID="b401364f-427d-4c26-a4ca-4421c3e4a8bd" containerID="e5f787ef76e351243d1c046d28f009bdbe9b1aa84c4e824b1e20e37f79425057" exitCode=0
Jul 29 19:41:43.207061 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:43.206893    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg"
Jul 29 19:41:43.207061 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:43.206925    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg" event={"ID":"b401364f-427d-4c26-a4ca-4421c3e4a8bd","Type":"ContainerDied","Data":"e5f787ef76e351243d1c046d28f009bdbe9b1aa84c4e824b1e20e37f79425057"}
Jul 29 19:41:43.207061 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:43.206986    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg" event={"ID":"b401364f-427d-4c26-a4ca-4421c3e4a8bd","Type":"ContainerDied","Data":"22926f171bd8bbc1dded9dd13b2b171abbf033bffdbfaa4a62af5cc0a1c67d9f"}
Jul 29 19:41:43.207061 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:43.207008    2557 scope.go:117] "RemoveContainer" containerID="e5f787ef76e351243d1c046d28f009bdbe9b1aa84c4e824b1e20e37f79425057"
Jul 29 19:41:43.216642 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:43.216625    2557 scope.go:117] "RemoveContainer" containerID="e5f787ef76e351243d1c046d28f009bdbe9b1aa84c4e824b1e20e37f79425057"
Jul 29 19:41:43.216893 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:41:43.216873    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5f787ef76e351243d1c046d28f009bdbe9b1aa84c4e824b1e20e37f79425057\": container with ID starting with e5f787ef76e351243d1c046d28f009bdbe9b1aa84c4e824b1e20e37f79425057 not found: ID does not exist" containerID="e5f787ef76e351243d1c046d28f009bdbe9b1aa84c4e824b1e20e37f79425057"
Jul 29 19:41:43.216943 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:43.216903    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5f787ef76e351243d1c046d28f009bdbe9b1aa84c4e824b1e20e37f79425057"} err="failed to get container status \"e5f787ef76e351243d1c046d28f009bdbe9b1aa84c4e824b1e20e37f79425057\": rpc error: code = NotFound desc = could not find container \"e5f787ef76e351243d1c046d28f009bdbe9b1aa84c4e824b1e20e37f79425057\": container with ID starting with e5f787ef76e351243d1c046d28f009bdbe9b1aa84c4e824b1e20e37f79425057 not found: ID does not exist"
Jul 29 19:41:43.232670 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:43.232643    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg"]
Jul 29 19:41:43.239863 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:43.239844    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-0a3fb2f7-kserve-56885b776c-d5kgg"]
Jul 29 19:41:43.423845 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:43.423802    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r" podUID="8d065a9f-a461-4881-9177-7e36300f7e52" containerName="main" containerID="cri-o://d769fbb5e3fb5554a462b4939d0a7b5be698f24bf767393a7d14c2ba3a09a34a" gracePeriod=45
Jul 29 19:41:43.668182 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:43.668153    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r"
Jul 29 19:41:43.680283 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:43.680255    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b401364f-427d-4c26-a4ca-4421c3e4a8bd" path="/var/lib/kubelet/pods/b401364f-427d-4c26-a4ca-4421c3e4a8bd/volumes"
Jul 29 19:41:43.811354 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:43.811263    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d065a9f-a461-4881-9177-7e36300f7e52-tmp-dir\") pod \"8d065a9f-a461-4881-9177-7e36300f7e52\" (UID: \"8d065a9f-a461-4881-9177-7e36300f7e52\") "
Jul 29 19:41:43.811354 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:43.811307    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d065a9f-a461-4881-9177-7e36300f7e52-home\") pod \"8d065a9f-a461-4881-9177-7e36300f7e52\" (UID: \"8d065a9f-a461-4881-9177-7e36300f7e52\") "
Jul 29 19:41:43.811354 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:43.811345    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d065a9f-a461-4881-9177-7e36300f7e52-tls-certs\") pod \"8d065a9f-a461-4881-9177-7e36300f7e52\" (UID: \"8d065a9f-a461-4881-9177-7e36300f7e52\") "
Jul 29 19:41:43.811621 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:43.811390    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d065a9f-a461-4881-9177-7e36300f7e52-dshm\") pod \"8d065a9f-a461-4881-9177-7e36300f7e52\" (UID: \"8d065a9f-a461-4881-9177-7e36300f7e52\") "
Jul 29 19:41:43.811621 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:43.811416    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d065a9f-a461-4881-9177-7e36300f7e52-model-cache\") pod \"8d065a9f-a461-4881-9177-7e36300f7e52\" (UID: \"8d065a9f-a461-4881-9177-7e36300f7e52\") "
Jul 29 19:41:43.811621 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:43.811468    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zg2kw\" (UniqueName: \"kubernetes.io/projected/8d065a9f-a461-4881-9177-7e36300f7e52-kube-api-access-zg2kw\") pod \"8d065a9f-a461-4881-9177-7e36300f7e52\" (UID: \"8d065a9f-a461-4881-9177-7e36300f7e52\") "
Jul 29 19:41:43.811621 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:43.811539    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d065a9f-a461-4881-9177-7e36300f7e52-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8d065a9f-a461-4881-9177-7e36300f7e52" (UID: "8d065a9f-a461-4881-9177-7e36300f7e52"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:41:43.811807 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:43.811642    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d065a9f-a461-4881-9177-7e36300f7e52-home" (OuterVolumeSpecName: "home") pod "8d065a9f-a461-4881-9177-7e36300f7e52" (UID: "8d065a9f-a461-4881-9177-7e36300f7e52"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:41:43.811807 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:43.811666    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d065a9f-a461-4881-9177-7e36300f7e52-model-cache" (OuterVolumeSpecName: "model-cache") pod "8d065a9f-a461-4881-9177-7e36300f7e52" (UID: "8d065a9f-a461-4881-9177-7e36300f7e52"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:41:43.811807 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:43.811782    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d065a9f-a461-4881-9177-7e36300f7e52-tmp-dir\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:41:43.811807 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:43.811801    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d065a9f-a461-4881-9177-7e36300f7e52-home\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:41:43.811944 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:43.811814    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d065a9f-a461-4881-9177-7e36300f7e52-model-cache\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:41:43.813479 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:43.813456    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d065a9f-a461-4881-9177-7e36300f7e52-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8d065a9f-a461-4881-9177-7e36300f7e52" (UID: "8d065a9f-a461-4881-9177-7e36300f7e52"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:41:43.813562 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:43.813477    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d065a9f-a461-4881-9177-7e36300f7e52-dshm" (OuterVolumeSpecName: "dshm") pod "8d065a9f-a461-4881-9177-7e36300f7e52" (UID: "8d065a9f-a461-4881-9177-7e36300f7e52"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:41:43.813610 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:43.813592    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d065a9f-a461-4881-9177-7e36300f7e52-kube-api-access-zg2kw" (OuterVolumeSpecName: "kube-api-access-zg2kw") pod "8d065a9f-a461-4881-9177-7e36300f7e52" (UID: "8d065a9f-a461-4881-9177-7e36300f7e52"). InnerVolumeSpecName "kube-api-access-zg2kw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:41:43.912570 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:43.912528    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d065a9f-a461-4881-9177-7e36300f7e52-tls-certs\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:41:43.912570 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:43.912561    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d065a9f-a461-4881-9177-7e36300f7e52-dshm\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:41:43.912570 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:43.912573    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg2kw\" (UniqueName: \"kubernetes.io/projected/8d065a9f-a461-4881-9177-7e36300f7e52-kube-api-access-zg2kw\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:41:44.213257 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:44.213220    2557 generic.go:356] "Generic (PLEG): container finished" podID="8d065a9f-a461-4881-9177-7e36300f7e52" containerID="d769fbb5e3fb5554a462b4939d0a7b5be698f24bf767393a7d14c2ba3a09a34a" exitCode=0
Jul 29 19:41:44.213443 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:44.213260    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r"
Jul 29 19:41:44.213443 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:44.213315    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r" event={"ID":"8d065a9f-a461-4881-9177-7e36300f7e52","Type":"ContainerDied","Data":"d769fbb5e3fb5554a462b4939d0a7b5be698f24bf767393a7d14c2ba3a09a34a"}
Jul 29 19:41:44.213443 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:44.213355    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r" event={"ID":"8d065a9f-a461-4881-9177-7e36300f7e52","Type":"ContainerDied","Data":"d54c6a303ab10909be735461ba283c559b3624f8c852fbb98524da4e670893e0"}
Jul 29 19:41:44.213443 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:44.213372    2557 scope.go:117] "RemoveContainer" containerID="d769fbb5e3fb5554a462b4939d0a7b5be698f24bf767393a7d14c2ba3a09a34a"
Jul 29 19:41:44.221693 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:44.221677    2557 scope.go:117] "RemoveContainer" containerID="d769fbb5e3fb5554a462b4939d0a7b5be698f24bf767393a7d14c2ba3a09a34a"
Jul 29 19:41:44.221930 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:41:44.221912    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d769fbb5e3fb5554a462b4939d0a7b5be698f24bf767393a7d14c2ba3a09a34a\": container with ID starting with d769fbb5e3fb5554a462b4939d0a7b5be698f24bf767393a7d14c2ba3a09a34a not found: ID does not exist" containerID="d769fbb5e3fb5554a462b4939d0a7b5be698f24bf767393a7d14c2ba3a09a34a"
Jul 29 19:41:44.221989 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:44.221939    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d769fbb5e3fb5554a462b4939d0a7b5be698f24bf767393a7d14c2ba3a09a34a"} err="failed to get container status \"d769fbb5e3fb5554a462b4939d0a7b5be698f24bf767393a7d14c2ba3a09a34a\": rpc error: code = NotFound desc = could not find container \"d769fbb5e3fb5554a462b4939d0a7b5be698f24bf767393a7d14c2ba3a09a34a\": container with ID starting with d769fbb5e3fb5554a462b4939d0a7b5be698f24bf767393a7d14c2ba3a09a34a not found: ID does not exist"
Jul 29 19:41:44.237218 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:44.237192    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r"]
Jul 29 19:41:44.243596 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:44.243576    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-0a3fb2f7-kserve-57767698f5-rsh5r"]
Jul 29 19:41:45.679041 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:41:45.679012    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d065a9f-a461-4881-9177-7e36300f7e52" path="/var/lib/kubelet/pods/8d065a9f-a461-4881-9177-7e36300f7e52/volumes"
Jul 29 19:45:00.148977 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:00.148929    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755905-48dzc"]
Jul 29 19:45:00.149453 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:00.149318    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b401364f-427d-4c26-a4ca-4421c3e4a8bd" containerName="main"
Jul 29 19:45:00.149453 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:00.149330    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b401364f-427d-4c26-a4ca-4421c3e4a8bd" containerName="main"
Jul 29 19:45:00.149453 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:00.149338    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d065a9f-a461-4881-9177-7e36300f7e52" containerName="main"
Jul 29 19:45:00.149453 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:00.149344    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d065a9f-a461-4881-9177-7e36300f7e52" containerName="main"
Jul 29 19:45:00.149453 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:00.149409    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d065a9f-a461-4881-9177-7e36300f7e52" containerName="main"
Jul 29 19:45:00.149453 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:00.149420    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b401364f-427d-4c26-a4ca-4421c3e4a8bd" containerName="main"
Jul 29 19:45:00.152230 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:00.152213    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755905-48dzc"
Jul 29 19:45:00.156849 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:00.156827    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jsq2r\""
Jul 29 19:45:00.161856 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:00.161839    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755905-48dzc"
Jul 29 19:45:00.207658 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:00.207624    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755905-48dzc"]
Jul 29 19:45:00.299455 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:00.299430    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755905-48dzc"]
Jul 29 19:45:00.301937 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:45:00.301906    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6421eb66_5559_4396_a8e3_3be2e7a9ece1.slice/crio-961da36751970be6f63c268b9cf5d16fd82ccb1c0228d2ec2dc30effc1ea18e7 WatchSource:0}: Error finding container 961da36751970be6f63c268b9cf5d16fd82ccb1c0228d2ec2dc30effc1ea18e7: Status 404 returned error can't find the container with id 961da36751970be6f63c268b9cf5d16fd82ccb1c0228d2ec2dc30effc1ea18e7
Jul 29 19:45:01.088780 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:01.088744    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755905-48dzc" event={"ID":"6421eb66-5559-4396-a8e3-3be2e7a9ece1","Type":"ContainerStarted","Data":"617ff423a8b9e0babe954ffe776cd9e47ac1bfb28f479de0bf0a59c351fdc9d4"}
Jul 29 19:45:01.088972 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:01.088799    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755905-48dzc" event={"ID":"6421eb66-5559-4396-a8e3-3be2e7a9ece1","Type":"ContainerStarted","Data":"961da36751970be6f63c268b9cf5d16fd82ccb1c0228d2ec2dc30effc1ea18e7"}
Jul 29 19:45:01.113863 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:01.113808    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755905-48dzc" podStartSLOduration=1.113790879 podStartE2EDuration="1.113790879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:45:01.113395275 +0000 UTC m=+1671.940894332" watchObservedRunningTime="2026-07-29 19:45:01.113790879 +0000 UTC m=+1671.941289936"
Jul 29 19:45:02.094455 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:02.094422    2557 generic.go:356] "Generic (PLEG): container finished" podID="6421eb66-5559-4396-a8e3-3be2e7a9ece1" containerID="617ff423a8b9e0babe954ffe776cd9e47ac1bfb28f479de0bf0a59c351fdc9d4" exitCode=0
Jul 29 19:45:02.094837 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:02.094520    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755905-48dzc" event={"ID":"6421eb66-5559-4396-a8e3-3be2e7a9ece1","Type":"ContainerDied","Data":"617ff423a8b9e0babe954ffe776cd9e47ac1bfb28f479de0bf0a59c351fdc9d4"}
Jul 29 19:45:03.225125 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:03.225094    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755905-48dzc"
Jul 29 19:45:04.105137 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:04.105059    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755905-48dzc" event={"ID":"6421eb66-5559-4396-a8e3-3be2e7a9ece1","Type":"ContainerDied","Data":"961da36751970be6f63c268b9cf5d16fd82ccb1c0228d2ec2dc30effc1ea18e7"}
Jul 29 19:45:04.105137 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:04.105081    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755905-48dzc"
Jul 29 19:45:04.105137 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:04.105117    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="961da36751970be6f63c268b9cf5d16fd82ccb1c0228d2ec2dc30effc1ea18e7"
Jul 29 19:45:18.182857 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:18.182817    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d0aaa6d0-77dc675b96-gd77z"]
Jul 29 19:45:18.183289 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:18.183157    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6421eb66-5559-4396-a8e3-3be2e7a9ece1" containerName="cleanup"
Jul 29 19:45:18.183289 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:18.183168    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6421eb66-5559-4396-a8e3-3be2e7a9ece1" containerName="cleanup"
Jul 29 19:45:18.183289 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:18.183242    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6421eb66-5559-4396-a8e3-3be2e7a9ece1" containerName="cleanup"
Jul 29 19:45:18.186153 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:18.186136    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0aaa6d0-77dc675b96-gd77z"
Jul 29 19:45:18.189920 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:18.189898    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d0aaa6d0-dockercfg-tmvmf\""
Jul 29 19:45:18.190078 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:18.190060    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d0aaa6d0\""
Jul 29 19:45:18.203837 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:18.203815    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d0aaa6d0-77dc675b96-gd77z"]
Jul 29 19:45:18.363992 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:18.363923    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d5172d-afd3-4937-b076-73adac293485-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d0aaa6d0-77dc675b96-gd77z\" (UID: \"e4d5172d-afd3-4937-b076-73adac293485\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d0aaa6d0-77dc675b96-gd77z"
Jul 29 19:45:18.364166 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:18.364008    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwn2r\" (UniqueName: \"kubernetes.io/projected/e4d5172d-afd3-4937-b076-73adac293485-kube-api-access-bwn2r\") pod \"payload-pre-processing-e2e-ait-d0aaa6d0-77dc675b96-gd77z\" (UID: \"e4d5172d-afd3-4937-b076-73adac293485\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d0aaa6d0-77dc675b96-gd77z"
Jul 29 19:45:18.430555 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:18.430520    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d0aaa6d0-845587dbb6-ng9b5"]
Jul 29 19:45:18.433783 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:18.433737    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d0aaa6d0-845587dbb6-ng9b5"
Jul 29 19:45:18.444747 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:18.444721    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d0aaa6d0-845587dbb6-ng9b5"]
Jul 29 19:45:18.465024 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:18.465000    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d5172d-afd3-4937-b076-73adac293485-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d0aaa6d0-77dc675b96-gd77z\" (UID: \"e4d5172d-afd3-4937-b076-73adac293485\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d0aaa6d0-77dc675b96-gd77z"
Jul 29 19:45:18.465138 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:18.465044    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwn2r\" (UniqueName: \"kubernetes.io/projected/e4d5172d-afd3-4937-b076-73adac293485-kube-api-access-bwn2r\") pod \"payload-pre-processing-e2e-ait-d0aaa6d0-77dc675b96-gd77z\" (UID: \"e4d5172d-afd3-4937-b076-73adac293485\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d0aaa6d0-77dc675b96-gd77z"
Jul 29 19:45:18.465649 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:18.465632    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d5172d-afd3-4937-b076-73adac293485-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d0aaa6d0-77dc675b96-gd77z\" (UID: \"e4d5172d-afd3-4937-b076-73adac293485\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d0aaa6d0-77dc675b96-gd77z"
Jul 29 19:45:18.483281 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:18.483255    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwn2r\" (UniqueName: \"kubernetes.io/projected/e4d5172d-afd3-4937-b076-73adac293485-kube-api-access-bwn2r\") pod \"payload-pre-processing-e2e-ait-d0aaa6d0-77dc675b96-gd77z\" (UID: \"e4d5172d-afd3-4937-b076-73adac293485\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d0aaa6d0-77dc675b96-gd77z"
Jul 29 19:45:18.496198 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:18.496179    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0aaa6d0-77dc675b96-gd77z"
Jul 29 19:45:18.565863 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:18.565832    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49qzz\" (UniqueName: \"kubernetes.io/projected/4cb067da-1802-4cd4-8637-1a3e673f562f-kube-api-access-49qzz\") pod \"payload-processing-e2e-ait-d0aaa6d0-845587dbb6-ng9b5\" (UID: \"4cb067da-1802-4cd4-8637-1a3e673f562f\") " pod="openshift-ingress/payload-processing-e2e-ait-d0aaa6d0-845587dbb6-ng9b5"
Jul 29 19:45:18.566009 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:18.565871    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb067da-1802-4cd4-8637-1a3e673f562f-plugins-config-volume\") pod \"payload-processing-e2e-ait-d0aaa6d0-845587dbb6-ng9b5\" (UID: \"4cb067da-1802-4cd4-8637-1a3e673f562f\") " pod="openshift-ingress/payload-processing-e2e-ait-d0aaa6d0-845587dbb6-ng9b5"
Jul 29 19:45:18.640017 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:18.639991    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d0aaa6d0-77dc675b96-gd77z"]
Jul 29 19:45:18.642741 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:45:18.642711    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4d5172d_afd3_4937_b076_73adac293485.slice/crio-ae8feb43b5a005cd3e92bd956957eff425281490673d53108a733ae467140c10 WatchSource:0}: Error finding container ae8feb43b5a005cd3e92bd956957eff425281490673d53108a733ae467140c10: Status 404 returned error can't find the container with id ae8feb43b5a005cd3e92bd956957eff425281490673d53108a733ae467140c10
Jul 29 19:45:18.667382 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:18.667358    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49qzz\" (UniqueName: \"kubernetes.io/projected/4cb067da-1802-4cd4-8637-1a3e673f562f-kube-api-access-49qzz\") pod \"payload-processing-e2e-ait-d0aaa6d0-845587dbb6-ng9b5\" (UID: \"4cb067da-1802-4cd4-8637-1a3e673f562f\") " pod="openshift-ingress/payload-processing-e2e-ait-d0aaa6d0-845587dbb6-ng9b5"
Jul 29 19:45:18.667473 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:18.667392    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb067da-1802-4cd4-8637-1a3e673f562f-plugins-config-volume\") pod \"payload-processing-e2e-ait-d0aaa6d0-845587dbb6-ng9b5\" (UID: \"4cb067da-1802-4cd4-8637-1a3e673f562f\") " pod="openshift-ingress/payload-processing-e2e-ait-d0aaa6d0-845587dbb6-ng9b5"
Jul 29 19:45:18.668011 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:18.667992    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb067da-1802-4cd4-8637-1a3e673f562f-plugins-config-volume\") pod \"payload-processing-e2e-ait-d0aaa6d0-845587dbb6-ng9b5\" (UID: \"4cb067da-1802-4cd4-8637-1a3e673f562f\") " pod="openshift-ingress/payload-processing-e2e-ait-d0aaa6d0-845587dbb6-ng9b5"
Jul 29 19:45:18.682819 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:18.682800    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49qzz\" (UniqueName: \"kubernetes.io/projected/4cb067da-1802-4cd4-8637-1a3e673f562f-kube-api-access-49qzz\") pod \"payload-processing-e2e-ait-d0aaa6d0-845587dbb6-ng9b5\" (UID: \"4cb067da-1802-4cd4-8637-1a3e673f562f\") " pod="openshift-ingress/payload-processing-e2e-ait-d0aaa6d0-845587dbb6-ng9b5"
Jul 29 19:45:18.744800 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:18.744768    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d0aaa6d0-845587dbb6-ng9b5"
Jul 29 19:45:18.876424 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:18.876397    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d0aaa6d0-845587dbb6-ng9b5"]
Jul 29 19:45:18.878631 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:45:18.878610    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cb067da_1802_4cd4_8637_1a3e673f562f.slice/crio-f350b569ff850f33389638b6bd4034c770b9560f9ecd71ee70575d61556b9a7d WatchSource:0}: Error finding container f350b569ff850f33389638b6bd4034c770b9560f9ecd71ee70575d61556b9a7d: Status 404 returned error can't find the container with id f350b569ff850f33389638b6bd4034c770b9560f9ecd71ee70575d61556b9a7d
Jul 29 19:45:19.176102 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:19.176059    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d0aaa6d0-845587dbb6-ng9b5" event={"ID":"4cb067da-1802-4cd4-8637-1a3e673f562f","Type":"ContainerStarted","Data":"578357cb30f998860cc69ee816fea908f8d274b58621a01328c7d361cf314b5e"}
Jul 29 19:45:19.176286 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:19.176127    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d0aaa6d0-845587dbb6-ng9b5"
Jul 29 19:45:19.176286 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:19.176153    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d0aaa6d0-845587dbb6-ng9b5" event={"ID":"4cb067da-1802-4cd4-8637-1a3e673f562f","Type":"ContainerStarted","Data":"f350b569ff850f33389638b6bd4034c770b9560f9ecd71ee70575d61556b9a7d"}
Jul 29 19:45:19.177557 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:19.177527    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0aaa6d0-77dc675b96-gd77z" event={"ID":"e4d5172d-afd3-4937-b076-73adac293485","Type":"ContainerStarted","Data":"aa7d246de1b06027b694e739a02fc134fa244b2e439bddca6c92778e525d680d"}
Jul 29 19:45:19.177676 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:19.177573    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0aaa6d0-77dc675b96-gd77z" event={"ID":"e4d5172d-afd3-4937-b076-73adac293485","Type":"ContainerStarted","Data":"ae8feb43b5a005cd3e92bd956957eff425281490673d53108a733ae467140c10"}
Jul 29 19:45:19.177676 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:19.177599    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0aaa6d0-77dc675b96-gd77z"
Jul 29 19:45:19.195297 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:19.195253    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d0aaa6d0-845587dbb6-ng9b5" podStartSLOduration=1.195236406 podStartE2EDuration="1.195236406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:45:19.195028864 +0000 UTC m=+1690.022527922" watchObservedRunningTime="2026-07-29 19:45:19.195236406 +0000 UTC m=+1690.022735465"
Jul 29 19:45:19.214776 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:19.214733    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0aaa6d0-77dc675b96-gd77z" podStartSLOduration=1.214720091 podStartE2EDuration="1.214720091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:45:19.21287944 +0000 UTC m=+1690.040378499" watchObservedRunningTime="2026-07-29 19:45:19.214720091 +0000 UTC m=+1690.042219180"
Jul 29 19:45:30.184117 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:30.184084    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d0aaa6d0-845587dbb6-ng9b5"
Jul 29 19:45:30.184494 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:30.184153    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0aaa6d0-77dc675b96-gd77z"
Jul 29 19:45:41.239042 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:41.239007    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0aaa6d0-drhzk"]
Jul 29 19:45:41.242214 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:41.242197    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0aaa6d0-drhzk"
Jul 29 19:45:41.245564 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:41.245542    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jsq2r\""
Jul 29 19:45:41.252875 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:41.252853    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0aaa6d0-drhzk"]
Jul 29 19:45:41.348045 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:41.348014    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8e6c1c9-3653-41ec-82c8-f1c2cbdcd8d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d0aaa6d0-drhzk\" (UID: \"c8e6c1c9-3653-41ec-82c8-f1c2cbdcd8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0aaa6d0-drhzk"
Jul 29 19:45:41.448743 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:41.448709    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8e6c1c9-3653-41ec-82c8-f1c2cbdcd8d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d0aaa6d0-drhzk\" (UID: \"c8e6c1c9-3653-41ec-82c8-f1c2cbdcd8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0aaa6d0-drhzk"
Jul 29 19:45:41.449380 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:41.449361    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8e6c1c9-3653-41ec-82c8-f1c2cbdcd8d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d0aaa6d0-drhzk\" (UID: \"c8e6c1c9-3653-41ec-82c8-f1c2cbdcd8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0aaa6d0-drhzk"
Jul 29 19:45:41.553946 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:41.553873    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0aaa6d0-drhzk"
Jul 29 19:45:41.681976 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:41.681944    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0aaa6d0-drhzk"]
Jul 29 19:45:41.684018 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:45:41.683986    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e6c1c9_3653_41ec_82c8_f1c2cbdcd8d0.slice/crio-e44d6d312bffa706921ab47d25827146e749b1acdd4864c66e9b06e82c52702e WatchSource:0}: Error finding container e44d6d312bffa706921ab47d25827146e749b1acdd4864c66e9b06e82c52702e: Status 404 returned error can't find the container with id e44d6d312bffa706921ab47d25827146e749b1acdd4864c66e9b06e82c52702e
Jul 29 19:45:42.282871 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:42.282843    2557 generic.go:356] "Generic (PLEG): container finished" podID="c8e6c1c9-3653-41ec-82c8-f1c2cbdcd8d0" containerID="9c8068b5a68b192400f4470aec4c68f1c6d7654edd2b6d50c05ebb26744217a2" exitCode=0
Jul 29 19:45:42.283277 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:42.282900    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0aaa6d0-drhzk" event={"ID":"c8e6c1c9-3653-41ec-82c8-f1c2cbdcd8d0","Type":"ContainerDied","Data":"9c8068b5a68b192400f4470aec4c68f1c6d7654edd2b6d50c05ebb26744217a2"}
Jul 29 19:45:42.283277 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:42.282941    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0aaa6d0-drhzk" event={"ID":"c8e6c1c9-3653-41ec-82c8-f1c2cbdcd8d0","Type":"ContainerStarted","Data":"e44d6d312bffa706921ab47d25827146e749b1acdd4864c66e9b06e82c52702e"}
Jul 29 19:45:43.415771 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:43.415746    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0aaa6d0-drhzk"
Jul 29 19:45:43.468053 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:43.468025    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8e6c1c9-3653-41ec-82c8-f1c2cbdcd8d0-maas-api-service-ca\") pod \"c8e6c1c9-3653-41ec-82c8-f1c2cbdcd8d0\" (UID: \"c8e6c1c9-3653-41ec-82c8-f1c2cbdcd8d0\") "
Jul 29 19:45:43.468342 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:43.468321    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8e6c1c9-3653-41ec-82c8-f1c2cbdcd8d0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c8e6c1c9-3653-41ec-82c8-f1c2cbdcd8d0" (UID: "c8e6c1c9-3653-41ec-82c8-f1c2cbdcd8d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:45:43.569034 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:43.568936    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8e6c1c9-3653-41ec-82c8-f1c2cbdcd8d0-maas-api-service-ca\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:45:44.293715 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:44.293678    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0aaa6d0-drhzk" event={"ID":"c8e6c1c9-3653-41ec-82c8-f1c2cbdcd8d0","Type":"ContainerDied","Data":"e44d6d312bffa706921ab47d25827146e749b1acdd4864c66e9b06e82c52702e"}
Jul 29 19:45:44.293715 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:44.293711    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0aaa6d0-drhzk"
Jul 29 19:45:44.293930 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:44.293755    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e44d6d312bffa706921ab47d25827146e749b1acdd4864c66e9b06e82c52702e"
Jul 29 19:45:47.756178 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:47.756145    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d0aaa6d0-845587dbb6-ng9b5"]
Jul 29 19:45:47.758611 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:47.756363    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d0aaa6d0-845587dbb6-ng9b5" podUID="4cb067da-1802-4cd4-8637-1a3e673f562f" containerName="payload-processing" containerID="cri-o://578357cb30f998860cc69ee816fea908f8d274b58621a01328c7d361cf314b5e" gracePeriod=30
Jul 29 19:45:47.762899 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:47.762869    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d0aaa6d0-77dc675b96-gd77z"]
Jul 29 19:45:47.763212 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:47.763180    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0aaa6d0-77dc675b96-gd77z" podUID="e4d5172d-afd3-4937-b076-73adac293485" containerName="payload-pre-processing" containerID="cri-o://aa7d246de1b06027b694e739a02fc134fa244b2e439bddca6c92778e525d680d" gracePeriod=30
Jul 29 19:45:47.929129 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:47.929064    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0aaa6d0-drhzk"]
Jul 29 19:45:47.930826 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:47.930794    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0aaa6d0-drhzk"]
Jul 29 19:45:48.030143 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:48.030123    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d0aaa6d0-845587dbb6-ng9b5"
Jul 29 19:45:48.033207 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:48.033185    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0aaa6d0-77dc675b96-gd77z"
Jul 29 19:45:48.106397 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:48.106370    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d5172d-afd3-4937-b076-73adac293485-plugins-config-volume\") pod \"e4d5172d-afd3-4937-b076-73adac293485\" (UID: \"e4d5172d-afd3-4937-b076-73adac293485\") "
Jul 29 19:45:48.106397 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:48.106428    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb067da-1802-4cd4-8637-1a3e673f562f-plugins-config-volume\") pod \"4cb067da-1802-4cd4-8637-1a3e673f562f\" (UID: \"4cb067da-1802-4cd4-8637-1a3e673f562f\") "
Jul 29 19:45:48.106667 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:48.106486    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwn2r\" (UniqueName: \"kubernetes.io/projected/e4d5172d-afd3-4937-b076-73adac293485-kube-api-access-bwn2r\") pod \"e4d5172d-afd3-4937-b076-73adac293485\" (UID: \"e4d5172d-afd3-4937-b076-73adac293485\") "
Jul 29 19:45:48.106667 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:48.106529    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49qzz\" (UniqueName: \"kubernetes.io/projected/4cb067da-1802-4cd4-8637-1a3e673f562f-kube-api-access-49qzz\") pod \"4cb067da-1802-4cd4-8637-1a3e673f562f\" (UID: \"4cb067da-1802-4cd4-8637-1a3e673f562f\") "
Jul 29 19:45:48.106810 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:48.106784    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4d5172d-afd3-4937-b076-73adac293485-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e4d5172d-afd3-4937-b076-73adac293485" (UID: "e4d5172d-afd3-4937-b076-73adac293485"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:45:48.106873 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:48.106849    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cb067da-1802-4cd4-8637-1a3e673f562f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4cb067da-1802-4cd4-8637-1a3e673f562f" (UID: "4cb067da-1802-4cd4-8637-1a3e673f562f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:45:48.108721 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:48.108690    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cb067da-1802-4cd4-8637-1a3e673f562f-kube-api-access-49qzz" (OuterVolumeSpecName: "kube-api-access-49qzz") pod "4cb067da-1802-4cd4-8637-1a3e673f562f" (UID: "4cb067da-1802-4cd4-8637-1a3e673f562f"). InnerVolumeSpecName "kube-api-access-49qzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:45:48.108835 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:48.108699    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4d5172d-afd3-4937-b076-73adac293485-kube-api-access-bwn2r" (OuterVolumeSpecName: "kube-api-access-bwn2r") pod "e4d5172d-afd3-4937-b076-73adac293485" (UID: "e4d5172d-afd3-4937-b076-73adac293485"). InnerVolumeSpecName "kube-api-access-bwn2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:45:48.207875 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:48.207839    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwn2r\" (UniqueName: \"kubernetes.io/projected/e4d5172d-afd3-4937-b076-73adac293485-kube-api-access-bwn2r\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:45:48.207875 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:48.207867    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49qzz\" (UniqueName: \"kubernetes.io/projected/4cb067da-1802-4cd4-8637-1a3e673f562f-kube-api-access-49qzz\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:45:48.207875 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:48.207877    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d5172d-afd3-4937-b076-73adac293485-plugins-config-volume\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:45:48.208111 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:48.207887    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb067da-1802-4cd4-8637-1a3e673f562f-plugins-config-volume\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:45:48.313625 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:48.313541    2557 generic.go:356] "Generic (PLEG): container finished" podID="4cb067da-1802-4cd4-8637-1a3e673f562f" containerID="578357cb30f998860cc69ee816fea908f8d274b58621a01328c7d361cf314b5e" exitCode=0
Jul 29 19:45:48.313625 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:48.313583    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d0aaa6d0-845587dbb6-ng9b5"
Jul 29 19:45:48.313835 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:48.313635    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d0aaa6d0-845587dbb6-ng9b5" event={"ID":"4cb067da-1802-4cd4-8637-1a3e673f562f","Type":"ContainerDied","Data":"578357cb30f998860cc69ee816fea908f8d274b58621a01328c7d361cf314b5e"}
Jul 29 19:45:48.313835 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:48.313689    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d0aaa6d0-845587dbb6-ng9b5" event={"ID":"4cb067da-1802-4cd4-8637-1a3e673f562f","Type":"ContainerDied","Data":"f350b569ff850f33389638b6bd4034c770b9560f9ecd71ee70575d61556b9a7d"}
Jul 29 19:45:48.313835 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:48.313719    2557 scope.go:117] "RemoveContainer" containerID="578357cb30f998860cc69ee816fea908f8d274b58621a01328c7d361cf314b5e"
Jul 29 19:45:48.314958 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:48.314939    2557 generic.go:356] "Generic (PLEG): container finished" podID="e4d5172d-afd3-4937-b076-73adac293485" containerID="aa7d246de1b06027b694e739a02fc134fa244b2e439bddca6c92778e525d680d" exitCode=0
Jul 29 19:45:48.315075 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:48.314990    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0aaa6d0-77dc675b96-gd77z" event={"ID":"e4d5172d-afd3-4937-b076-73adac293485","Type":"ContainerDied","Data":"aa7d246de1b06027b694e739a02fc134fa244b2e439bddca6c92778e525d680d"}
Jul 29 19:45:48.315075 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:48.315013    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0aaa6d0-77dc675b96-gd77z"
Jul 29 19:45:48.315075 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:48.315045    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0aaa6d0-77dc675b96-gd77z" event={"ID":"e4d5172d-afd3-4937-b076-73adac293485","Type":"ContainerDied","Data":"ae8feb43b5a005cd3e92bd956957eff425281490673d53108a733ae467140c10"}
Jul 29 19:45:48.322852 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:48.322830    2557 scope.go:117] "RemoveContainer" containerID="578357cb30f998860cc69ee816fea908f8d274b58621a01328c7d361cf314b5e"
Jul 29 19:45:48.323204 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:45:48.323177    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"578357cb30f998860cc69ee816fea908f8d274b58621a01328c7d361cf314b5e\": container with ID starting with 578357cb30f998860cc69ee816fea908f8d274b58621a01328c7d361cf314b5e not found: ID does not exist" containerID="578357cb30f998860cc69ee816fea908f8d274b58621a01328c7d361cf314b5e"
Jul 29 19:45:48.323320 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:48.323213    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"578357cb30f998860cc69ee816fea908f8d274b58621a01328c7d361cf314b5e"} err="failed to get container status \"578357cb30f998860cc69ee816fea908f8d274b58621a01328c7d361cf314b5e\": rpc error: code = NotFound desc = could not find container \"578357cb30f998860cc69ee816fea908f8d274b58621a01328c7d361cf314b5e\": container with ID starting with 578357cb30f998860cc69ee816fea908f8d274b58621a01328c7d361cf314b5e not found: ID does not exist"
Jul 29 19:45:48.323320 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:48.323236    2557 scope.go:117] "RemoveContainer" containerID="aa7d246de1b06027b694e739a02fc134fa244b2e439bddca6c92778e525d680d"
Jul 29 19:45:48.330980 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:48.330951    2557 scope.go:117] "RemoveContainer" containerID="aa7d246de1b06027b694e739a02fc134fa244b2e439bddca6c92778e525d680d"
Jul 29 19:45:48.331234 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:45:48.331216    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa7d246de1b06027b694e739a02fc134fa244b2e439bddca6c92778e525d680d\": container with ID starting with aa7d246de1b06027b694e739a02fc134fa244b2e439bddca6c92778e525d680d not found: ID does not exist" containerID="aa7d246de1b06027b694e739a02fc134fa244b2e439bddca6c92778e525d680d"
Jul 29 19:45:48.331294 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:48.331241    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa7d246de1b06027b694e739a02fc134fa244b2e439bddca6c92778e525d680d"} err="failed to get container status \"aa7d246de1b06027b694e739a02fc134fa244b2e439bddca6c92778e525d680d\": rpc error: code = NotFound desc = could not find container \"aa7d246de1b06027b694e739a02fc134fa244b2e439bddca6c92778e525d680d\": container with ID starting with aa7d246de1b06027b694e739a02fc134fa244b2e439bddca6c92778e525d680d not found: ID does not exist"
Jul 29 19:45:48.344137 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:48.344117    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d0aaa6d0-845587dbb6-ng9b5"]
Jul 29 19:45:48.354116 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:48.354093    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d0aaa6d0-845587dbb6-ng9b5"]
Jul 29 19:45:48.362735 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:48.362713    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d0aaa6d0-77dc675b96-gd77z"]
Jul 29 19:45:48.368735 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:48.368714    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d0aaa6d0-77dc675b96-gd77z"]
Jul 29 19:45:49.678202 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:49.678170    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cb067da-1802-4cd4-8637-1a3e673f562f" path="/var/lib/kubelet/pods/4cb067da-1802-4cd4-8637-1a3e673f562f/volumes"
Jul 29 19:45:49.678604 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:49.678587    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8e6c1c9-3653-41ec-82c8-f1c2cbdcd8d0" path="/var/lib/kubelet/pods/c8e6c1c9-3653-41ec-82c8-f1c2cbdcd8d0/volumes"
Jul 29 19:45:49.679001 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:49.678987    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4d5172d-afd3-4937-b076-73adac293485" path="/var/lib/kubelet/pods/e4d5172d-afd3-4937-b076-73adac293485/volumes"
Jul 29 19:45:52.114582 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:52.114547    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc"]
Jul 29 19:45:52.115097 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:52.115077    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cb067da-1802-4cd4-8637-1a3e673f562f" containerName="payload-processing"
Jul 29 19:45:52.115165 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:52.115099    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cb067da-1802-4cd4-8637-1a3e673f562f" containerName="payload-processing"
Jul 29 19:45:52.115165 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:52.115111    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8e6c1c9-3653-41ec-82c8-f1c2cbdcd8d0" containerName="revoke-keys"
Jul 29 19:45:52.115165 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:52.115119    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8e6c1c9-3653-41ec-82c8-f1c2cbdcd8d0" containerName="revoke-keys"
Jul 29 19:45:52.115328 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:52.115164    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4d5172d-afd3-4937-b076-73adac293485" containerName="payload-pre-processing"
Jul 29 19:45:52.115328 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:52.115181    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4d5172d-afd3-4937-b076-73adac293485" containerName="payload-pre-processing"
Jul 29 19:45:52.115328 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:52.115288    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8e6c1c9-3653-41ec-82c8-f1c2cbdcd8d0" containerName="revoke-keys"
Jul 29 19:45:52.115328 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:52.115303    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cb067da-1802-4cd4-8637-1a3e673f562f" containerName="payload-processing"
Jul 29 19:45:52.115328 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:52.115317    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4d5172d-afd3-4937-b076-73adac293485" containerName="payload-pre-processing"
Jul 29 19:45:52.119670 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:52.119649    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc"
Jul 29 19:45:52.122305 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:52.122276    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-13baf70f-serving-cert\""
Jul 29 19:45:52.122442 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:52.122396    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4dw6s\""
Jul 29 19:45:52.129747 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:52.129722    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc"]
Jul 29 19:45:52.243754 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:52.243711    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wskb6\" (UniqueName: \"kubernetes.io/projected/c1d9c868-3873-4067-b133-7aeb46f61f50-kube-api-access-wskb6\") pod \"maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc\" (UID: \"c1d9c868-3873-4067-b133-7aeb46f61f50\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc"
Jul 29 19:45:52.243754 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:52.243751    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1d9c868-3873-4067-b133-7aeb46f61f50-maas-api-tls\") pod \"maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc\" (UID: \"c1d9c868-3873-4067-b133-7aeb46f61f50\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc"
Jul 29 19:45:52.244006 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:52.243864    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1d9c868-3873-4067-b133-7aeb46f61f50-openshift-service-ca\") pod \"maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc\" (UID: \"c1d9c868-3873-4067-b133-7aeb46f61f50\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc"
Jul 29 19:45:52.345293 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:52.345262    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wskb6\" (UniqueName: \"kubernetes.io/projected/c1d9c868-3873-4067-b133-7aeb46f61f50-kube-api-access-wskb6\") pod \"maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc\" (UID: \"c1d9c868-3873-4067-b133-7aeb46f61f50\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc"
Jul 29 19:45:52.345293 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:52.345294    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1d9c868-3873-4067-b133-7aeb46f61f50-maas-api-tls\") pod \"maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc\" (UID: \"c1d9c868-3873-4067-b133-7aeb46f61f50\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc"
Jul 29 19:45:52.345498 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:52.345344    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1d9c868-3873-4067-b133-7aeb46f61f50-openshift-service-ca\") pod \"maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc\" (UID: \"c1d9c868-3873-4067-b133-7aeb46f61f50\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc"
Jul 29 19:45:52.345498 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:45:52.345478    2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-13baf70f-serving-cert: secret "maas-api-e2e-migrate-13baf70f-serving-cert" not found
Jul 29 19:45:52.345576 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:45:52.345539    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1d9c868-3873-4067-b133-7aeb46f61f50-maas-api-tls podName:c1d9c868-3873-4067-b133-7aeb46f61f50 nodeName:}" failed. No retries permitted until 2026-07-29 19:45:52.845520516 +0000 UTC m=+1723.673019563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c1d9c868-3873-4067-b133-7aeb46f61f50-maas-api-tls") pod "maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc" (UID: "c1d9c868-3873-4067-b133-7aeb46f61f50") : secret "maas-api-e2e-migrate-13baf70f-serving-cert" not found
Jul 29 19:45:52.346100 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:52.346082    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1d9c868-3873-4067-b133-7aeb46f61f50-openshift-service-ca\") pod \"maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc\" (UID: \"c1d9c868-3873-4067-b133-7aeb46f61f50\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc"
Jul 29 19:45:52.357043 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:52.357021    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wskb6\" (UniqueName: \"kubernetes.io/projected/c1d9c868-3873-4067-b133-7aeb46f61f50-kube-api-access-wskb6\") pod \"maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc\" (UID: \"c1d9c868-3873-4067-b133-7aeb46f61f50\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc"
Jul 29 19:45:52.850957 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:52.850922    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1d9c868-3873-4067-b133-7aeb46f61f50-maas-api-tls\") pod \"maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc\" (UID: \"c1d9c868-3873-4067-b133-7aeb46f61f50\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc"
Jul 29 19:45:52.853341 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:52.853314    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1d9c868-3873-4067-b133-7aeb46f61f50-maas-api-tls\") pod \"maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc\" (UID: \"c1d9c868-3873-4067-b133-7aeb46f61f50\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc"
Jul 29 19:45:53.032248 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:53.032211    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc"
Jul 29 19:45:53.054921 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:53.054893    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-13baf70f-64f7ccff68-6ssl6"]
Jul 29 19:45:53.058749 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:53.058726    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-13baf70f-64f7ccff68-6ssl6"
Jul 29 19:45:53.061825 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:53.061806    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-13baf70f-dockercfg-lbdxk\""
Jul 29 19:45:53.062147 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:53.062122    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-13baf70f\""
Jul 29 19:45:53.071077 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:53.071054    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-13baf70f-64f7ccff68-6ssl6"]
Jul 29 19:45:53.127679 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:53.127647    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-13baf70f-664565f85c-mv8b2"]
Jul 29 19:45:53.131587 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:53.131551    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-13baf70f-664565f85c-mv8b2"
Jul 29 19:45:53.142375 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:53.142354    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-13baf70f-664565f85c-mv8b2"]
Jul 29 19:45:53.153959 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:53.153885    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zz76\" (UniqueName: \"kubernetes.io/projected/ced1247a-4b4c-48e1-a061-3e49a7265c19-kube-api-access-5zz76\") pod \"payload-pre-processing-e2e-migrate-13baf70f-64f7ccff68-6ssl6\" (UID: \"ced1247a-4b4c-48e1-a061-3e49a7265c19\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-13baf70f-64f7ccff68-6ssl6"
Jul 29 19:45:53.154111 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:53.154069    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ced1247a-4b4c-48e1-a061-3e49a7265c19-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-13baf70f-64f7ccff68-6ssl6\" (UID: \"ced1247a-4b4c-48e1-a061-3e49a7265c19\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-13baf70f-64f7ccff68-6ssl6"
Jul 29 19:45:53.183980 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:53.183943    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc"]
Jul 29 19:45:53.185977 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:45:53.185938    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d9c868_3873_4067_b133_7aeb46f61f50.slice/crio-bd65238f7ceda7d8c1b1123afcd230ee1c5d2fefb34a56228b980878d6e89472 WatchSource:0}: Error finding container bd65238f7ceda7d8c1b1123afcd230ee1c5d2fefb34a56228b980878d6e89472: Status 404 returned error can't find the container with id bd65238f7ceda7d8c1b1123afcd230ee1c5d2fefb34a56228b980878d6e89472
Jul 29 19:45:53.187356 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:53.187337    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 19:45:53.255518 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:53.255487    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zz76\" (UniqueName: \"kubernetes.io/projected/ced1247a-4b4c-48e1-a061-3e49a7265c19-kube-api-access-5zz76\") pod \"payload-pre-processing-e2e-migrate-13baf70f-64f7ccff68-6ssl6\" (UID: \"ced1247a-4b4c-48e1-a061-3e49a7265c19\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-13baf70f-64f7ccff68-6ssl6"
Jul 29 19:45:53.255640 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:53.255575    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ced1247a-4b4c-48e1-a061-3e49a7265c19-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-13baf70f-64f7ccff68-6ssl6\" (UID: \"ced1247a-4b4c-48e1-a061-3e49a7265c19\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-13baf70f-64f7ccff68-6ssl6"
Jul 29 19:45:53.255640 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:53.255600    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-887d6\" (UniqueName: \"kubernetes.io/projected/5c057a00-afe2-419d-929d-dbc8624e3ee5-kube-api-access-887d6\") pod \"payload-processing-e2e-migrate-13baf70f-664565f85c-mv8b2\" (UID: \"5c057a00-afe2-419d-929d-dbc8624e3ee5\") " pod="openshift-ingress/payload-processing-e2e-migrate-13baf70f-664565f85c-mv8b2"
Jul 29 19:45:53.255640 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:53.255630    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c057a00-afe2-419d-929d-dbc8624e3ee5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-13baf70f-664565f85c-mv8b2\" (UID: \"5c057a00-afe2-419d-929d-dbc8624e3ee5\") " pod="openshift-ingress/payload-processing-e2e-migrate-13baf70f-664565f85c-mv8b2"
Jul 29 19:45:53.256284 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:53.256266    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ced1247a-4b4c-48e1-a061-3e49a7265c19-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-13baf70f-64f7ccff68-6ssl6\" (UID: \"ced1247a-4b4c-48e1-a061-3e49a7265c19\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-13baf70f-64f7ccff68-6ssl6"
Jul 29 19:45:53.265767 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:53.265747    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zz76\" (UniqueName: \"kubernetes.io/projected/ced1247a-4b4c-48e1-a061-3e49a7265c19-kube-api-access-5zz76\") pod \"payload-pre-processing-e2e-migrate-13baf70f-64f7ccff68-6ssl6\" (UID: \"ced1247a-4b4c-48e1-a061-3e49a7265c19\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-13baf70f-64f7ccff68-6ssl6"
Jul 29 19:45:53.342370 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:53.342333    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc" event={"ID":"c1d9c868-3873-4067-b133-7aeb46f61f50","Type":"ContainerStarted","Data":"bd65238f7ceda7d8c1b1123afcd230ee1c5d2fefb34a56228b980878d6e89472"}
Jul 29 19:45:53.357022 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:53.356988    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-887d6\" (UniqueName: \"kubernetes.io/projected/5c057a00-afe2-419d-929d-dbc8624e3ee5-kube-api-access-887d6\") pod \"payload-processing-e2e-migrate-13baf70f-664565f85c-mv8b2\" (UID: \"5c057a00-afe2-419d-929d-dbc8624e3ee5\") " pod="openshift-ingress/payload-processing-e2e-migrate-13baf70f-664565f85c-mv8b2"
Jul 29 19:45:53.357125 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:53.357051    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c057a00-afe2-419d-929d-dbc8624e3ee5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-13baf70f-664565f85c-mv8b2\" (UID: \"5c057a00-afe2-419d-929d-dbc8624e3ee5\") " pod="openshift-ingress/payload-processing-e2e-migrate-13baf70f-664565f85c-mv8b2"
Jul 29 19:45:53.357725 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:53.357703    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c057a00-afe2-419d-929d-dbc8624e3ee5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-13baf70f-664565f85c-mv8b2\" (UID: \"5c057a00-afe2-419d-929d-dbc8624e3ee5\") " pod="openshift-ingress/payload-processing-e2e-migrate-13baf70f-664565f85c-mv8b2"
Jul 29 19:45:53.367304 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:53.367280    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-887d6\" (UniqueName: \"kubernetes.io/projected/5c057a00-afe2-419d-929d-dbc8624e3ee5-kube-api-access-887d6\") pod \"payload-processing-e2e-migrate-13baf70f-664565f85c-mv8b2\" (UID: \"5c057a00-afe2-419d-929d-dbc8624e3ee5\") " pod="openshift-ingress/payload-processing-e2e-migrate-13baf70f-664565f85c-mv8b2"
Jul 29 19:45:53.370897 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:53.370836    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-13baf70f-64f7ccff68-6ssl6"
Jul 29 19:45:53.446396 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:53.446362    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-13baf70f-664565f85c-mv8b2"
Jul 29 19:45:53.514081 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:53.514043    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-13baf70f-64f7ccff68-6ssl6"]
Jul 29 19:45:53.515488 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:45:53.515464    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced1247a_4b4c_48e1_a061_3e49a7265c19.slice/crio-5c1a8ad69e90d60e7d0ff47809ac6068ea74828bba271acf2190e447a9520f4c WatchSource:0}: Error finding container 5c1a8ad69e90d60e7d0ff47809ac6068ea74828bba271acf2190e447a9520f4c: Status 404 returned error can't find the container with id 5c1a8ad69e90d60e7d0ff47809ac6068ea74828bba271acf2190e447a9520f4c
Jul 29 19:45:53.597023 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:53.596993    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-13baf70f-664565f85c-mv8b2"]
Jul 29 19:45:53.599216 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:45:53.599183    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c057a00_afe2_419d_929d_dbc8624e3ee5.slice/crio-e875a0ec1e3a7401b3422498506a0968fc2cfd9d8e54bbe9e95246ea9a8c28f2 WatchSource:0}: Error finding container e875a0ec1e3a7401b3422498506a0968fc2cfd9d8e54bbe9e95246ea9a8c28f2: Status 404 returned error can't find the container with id e875a0ec1e3a7401b3422498506a0968fc2cfd9d8e54bbe9e95246ea9a8c28f2
Jul 29 19:45:54.360382 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:54.360335    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-13baf70f-64f7ccff68-6ssl6" event={"ID":"ced1247a-4b4c-48e1-a061-3e49a7265c19","Type":"ContainerStarted","Data":"9b9fbf294476bfe65c17debbeff9b5ad50c00b5e9a487edbb028450bb38217ad"}
Jul 29 19:45:54.360856 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:54.360450    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-13baf70f-64f7ccff68-6ssl6" event={"ID":"ced1247a-4b4c-48e1-a061-3e49a7265c19","Type":"ContainerStarted","Data":"5c1a8ad69e90d60e7d0ff47809ac6068ea74828bba271acf2190e447a9520f4c"}
Jul 29 19:45:54.360856 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:54.360481    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-13baf70f-64f7ccff68-6ssl6"
Jul 29 19:45:54.362544 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:54.362506    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-13baf70f-664565f85c-mv8b2" event={"ID":"5c057a00-afe2-419d-929d-dbc8624e3ee5","Type":"ContainerStarted","Data":"a187ea6cb44aabc878432796712625404df871ce80a0f54b502d5ff78c7bfc4c"}
Jul 29 19:45:54.362668 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:54.362580    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-13baf70f-664565f85c-mv8b2" event={"ID":"5c057a00-afe2-419d-929d-dbc8624e3ee5","Type":"ContainerStarted","Data":"e875a0ec1e3a7401b3422498506a0968fc2cfd9d8e54bbe9e95246ea9a8c28f2"}
Jul 29 19:45:54.362736 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:54.362716    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-13baf70f-664565f85c-mv8b2"
Jul 29 19:45:54.382543 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:54.382482    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-13baf70f-64f7ccff68-6ssl6" podStartSLOduration=1.382462831 podStartE2EDuration="1.382462831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:45:54.379860478 +0000 UTC m=+1725.207359536" watchObservedRunningTime="2026-07-29 19:45:54.382462831 +0000 UTC m=+1725.209961891"
Jul 29 19:45:54.399177 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:54.399112    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-13baf70f-664565f85c-mv8b2" podStartSLOduration=1.39909279 podStartE2EDuration="1.39909279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:45:54.398887881 +0000 UTC m=+1725.226387053" watchObservedRunningTime="2026-07-29 19:45:54.39909279 +0000 UTC m=+1725.226591846"
Jul 29 19:45:56.373417 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:56.373365    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc" event={"ID":"c1d9c868-3873-4067-b133-7aeb46f61f50","Type":"ContainerStarted","Data":"e83a4db30d459507ea798013e9e70123283513cfe38a1e8de13c7c7ee3eef8cd"}
Jul 29 19:45:56.373895 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:56.373430    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc"
Jul 29 19:45:56.393298 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:45:56.393248    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc" podStartSLOduration=2.193176885 podStartE2EDuration="4.39323098s" totalImagesPullingTime="2.200054095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:45:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:45:56.392094576 +0000 UTC m=+1727.219593633" watchObservedRunningTime="2026-07-29 19:45:56.39323098 +0000 UTC m=+1727.220730040"
Jul 29 19:46:02.383874 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:02.383838    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc"
Jul 29 19:46:05.369517 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:05.369487    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-13baf70f-664565f85c-mv8b2"
Jul 29 19:46:05.369864 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:05.369566    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-13baf70f-64f7ccff68-6ssl6"
Jul 29 19:46:16.989775 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:16.989744    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13baf70f-fjnzf"]
Jul 29 19:46:16.994317 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:16.994288    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13baf70f-fjnzf"
Jul 29 19:46:16.997861 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:16.997670    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jsq2r\""
Jul 29 19:46:17.001125 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:17.001098    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13baf70f-fjnzf"]
Jul 29 19:46:17.070146 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:17.070112    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a7fe31-2c46-4a66-b4ed-c8b86abc5311-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-13baf70f-fjnzf\" (UID: \"87a7fe31-2c46-4a66-b4ed-c8b86abc5311\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13baf70f-fjnzf"
Jul 29 19:46:17.170610 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:17.170574    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a7fe31-2c46-4a66-b4ed-c8b86abc5311-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-13baf70f-fjnzf\" (UID: \"87a7fe31-2c46-4a66-b4ed-c8b86abc5311\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13baf70f-fjnzf"
Jul 29 19:46:17.171228 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:17.171208    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a7fe31-2c46-4a66-b4ed-c8b86abc5311-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-13baf70f-fjnzf\" (UID: \"87a7fe31-2c46-4a66-b4ed-c8b86abc5311\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13baf70f-fjnzf"
Jul 29 19:46:17.308329 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:17.308243    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13baf70f-fjnzf"
Jul 29 19:46:17.437397 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:17.437370    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13baf70f-fjnzf"]
Jul 29 19:46:17.439543 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:46:17.439513    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a7fe31_2c46_4a66_b4ed_c8b86abc5311.slice/crio-5d99eaddc5e257e85193af19c2aad7f0a21c6a49a18f7723af1ce450b9752651 WatchSource:0}: Error finding container 5d99eaddc5e257e85193af19c2aad7f0a21c6a49a18f7723af1ce450b9752651: Status 404 returned error can't find the container with id 5d99eaddc5e257e85193af19c2aad7f0a21c6a49a18f7723af1ce450b9752651
Jul 29 19:46:17.473007 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:17.472977    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13baf70f-fjnzf" event={"ID":"87a7fe31-2c46-4a66-b4ed-c8b86abc5311","Type":"ContainerStarted","Data":"5d99eaddc5e257e85193af19c2aad7f0a21c6a49a18f7723af1ce450b9752651"}
Jul 29 19:46:18.481940 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:18.481846    2557 generic.go:356] "Generic (PLEG): container finished" podID="87a7fe31-2c46-4a66-b4ed-c8b86abc5311" containerID="ff2a160835ad6473af28772261f0c487991516a30f91f78705249c43e9205a80" exitCode=0
Jul 29 19:46:18.482346 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:18.481930    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13baf70f-fjnzf" event={"ID":"87a7fe31-2c46-4a66-b4ed-c8b86abc5311","Type":"ContainerDied","Data":"ff2a160835ad6473af28772261f0c487991516a30f91f78705249c43e9205a80"}
Jul 29 19:46:19.617467 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:19.617445    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13baf70f-fjnzf"
Jul 29 19:46:19.689910 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:19.689884    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a7fe31-2c46-4a66-b4ed-c8b86abc5311-maas-api-service-ca\") pod \"87a7fe31-2c46-4a66-b4ed-c8b86abc5311\" (UID: \"87a7fe31-2c46-4a66-b4ed-c8b86abc5311\") "
Jul 29 19:46:19.690213 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:19.690193    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87a7fe31-2c46-4a66-b4ed-c8b86abc5311-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "87a7fe31-2c46-4a66-b4ed-c8b86abc5311" (UID: "87a7fe31-2c46-4a66-b4ed-c8b86abc5311"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:46:19.791248 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:19.791159    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a7fe31-2c46-4a66-b4ed-c8b86abc5311-maas-api-service-ca\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:46:20.492552 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:20.492518    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13baf70f-fjnzf" event={"ID":"87a7fe31-2c46-4a66-b4ed-c8b86abc5311","Type":"ContainerDied","Data":"5d99eaddc5e257e85193af19c2aad7f0a21c6a49a18f7723af1ce450b9752651"}
Jul 29 19:46:20.492552 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:20.492546    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13baf70f-fjnzf"
Jul 29 19:46:20.492750 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:20.492569    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d99eaddc5e257e85193af19c2aad7f0a21c6a49a18f7723af1ce450b9752651"
Jul 29 19:46:23.456297 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:23.456263    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc"]
Jul 29 19:46:23.456701 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:23.456473    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc" podUID="c1d9c868-3873-4067-b133-7aeb46f61f50" containerName="maas-api" containerID="cri-o://e83a4db30d459507ea798013e9e70123283513cfe38a1e8de13c7c7ee3eef8cd" gracePeriod=30
Jul 29 19:46:23.521094 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:23.521062    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-13baf70f-664565f85c-mv8b2"]
Jul 29 19:46:23.521335 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:23.521310    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-13baf70f-664565f85c-mv8b2" podUID="5c057a00-afe2-419d-929d-dbc8624e3ee5" containerName="payload-processing" containerID="cri-o://a187ea6cb44aabc878432796712625404df871ce80a0f54b502d5ff78c7bfc4c" gracePeriod=30
Jul 29 19:46:23.525111 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:23.525083    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-13baf70f-64f7ccff68-6ssl6"]
Jul 29 19:46:23.525443 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:23.525350    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-13baf70f-64f7ccff68-6ssl6" podUID="ced1247a-4b4c-48e1-a061-3e49a7265c19" containerName="payload-pre-processing" containerID="cri-o://9b9fbf294476bfe65c17debbeff9b5ad50c00b5e9a487edbb028450bb38217ad" gracePeriod=30
Jul 29 19:46:23.679814 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:23.679783    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13baf70f-fjnzf"]
Jul 29 19:46:23.679814 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:23.679823    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13baf70f-fjnzf"]
Jul 29 19:46:23.779281 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:23.779250    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc"
Jul 29 19:46:23.809118 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:23.809082    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-13baf70f-664565f85c-mv8b2"
Jul 29 19:46:23.825896 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:23.825878    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wskb6\" (UniqueName: \"kubernetes.io/projected/c1d9c868-3873-4067-b133-7aeb46f61f50-kube-api-access-wskb6\") pod \"c1d9c868-3873-4067-b133-7aeb46f61f50\" (UID: \"c1d9c868-3873-4067-b133-7aeb46f61f50\") "
Jul 29 19:46:23.826056 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:23.825917    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1d9c868-3873-4067-b133-7aeb46f61f50-maas-api-tls\") pod \"c1d9c868-3873-4067-b133-7aeb46f61f50\" (UID: \"c1d9c868-3873-4067-b133-7aeb46f61f50\") "
Jul 29 19:46:23.826056 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:23.825934    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-887d6\" (UniqueName: \"kubernetes.io/projected/5c057a00-afe2-419d-929d-dbc8624e3ee5-kube-api-access-887d6\") pod \"5c057a00-afe2-419d-929d-dbc8624e3ee5\" (UID: \"5c057a00-afe2-419d-929d-dbc8624e3ee5\") "
Jul 29 19:46:23.826162 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:23.826053    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1d9c868-3873-4067-b133-7aeb46f61f50-openshift-service-ca\") pod \"c1d9c868-3873-4067-b133-7aeb46f61f50\" (UID: \"c1d9c868-3873-4067-b133-7aeb46f61f50\") "
Jul 29 19:46:23.826162 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:23.826091    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c057a00-afe2-419d-929d-dbc8624e3ee5-plugins-config-volume\") pod \"5c057a00-afe2-419d-929d-dbc8624e3ee5\" (UID: \"5c057a00-afe2-419d-929d-dbc8624e3ee5\") "
Jul 29 19:46:23.826780 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:23.826697    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1d9c868-3873-4067-b133-7aeb46f61f50-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c1d9c868-3873-4067-b133-7aeb46f61f50" (UID: "c1d9c868-3873-4067-b133-7aeb46f61f50"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:46:23.826780 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:23.826716    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c057a00-afe2-419d-929d-dbc8624e3ee5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5c057a00-afe2-419d-929d-dbc8624e3ee5" (UID: "5c057a00-afe2-419d-929d-dbc8624e3ee5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:46:23.828832 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:23.828794    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1d9c868-3873-4067-b133-7aeb46f61f50-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c1d9c868-3873-4067-b133-7aeb46f61f50" (UID: "c1d9c868-3873-4067-b133-7aeb46f61f50"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:46:23.828936 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:23.828912    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d9c868-3873-4067-b133-7aeb46f61f50-kube-api-access-wskb6" (OuterVolumeSpecName: "kube-api-access-wskb6") pod "c1d9c868-3873-4067-b133-7aeb46f61f50" (UID: "c1d9c868-3873-4067-b133-7aeb46f61f50"). InnerVolumeSpecName "kube-api-access-wskb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:46:23.829028 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:23.828981    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c057a00-afe2-419d-929d-dbc8624e3ee5-kube-api-access-887d6" (OuterVolumeSpecName: "kube-api-access-887d6") pod "5c057a00-afe2-419d-929d-dbc8624e3ee5" (UID: "5c057a00-afe2-419d-929d-dbc8624e3ee5"). InnerVolumeSpecName "kube-api-access-887d6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:46:23.835452 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:23.835432    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-13baf70f-64f7ccff68-6ssl6"
Jul 29 19:46:23.926656 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:23.926629    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ced1247a-4b4c-48e1-a061-3e49a7265c19-plugins-config-volume\") pod \"ced1247a-4b4c-48e1-a061-3e49a7265c19\" (UID: \"ced1247a-4b4c-48e1-a061-3e49a7265c19\") "
Jul 29 19:46:23.926795 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:23.926729    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zz76\" (UniqueName: \"kubernetes.io/projected/ced1247a-4b4c-48e1-a061-3e49a7265c19-kube-api-access-5zz76\") pod \"ced1247a-4b4c-48e1-a061-3e49a7265c19\" (UID: \"ced1247a-4b4c-48e1-a061-3e49a7265c19\") "
Jul 29 19:46:23.926940 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:23.926920    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wskb6\" (UniqueName: \"kubernetes.io/projected/c1d9c868-3873-4067-b133-7aeb46f61f50-kube-api-access-wskb6\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:46:23.926995 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:23.926946    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1d9c868-3873-4067-b133-7aeb46f61f50-maas-api-tls\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:46:23.926995 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:23.926975    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-887d6\" (UniqueName: \"kubernetes.io/projected/5c057a00-afe2-419d-929d-dbc8624e3ee5-kube-api-access-887d6\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:46:23.926995 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:23.926989    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1d9c868-3873-4067-b133-7aeb46f61f50-openshift-service-ca\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:46:23.927094 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:23.927003    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c057a00-afe2-419d-929d-dbc8624e3ee5-plugins-config-volume\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:46:23.927094 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:23.927040    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced1247a-4b4c-48e1-a061-3e49a7265c19-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ced1247a-4b4c-48e1-a061-3e49a7265c19" (UID: "ced1247a-4b4c-48e1-a061-3e49a7265c19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:46:23.929020 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:23.928999    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced1247a-4b4c-48e1-a061-3e49a7265c19-kube-api-access-5zz76" (OuterVolumeSpecName: "kube-api-access-5zz76") pod "ced1247a-4b4c-48e1-a061-3e49a7265c19" (UID: "ced1247a-4b4c-48e1-a061-3e49a7265c19"). InnerVolumeSpecName "kube-api-access-5zz76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:46:24.028406 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:24.028323    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ced1247a-4b4c-48e1-a061-3e49a7265c19-plugins-config-volume\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:46:24.028406 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:24.028356    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zz76\" (UniqueName: \"kubernetes.io/projected/ced1247a-4b4c-48e1-a061-3e49a7265c19-kube-api-access-5zz76\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:46:24.513225 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:24.513192    2557 generic.go:356] "Generic (PLEG): container finished" podID="c1d9c868-3873-4067-b133-7aeb46f61f50" containerID="e83a4db30d459507ea798013e9e70123283513cfe38a1e8de13c7c7ee3eef8cd" exitCode=0
Jul 29 19:46:24.513614 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:24.513238    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc"
Jul 29 19:46:24.513614 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:24.513278    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc" event={"ID":"c1d9c868-3873-4067-b133-7aeb46f61f50","Type":"ContainerDied","Data":"e83a4db30d459507ea798013e9e70123283513cfe38a1e8de13c7c7ee3eef8cd"}
Jul 29 19:46:24.513614 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:24.513337    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc" event={"ID":"c1d9c868-3873-4067-b133-7aeb46f61f50","Type":"ContainerDied","Data":"bd65238f7ceda7d8c1b1123afcd230ee1c5d2fefb34a56228b980878d6e89472"}
Jul 29 19:46:24.513614 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:24.513370    2557 scope.go:117] "RemoveContainer" containerID="e83a4db30d459507ea798013e9e70123283513cfe38a1e8de13c7c7ee3eef8cd"
Jul 29 19:46:24.514706 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:24.514689    2557 generic.go:356] "Generic (PLEG): container finished" podID="ced1247a-4b4c-48e1-a061-3e49a7265c19" containerID="9b9fbf294476bfe65c17debbeff9b5ad50c00b5e9a487edbb028450bb38217ad" exitCode=0
Jul 29 19:46:24.514776 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:24.514735    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-13baf70f-64f7ccff68-6ssl6"
Jul 29 19:46:24.514832 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:24.514779    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-13baf70f-64f7ccff68-6ssl6" event={"ID":"ced1247a-4b4c-48e1-a061-3e49a7265c19","Type":"ContainerDied","Data":"9b9fbf294476bfe65c17debbeff9b5ad50c00b5e9a487edbb028450bb38217ad"}
Jul 29 19:46:24.514886 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:24.514834    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-13baf70f-64f7ccff68-6ssl6" event={"ID":"ced1247a-4b4c-48e1-a061-3e49a7265c19","Type":"ContainerDied","Data":"5c1a8ad69e90d60e7d0ff47809ac6068ea74828bba271acf2190e447a9520f4c"}
Jul 29 19:46:24.516227 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:24.516202    2557 generic.go:356] "Generic (PLEG): container finished" podID="5c057a00-afe2-419d-929d-dbc8624e3ee5" containerID="a187ea6cb44aabc878432796712625404df871ce80a0f54b502d5ff78c7bfc4c" exitCode=0
Jul 29 19:46:24.516369 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:24.516251    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-13baf70f-664565f85c-mv8b2"
Jul 29 19:46:24.516369 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:24.516252    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-13baf70f-664565f85c-mv8b2" event={"ID":"5c057a00-afe2-419d-929d-dbc8624e3ee5","Type":"ContainerDied","Data":"a187ea6cb44aabc878432796712625404df871ce80a0f54b502d5ff78c7bfc4c"}
Jul 29 19:46:24.516369 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:24.516350    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-13baf70f-664565f85c-mv8b2" event={"ID":"5c057a00-afe2-419d-929d-dbc8624e3ee5","Type":"ContainerDied","Data":"e875a0ec1e3a7401b3422498506a0968fc2cfd9d8e54bbe9e95246ea9a8c28f2"}
Jul 29 19:46:24.524872 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:24.524855    2557 scope.go:117] "RemoveContainer" containerID="e83a4db30d459507ea798013e9e70123283513cfe38a1e8de13c7c7ee3eef8cd"
Jul 29 19:46:24.525186 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:46:24.525160    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e83a4db30d459507ea798013e9e70123283513cfe38a1e8de13c7c7ee3eef8cd\": container with ID starting with e83a4db30d459507ea798013e9e70123283513cfe38a1e8de13c7c7ee3eef8cd not found: ID does not exist" containerID="e83a4db30d459507ea798013e9e70123283513cfe38a1e8de13c7c7ee3eef8cd"
Jul 29 19:46:24.525279 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:24.525199    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e83a4db30d459507ea798013e9e70123283513cfe38a1e8de13c7c7ee3eef8cd"} err="failed to get container status \"e83a4db30d459507ea798013e9e70123283513cfe38a1e8de13c7c7ee3eef8cd\": rpc error: code = NotFound desc = could not find container \"e83a4db30d459507ea798013e9e70123283513cfe38a1e8de13c7c7ee3eef8cd\": container with ID starting with e83a4db30d459507ea798013e9e70123283513cfe38a1e8de13c7c7ee3eef8cd not found: ID does not exist"
Jul 29 19:46:24.525279 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:24.525223    2557 scope.go:117] "RemoveContainer" containerID="9b9fbf294476bfe65c17debbeff9b5ad50c00b5e9a487edbb028450bb38217ad"
Jul 29 19:46:24.535060 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:24.535039    2557 scope.go:117] "RemoveContainer" containerID="9b9fbf294476bfe65c17debbeff9b5ad50c00b5e9a487edbb028450bb38217ad"
Jul 29 19:46:24.535366 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:46:24.535341    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b9fbf294476bfe65c17debbeff9b5ad50c00b5e9a487edbb028450bb38217ad\": container with ID starting with 9b9fbf294476bfe65c17debbeff9b5ad50c00b5e9a487edbb028450bb38217ad not found: ID does not exist" containerID="9b9fbf294476bfe65c17debbeff9b5ad50c00b5e9a487edbb028450bb38217ad"
Jul 29 19:46:24.535443 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:24.535376    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b9fbf294476bfe65c17debbeff9b5ad50c00b5e9a487edbb028450bb38217ad"} err="failed to get container status \"9b9fbf294476bfe65c17debbeff9b5ad50c00b5e9a487edbb028450bb38217ad\": rpc error: code = NotFound desc = could not find container \"9b9fbf294476bfe65c17debbeff9b5ad50c00b5e9a487edbb028450bb38217ad\": container with ID starting with 9b9fbf294476bfe65c17debbeff9b5ad50c00b5e9a487edbb028450bb38217ad not found: ID does not exist"
Jul 29 19:46:24.535443 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:24.535398    2557 scope.go:117] "RemoveContainer" containerID="a187ea6cb44aabc878432796712625404df871ce80a0f54b502d5ff78c7bfc4c"
Jul 29 19:46:24.545481 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:24.545453    2557 scope.go:117] "RemoveContainer" containerID="a187ea6cb44aabc878432796712625404df871ce80a0f54b502d5ff78c7bfc4c"
Jul 29 19:46:24.545782 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:46:24.545759    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a187ea6cb44aabc878432796712625404df871ce80a0f54b502d5ff78c7bfc4c\": container with ID starting with a187ea6cb44aabc878432796712625404df871ce80a0f54b502d5ff78c7bfc4c not found: ID does not exist" containerID="a187ea6cb44aabc878432796712625404df871ce80a0f54b502d5ff78c7bfc4c"
Jul 29 19:46:24.545851 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:24.545792    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a187ea6cb44aabc878432796712625404df871ce80a0f54b502d5ff78c7bfc4c"} err="failed to get container status \"a187ea6cb44aabc878432796712625404df871ce80a0f54b502d5ff78c7bfc4c\": rpc error: code = NotFound desc = could not find container \"a187ea6cb44aabc878432796712625404df871ce80a0f54b502d5ff78c7bfc4c\": container with ID starting with a187ea6cb44aabc878432796712625404df871ce80a0f54b502d5ff78c7bfc4c not found: ID does not exist"
Jul 29 19:46:24.556762 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:24.556740    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-13baf70f-664565f85c-mv8b2"]
Jul 29 19:46:24.568951 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:24.568929    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-13baf70f-664565f85c-mv8b2"]
Jul 29 19:46:24.589332 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:24.589305    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-13baf70f-64f7ccff68-6ssl6"]
Jul 29 19:46:24.599741 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:24.599713    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-13baf70f-64f7ccff68-6ssl6"]
Jul 29 19:46:24.616914 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:24.616891    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc"]
Jul 29 19:46:24.622434 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:24.622412    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-13baf70f-687649bf4f-69qgc"]
Jul 29 19:46:25.678716 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:25.678686    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c057a00-afe2-419d-929d-dbc8624e3ee5" path="/var/lib/kubelet/pods/5c057a00-afe2-419d-929d-dbc8624e3ee5/volumes"
Jul 29 19:46:25.679078 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:25.679012    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87a7fe31-2c46-4a66-b4ed-c8b86abc5311" path="/var/lib/kubelet/pods/87a7fe31-2c46-4a66-b4ed-c8b86abc5311/volumes"
Jul 29 19:46:25.679373 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:25.679358    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1d9c868-3873-4067-b133-7aeb46f61f50" path="/var/lib/kubelet/pods/c1d9c868-3873-4067-b133-7aeb46f61f50/volumes"
Jul 29 19:46:25.679662 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:25.679650    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ced1247a-4b4c-48e1-a061-3e49a7265c19" path="/var/lib/kubelet/pods/ced1247a-4b4c-48e1-a061-3e49a7265c19/volumes"
Jul 29 19:46:28.170151 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:28.170114    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5"]
Jul 29 19:46:28.170609 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:28.170590    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c057a00-afe2-419d-929d-dbc8624e3ee5" containerName="payload-processing"
Jul 29 19:46:28.170649 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:28.170611    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c057a00-afe2-419d-929d-dbc8624e3ee5" containerName="payload-processing"
Jul 29 19:46:28.170649 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:28.170636    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ced1247a-4b4c-48e1-a061-3e49a7265c19" containerName="payload-pre-processing"
Jul 29 19:46:28.170649 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:28.170644    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ced1247a-4b4c-48e1-a061-3e49a7265c19" containerName="payload-pre-processing"
Jul 29 19:46:28.170747 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:28.170658    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1d9c868-3873-4067-b133-7aeb46f61f50" containerName="maas-api"
Jul 29 19:46:28.170747 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:28.170668    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1d9c868-3873-4067-b133-7aeb46f61f50" containerName="maas-api"
Jul 29 19:46:28.170747 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:28.170682    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87a7fe31-2c46-4a66-b4ed-c8b86abc5311" containerName="revoke-keys"
Jul 29 19:46:28.170747 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:28.170690    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="87a7fe31-2c46-4a66-b4ed-c8b86abc5311" containerName="revoke-keys"
Jul 29 19:46:28.170859 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:28.170779    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="87a7fe31-2c46-4a66-b4ed-c8b86abc5311" containerName="revoke-keys"
Jul 29 19:46:28.170859 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:28.170793    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ced1247a-4b4c-48e1-a061-3e49a7265c19" containerName="payload-pre-processing"
Jul 29 19:46:28.170859 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:28.170803    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1d9c868-3873-4067-b133-7aeb46f61f50" containerName="maas-api"
Jul 29 19:46:28.170859 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:28.170814    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c057a00-afe2-419d-929d-dbc8624e3ee5" containerName="payload-processing"
Jul 29 19:46:28.176519 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:28.176499    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5"
Jul 29 19:46:28.179263 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:28.179240    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a4a71d55-serving-cert\""
Jul 29 19:46:28.179371 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:28.179322    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4dw6s\""
Jul 29 19:46:28.184206 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:28.183790    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5"]
Jul 29 19:46:28.268020 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:28.267987    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dcef7b6-4351-4460-9cf1-54b699021513-openshift-service-ca\") pod \"maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5\" (UID: \"5dcef7b6-4351-4460-9cf1-54b699021513\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5"
Jul 29 19:46:28.268179 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:28.268071    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d48jv\" (UniqueName: \"kubernetes.io/projected/5dcef7b6-4351-4460-9cf1-54b699021513-kube-api-access-d48jv\") pod \"maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5\" (UID: \"5dcef7b6-4351-4460-9cf1-54b699021513\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5"
Jul 29 19:46:28.268179 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:28.268116    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dcef7b6-4351-4460-9cf1-54b699021513-maas-api-tls\") pod \"maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5\" (UID: \"5dcef7b6-4351-4460-9cf1-54b699021513\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5"
Jul 29 19:46:28.369350 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:28.369313    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d48jv\" (UniqueName: \"kubernetes.io/projected/5dcef7b6-4351-4460-9cf1-54b699021513-kube-api-access-d48jv\") pod \"maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5\" (UID: \"5dcef7b6-4351-4460-9cf1-54b699021513\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5"
Jul 29 19:46:28.369526 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:28.369364    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dcef7b6-4351-4460-9cf1-54b699021513-maas-api-tls\") pod \"maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5\" (UID: \"5dcef7b6-4351-4460-9cf1-54b699021513\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5"
Jul 29 19:46:28.369526 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:28.369409    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dcef7b6-4351-4460-9cf1-54b699021513-openshift-service-ca\") pod \"maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5\" (UID: \"5dcef7b6-4351-4460-9cf1-54b699021513\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5"
Jul 29 19:46:28.369656 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:46:28.369558    2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-a4a71d55-serving-cert: secret "maas-api-e2e-ait-a4a71d55-serving-cert" not found
Jul 29 19:46:28.369716 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:46:28.369666    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dcef7b6-4351-4460-9cf1-54b699021513-maas-api-tls podName:5dcef7b6-4351-4460-9cf1-54b699021513 nodeName:}" failed. No retries permitted until 2026-07-29 19:46:28.86963058 +0000 UTC m=+1759.697129625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5dcef7b6-4351-4460-9cf1-54b699021513-maas-api-tls") pod "maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5" (UID: "5dcef7b6-4351-4460-9cf1-54b699021513") : secret "maas-api-e2e-ait-a4a71d55-serving-cert" not found
Jul 29 19:46:28.370024 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:28.370006    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dcef7b6-4351-4460-9cf1-54b699021513-openshift-service-ca\") pod \"maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5\" (UID: \"5dcef7b6-4351-4460-9cf1-54b699021513\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5"
Jul 29 19:46:28.380715 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:28.380691    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d48jv\" (UniqueName: \"kubernetes.io/projected/5dcef7b6-4351-4460-9cf1-54b699021513-kube-api-access-d48jv\") pod \"maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5\" (UID: \"5dcef7b6-4351-4460-9cf1-54b699021513\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5"
Jul 29 19:46:28.873394 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:28.873354    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dcef7b6-4351-4460-9cf1-54b699021513-maas-api-tls\") pod \"maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5\" (UID: \"5dcef7b6-4351-4460-9cf1-54b699021513\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5"
Jul 29 19:46:28.875741 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:28.875711    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dcef7b6-4351-4460-9cf1-54b699021513-maas-api-tls\") pod \"maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5\" (UID: \"5dcef7b6-4351-4460-9cf1-54b699021513\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5"
Jul 29 19:46:28.996810 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:28.996776    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a4a71d55-794d745b88-jf5tt"]
Jul 29 19:46:28.999235 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:28.999216    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a4a71d55-794d745b88-jf5tt"
Jul 29 19:46:29.002330 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:29.002305    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a4a71d55-dockercfg-cfpt7\""
Jul 29 19:46:29.002448 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:29.002336    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a4a71d55\""
Jul 29 19:46:29.015410 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:29.015387    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a4a71d55-794d745b88-jf5tt"]
Jul 29 19:46:29.051327 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:29.051298    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a4a71d55-56789b7844-n8zxl"]
Jul 29 19:46:29.053895 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:29.053877    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4a71d55-56789b7844-n8zxl"
Jul 29 19:46:29.064499 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:29.064474    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a4a71d55-56789b7844-n8zxl"]
Jul 29 19:46:29.074898 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:29.074870    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-599dp\" (UniqueName: \"kubernetes.io/projected/6dac2f7e-58e6-4494-944b-8fdc5ac94a82-kube-api-access-599dp\") pod \"payload-processing-e2e-ait-a4a71d55-794d745b88-jf5tt\" (UID: \"6dac2f7e-58e6-4494-944b-8fdc5ac94a82\") " pod="openshift-ingress/payload-processing-e2e-ait-a4a71d55-794d745b88-jf5tt"
Jul 29 19:46:29.075045 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:29.074924    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dac2f7e-58e6-4494-944b-8fdc5ac94a82-plugins-config-volume\") pod \"payload-processing-e2e-ait-a4a71d55-794d745b88-jf5tt\" (UID: \"6dac2f7e-58e6-4494-944b-8fdc5ac94a82\") " pod="openshift-ingress/payload-processing-e2e-ait-a4a71d55-794d745b88-jf5tt"
Jul 29 19:46:29.087675 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:29.087653    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5"
Jul 29 19:46:29.175543 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:29.175511    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-599dp\" (UniqueName: \"kubernetes.io/projected/6dac2f7e-58e6-4494-944b-8fdc5ac94a82-kube-api-access-599dp\") pod \"payload-processing-e2e-ait-a4a71d55-794d745b88-jf5tt\" (UID: \"6dac2f7e-58e6-4494-944b-8fdc5ac94a82\") " pod="openshift-ingress/payload-processing-e2e-ait-a4a71d55-794d745b88-jf5tt"
Jul 29 19:46:29.176079 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:29.176050    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc68x\" (UniqueName: \"kubernetes.io/projected/82c7f59c-6550-417f-b9e6-72635ff8d686-kube-api-access-bc68x\") pod \"payload-pre-processing-e2e-ait-a4a71d55-56789b7844-n8zxl\" (UID: \"82c7f59c-6550-417f-b9e6-72635ff8d686\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a4a71d55-56789b7844-n8zxl"
Jul 29 19:46:29.176242 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:29.176219    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dac2f7e-58e6-4494-944b-8fdc5ac94a82-plugins-config-volume\") pod \"payload-processing-e2e-ait-a4a71d55-794d745b88-jf5tt\" (UID: \"6dac2f7e-58e6-4494-944b-8fdc5ac94a82\") " pod="openshift-ingress/payload-processing-e2e-ait-a4a71d55-794d745b88-jf5tt"
Jul 29 19:46:29.176320 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:29.176304    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82c7f59c-6550-417f-b9e6-72635ff8d686-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a4a71d55-56789b7844-n8zxl\" (UID: \"82c7f59c-6550-417f-b9e6-72635ff8d686\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a4a71d55-56789b7844-n8zxl"
Jul 29 19:46:29.177081 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:29.177060    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dac2f7e-58e6-4494-944b-8fdc5ac94a82-plugins-config-volume\") pod \"payload-processing-e2e-ait-a4a71d55-794d745b88-jf5tt\" (UID: \"6dac2f7e-58e6-4494-944b-8fdc5ac94a82\") " pod="openshift-ingress/payload-processing-e2e-ait-a4a71d55-794d745b88-jf5tt"
Jul 29 19:46:29.185155 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:29.185107    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-599dp\" (UniqueName: \"kubernetes.io/projected/6dac2f7e-58e6-4494-944b-8fdc5ac94a82-kube-api-access-599dp\") pod \"payload-processing-e2e-ait-a4a71d55-794d745b88-jf5tt\" (UID: \"6dac2f7e-58e6-4494-944b-8fdc5ac94a82\") " pod="openshift-ingress/payload-processing-e2e-ait-a4a71d55-794d745b88-jf5tt"
Jul 29 19:46:29.225389 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:29.225329    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5"]
Jul 29 19:46:29.227955 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:46:29.227928    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dcef7b6_4351_4460_9cf1_54b699021513.slice/crio-17320ed3eb6586675d7d6e6900b32bf577341dbbe682f31d246bd8c96b2f6733 WatchSource:0}: Error finding container 17320ed3eb6586675d7d6e6900b32bf577341dbbe682f31d246bd8c96b2f6733: Status 404 returned error can't find the container with id 17320ed3eb6586675d7d6e6900b32bf577341dbbe682f31d246bd8c96b2f6733
Jul 29 19:46:29.277008 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:29.276984    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82c7f59c-6550-417f-b9e6-72635ff8d686-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a4a71d55-56789b7844-n8zxl\" (UID: \"82c7f59c-6550-417f-b9e6-72635ff8d686\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a4a71d55-56789b7844-n8zxl"
Jul 29 19:46:29.277142 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:29.277044    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc68x\" (UniqueName: \"kubernetes.io/projected/82c7f59c-6550-417f-b9e6-72635ff8d686-kube-api-access-bc68x\") pod \"payload-pre-processing-e2e-ait-a4a71d55-56789b7844-n8zxl\" (UID: \"82c7f59c-6550-417f-b9e6-72635ff8d686\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a4a71d55-56789b7844-n8zxl"
Jul 29 19:46:29.277593 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:29.277573    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82c7f59c-6550-417f-b9e6-72635ff8d686-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a4a71d55-56789b7844-n8zxl\" (UID: \"82c7f59c-6550-417f-b9e6-72635ff8d686\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a4a71d55-56789b7844-n8zxl"
Jul 29 19:46:29.285124 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:29.285103    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc68x\" (UniqueName: \"kubernetes.io/projected/82c7f59c-6550-417f-b9e6-72635ff8d686-kube-api-access-bc68x\") pod \"payload-pre-processing-e2e-ait-a4a71d55-56789b7844-n8zxl\" (UID: \"82c7f59c-6550-417f-b9e6-72635ff8d686\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a4a71d55-56789b7844-n8zxl"
Jul 29 19:46:29.309489 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:29.309470    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a4a71d55-794d745b88-jf5tt"
Jul 29 19:46:29.367451 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:29.367417    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4a71d55-56789b7844-n8zxl"
Jul 29 19:46:29.449883 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:29.449860    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a4a71d55-794d745b88-jf5tt"]
Jul 29 19:46:29.451596 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:46:29.451569    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dac2f7e_58e6_4494_944b_8fdc5ac94a82.slice/crio-cb77842efd2ee19b5a3f6d2aeb3572cbad0b4833d32d3bd16a22c3b42ae0869f WatchSource:0}: Error finding container cb77842efd2ee19b5a3f6d2aeb3572cbad0b4833d32d3bd16a22c3b42ae0869f: Status 404 returned error can't find the container with id cb77842efd2ee19b5a3f6d2aeb3572cbad0b4833d32d3bd16a22c3b42ae0869f
Jul 29 19:46:29.511289 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:29.511265    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a4a71d55-56789b7844-n8zxl"]
Jul 29 19:46:29.512810 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:46:29.512785    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82c7f59c_6550_417f_b9e6_72635ff8d686.slice/crio-bb6fbfd1e09c0948885df8a74949569bcc8e478485795be35084448cbf26d6bd WatchSource:0}: Error finding container bb6fbfd1e09c0948885df8a74949569bcc8e478485795be35084448cbf26d6bd: Status 404 returned error can't find the container with id bb6fbfd1e09c0948885df8a74949569bcc8e478485795be35084448cbf26d6bd
Jul 29 19:46:29.542311 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:29.542285    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4a71d55-56789b7844-n8zxl" event={"ID":"82c7f59c-6550-417f-b9e6-72635ff8d686","Type":"ContainerStarted","Data":"bb6fbfd1e09c0948885df8a74949569bcc8e478485795be35084448cbf26d6bd"}
Jul 29 19:46:29.543608 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:29.543582    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5" event={"ID":"5dcef7b6-4351-4460-9cf1-54b699021513","Type":"ContainerStarted","Data":"17320ed3eb6586675d7d6e6900b32bf577341dbbe682f31d246bd8c96b2f6733"}
Jul 29 19:46:29.545139 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:29.545114    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a4a71d55-794d745b88-jf5tt" event={"ID":"6dac2f7e-58e6-4494-944b-8fdc5ac94a82","Type":"ContainerStarted","Data":"969afa219c60b60f2ab8dcd2a3ce074b86bcbcad5381ef710d0a379965980842"}
Jul 29 19:46:29.545329 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:29.545163    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a4a71d55-794d745b88-jf5tt" event={"ID":"6dac2f7e-58e6-4494-944b-8fdc5ac94a82","Type":"ContainerStarted","Data":"cb77842efd2ee19b5a3f6d2aeb3572cbad0b4833d32d3bd16a22c3b42ae0869f"}
Jul 29 19:46:29.545329 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:29.545294    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a4a71d55-794d745b88-jf5tt"
Jul 29 19:46:29.562249 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:29.562206    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a4a71d55-794d745b88-jf5tt" podStartSLOduration=1.562191664 podStartE2EDuration="1.562191664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:46:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:46:29.56149955 +0000 UTC m=+1760.388998611" watchObservedRunningTime="2026-07-29 19:46:29.562191664 +0000 UTC m=+1760.389690722"
Jul 29 19:46:30.558899 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:30.558857    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4a71d55-56789b7844-n8zxl" event={"ID":"82c7f59c-6550-417f-b9e6-72635ff8d686","Type":"ContainerStarted","Data":"90e5beaefbe6ac90c1f349230a40d2a07669b71b38b17f74b308f9d0de7d5eba"}
Jul 29 19:46:30.559401 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:30.558957    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4a71d55-56789b7844-n8zxl"
Jul 29 19:46:30.560694 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:30.560665    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5" event={"ID":"5dcef7b6-4351-4460-9cf1-54b699021513","Type":"ContainerStarted","Data":"fcf50ef3df7203004583e07489a43f07a63351f4bc76f88d7f98b4a2987100ff"}
Jul 29 19:46:30.560823 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:30.560801    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5"
Jul 29 19:46:30.577693 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:30.577643    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4a71d55-56789b7844-n8zxl" podStartSLOduration=1.577629215 podStartE2EDuration="1.577629215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:46:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:46:30.575873283 +0000 UTC m=+1761.403372342" watchObservedRunningTime="2026-07-29 19:46:30.577629215 +0000 UTC m=+1761.405128287"
Jul 29 19:46:30.593451 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:30.593407    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5" podStartSLOduration=2.227087801 podStartE2EDuration="2.593394445s" totalImagesPullingTime="366.306644ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:46:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:46:30.592317818 +0000 UTC m=+1761.419816879" watchObservedRunningTime="2026-07-29 19:46:30.593394445 +0000 UTC m=+1761.420893503"
Jul 29 19:46:36.571345 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:36.571316    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5"
Jul 29 19:46:40.563320 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:40.563290    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a4a71d55-794d745b88-jf5tt"
Jul 29 19:46:41.567844 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:41.567795    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4a71d55-56789b7844-n8zxl" podUID="82c7f59c-6550-417f-b9e6-72635ff8d686" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.45:9004: connect: connection refused"
Jul 29 19:46:51.566791 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:51.566757    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4a71d55-56789b7844-n8zxl"
Jul 29 19:46:57.935752 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:57.935719    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4a71d55-kw22c"]
Jul 29 19:46:57.939146 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:57.939128    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4a71d55-kw22c"
Jul 29 19:46:57.941972 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:57.941944    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jsq2r\""
Jul 29 19:46:57.950603 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:57.950578    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4a71d55-kw22c"]
Jul 29 19:46:58.135422 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:58.135385    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7758f49c-19ec-40f6-a7f3-ca9b01a98fd9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a4a71d55-kw22c\" (UID: \"7758f49c-19ec-40f6-a7f3-ca9b01a98fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4a71d55-kw22c"
Jul 29 19:46:58.236345 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:58.236264    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7758f49c-19ec-40f6-a7f3-ca9b01a98fd9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a4a71d55-kw22c\" (UID: \"7758f49c-19ec-40f6-a7f3-ca9b01a98fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4a71d55-kw22c"
Jul 29 19:46:58.236827 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:58.236807    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7758f49c-19ec-40f6-a7f3-ca9b01a98fd9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a4a71d55-kw22c\" (UID: \"7758f49c-19ec-40f6-a7f3-ca9b01a98fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4a71d55-kw22c"
Jul 29 19:46:58.250215 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:58.250190    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4a71d55-kw22c"
Jul 29 19:46:58.382780 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:58.382755    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4a71d55-kw22c"]
Jul 29 19:46:58.384749 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:46:58.384722    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7758f49c_19ec_40f6_a7f3_ca9b01a98fd9.slice/crio-500629cf352a1c8812ef63c71c900ebf536218e92666c8910f5fa42dda1450f6 WatchSource:0}: Error finding container 500629cf352a1c8812ef63c71c900ebf536218e92666c8910f5fa42dda1450f6: Status 404 returned error can't find the container with id 500629cf352a1c8812ef63c71c900ebf536218e92666c8910f5fa42dda1450f6
Jul 29 19:46:58.691142 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:58.691101    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4a71d55-kw22c" event={"ID":"7758f49c-19ec-40f6-a7f3-ca9b01a98fd9","Type":"ContainerStarted","Data":"fa1e0ec5bcacf35325b5d6ad97ed6f391a05e8d68c241869e509e2f8ba8e5e41"}
Jul 29 19:46:58.691327 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:58.691161    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4a71d55-kw22c" event={"ID":"7758f49c-19ec-40f6-a7f3-ca9b01a98fd9","Type":"ContainerStarted","Data":"500629cf352a1c8812ef63c71c900ebf536218e92666c8910f5fa42dda1450f6"}
Jul 29 19:46:58.709554 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:58.709512    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4a71d55-kw22c" podStartSLOduration=1.709496852 podStartE2EDuration="1.709496852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:46:58.70831384 +0000 UTC m=+1789.535812899" watchObservedRunningTime="2026-07-29 19:46:58.709496852 +0000 UTC m=+1789.536995909"
Jul 29 19:46:59.696624 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:59.696586    2557 generic.go:356] "Generic (PLEG): container finished" podID="7758f49c-19ec-40f6-a7f3-ca9b01a98fd9" containerID="fa1e0ec5bcacf35325b5d6ad97ed6f391a05e8d68c241869e509e2f8ba8e5e41" exitCode=0
Jul 29 19:46:59.697024 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:46:59.696671    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4a71d55-kw22c" event={"ID":"7758f49c-19ec-40f6-a7f3-ca9b01a98fd9","Type":"ContainerDied","Data":"fa1e0ec5bcacf35325b5d6ad97ed6f391a05e8d68c241869e509e2f8ba8e5e41"}
Jul 29 19:47:00.832927 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:00.832897    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4a71d55-kw22c"
Jul 29 19:47:00.859232 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:00.859205    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7758f49c-19ec-40f6-a7f3-ca9b01a98fd9-maas-api-service-ca\") pod \"7758f49c-19ec-40f6-a7f3-ca9b01a98fd9\" (UID: \"7758f49c-19ec-40f6-a7f3-ca9b01a98fd9\") "
Jul 29 19:47:00.859548 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:00.859521    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7758f49c-19ec-40f6-a7f3-ca9b01a98fd9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7758f49c-19ec-40f6-a7f3-ca9b01a98fd9" (UID: "7758f49c-19ec-40f6-a7f3-ca9b01a98fd9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:47:00.960575 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:00.960478    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7758f49c-19ec-40f6-a7f3-ca9b01a98fd9-maas-api-service-ca\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:47:01.707641 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:01.707611    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4a71d55-kw22c"
Jul 29 19:47:01.707641 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:01.707628    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4a71d55-kw22c" event={"ID":"7758f49c-19ec-40f6-a7f3-ca9b01a98fd9","Type":"ContainerDied","Data":"500629cf352a1c8812ef63c71c900ebf536218e92666c8910f5fa42dda1450f6"}
Jul 29 19:47:01.707814 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:01.707676    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="500629cf352a1c8812ef63c71c900ebf536218e92666c8910f5fa42dda1450f6"
Jul 29 19:47:01.712831 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:47:01.712804    2557 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7758f49c_19ec_40f6_a7f3_ca9b01a98fd9.slice\": RecentStats: unable to find data in memory cache]"
Jul 29 19:47:09.000278 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.000244    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5"]
Jul 29 19:47:09.000704 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.000499    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5" podUID="5dcef7b6-4351-4460-9cf1-54b699021513" containerName="maas-api" containerID="cri-o://fcf50ef3df7203004583e07489a43f07a63351f4bc76f88d7f98b4a2987100ff" gracePeriod=30
Jul 29 19:47:09.060497 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.060464    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a4a71d55-794d745b88-jf5tt"]
Jul 29 19:47:09.060745 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.060689    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a4a71d55-794d745b88-jf5tt" podUID="6dac2f7e-58e6-4494-944b-8fdc5ac94a82" containerName="payload-processing" containerID="cri-o://969afa219c60b60f2ab8dcd2a3ce074b86bcbcad5381ef710d0a379965980842" gracePeriod=30
Jul 29 19:47:09.065815 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.065787    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a4a71d55-56789b7844-n8zxl"]
Jul 29 19:47:09.066025 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.065981    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4a71d55-56789b7844-n8zxl" podUID="82c7f59c-6550-417f-b9e6-72635ff8d686" containerName="payload-pre-processing" containerID="cri-o://90e5beaefbe6ac90c1f349230a40d2a07669b71b38b17f74b308f9d0de7d5eba" gracePeriod=30
Jul 29 19:47:09.259259 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.259208    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4a71d55-kw22c"]
Jul 29 19:47:09.261921 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.261896    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4a71d55-kw22c"]
Jul 29 19:47:09.336232 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.335396    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a4a71d55-794d745b88-jf5tt"
Jul 29 19:47:09.359152 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.359123    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5"
Jul 29 19:47:09.362853 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.362830    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4a71d55-56789b7844-n8zxl"
Jul 29 19:47:09.447063 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.447030    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dcef7b6-4351-4460-9cf1-54b699021513-maas-api-tls\") pod \"5dcef7b6-4351-4460-9cf1-54b699021513\" (UID: \"5dcef7b6-4351-4460-9cf1-54b699021513\") "
Jul 29 19:47:09.447253 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.447079    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dcef7b6-4351-4460-9cf1-54b699021513-openshift-service-ca\") pod \"5dcef7b6-4351-4460-9cf1-54b699021513\" (UID: \"5dcef7b6-4351-4460-9cf1-54b699021513\") "
Jul 29 19:47:09.447253 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.447100    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-599dp\" (UniqueName: \"kubernetes.io/projected/6dac2f7e-58e6-4494-944b-8fdc5ac94a82-kube-api-access-599dp\") pod \"6dac2f7e-58e6-4494-944b-8fdc5ac94a82\" (UID: \"6dac2f7e-58e6-4494-944b-8fdc5ac94a82\") "
Jul 29 19:47:09.447253 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.447149    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82c7f59c-6550-417f-b9e6-72635ff8d686-plugins-config-volume\") pod \"82c7f59c-6550-417f-b9e6-72635ff8d686\" (UID: \"82c7f59c-6550-417f-b9e6-72635ff8d686\") "
Jul 29 19:47:09.447253 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.447173    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d48jv\" (UniqueName: \"kubernetes.io/projected/5dcef7b6-4351-4460-9cf1-54b699021513-kube-api-access-d48jv\") pod \"5dcef7b6-4351-4460-9cf1-54b699021513\" (UID: \"5dcef7b6-4351-4460-9cf1-54b699021513\") "
Jul 29 19:47:09.447253 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.447207    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bc68x\" (UniqueName: \"kubernetes.io/projected/82c7f59c-6550-417f-b9e6-72635ff8d686-kube-api-access-bc68x\") pod \"82c7f59c-6550-417f-b9e6-72635ff8d686\" (UID: \"82c7f59c-6550-417f-b9e6-72635ff8d686\") "
Jul 29 19:47:09.447505 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.447298    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dac2f7e-58e6-4494-944b-8fdc5ac94a82-plugins-config-volume\") pod \"6dac2f7e-58e6-4494-944b-8fdc5ac94a82\" (UID: \"6dac2f7e-58e6-4494-944b-8fdc5ac94a82\") "
Jul 29 19:47:09.447505 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.447466    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dcef7b6-4351-4460-9cf1-54b699021513-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5dcef7b6-4351-4460-9cf1-54b699021513" (UID: "5dcef7b6-4351-4460-9cf1-54b699021513"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:47:09.447603 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.447536    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82c7f59c-6550-417f-b9e6-72635ff8d686-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "82c7f59c-6550-417f-b9e6-72635ff8d686" (UID: "82c7f59c-6550-417f-b9e6-72635ff8d686"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:47:09.447807 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.447780    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dac2f7e-58e6-4494-944b-8fdc5ac94a82-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6dac2f7e-58e6-4494-944b-8fdc5ac94a82" (UID: "6dac2f7e-58e6-4494-944b-8fdc5ac94a82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:47:09.447865 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.447799    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dcef7b6-4351-4460-9cf1-54b699021513-openshift-service-ca\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:47:09.447865 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.447820    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82c7f59c-6550-417f-b9e6-72635ff8d686-plugins-config-volume\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:47:09.449474 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.449448    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dcef7b6-4351-4460-9cf1-54b699021513-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5dcef7b6-4351-4460-9cf1-54b699021513" (UID: "5dcef7b6-4351-4460-9cf1-54b699021513"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:47:09.449609 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.449474    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82c7f59c-6550-417f-b9e6-72635ff8d686-kube-api-access-bc68x" (OuterVolumeSpecName: "kube-api-access-bc68x") pod "82c7f59c-6550-417f-b9e6-72635ff8d686" (UID: "82c7f59c-6550-417f-b9e6-72635ff8d686"). InnerVolumeSpecName "kube-api-access-bc68x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:47:09.449609 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.449515    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dac2f7e-58e6-4494-944b-8fdc5ac94a82-kube-api-access-599dp" (OuterVolumeSpecName: "kube-api-access-599dp") pod "6dac2f7e-58e6-4494-944b-8fdc5ac94a82" (UID: "6dac2f7e-58e6-4494-944b-8fdc5ac94a82"). InnerVolumeSpecName "kube-api-access-599dp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:47:09.449773 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.449758    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dcef7b6-4351-4460-9cf1-54b699021513-kube-api-access-d48jv" (OuterVolumeSpecName: "kube-api-access-d48jv") pod "5dcef7b6-4351-4460-9cf1-54b699021513" (UID: "5dcef7b6-4351-4460-9cf1-54b699021513"). InnerVolumeSpecName "kube-api-access-d48jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:47:09.549041 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.548972    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-599dp\" (UniqueName: \"kubernetes.io/projected/6dac2f7e-58e6-4494-944b-8fdc5ac94a82-kube-api-access-599dp\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:47:09.549041 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.549001    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d48jv\" (UniqueName: \"kubernetes.io/projected/5dcef7b6-4351-4460-9cf1-54b699021513-kube-api-access-d48jv\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:47:09.549041 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.549011    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc68x\" (UniqueName: \"kubernetes.io/projected/82c7f59c-6550-417f-b9e6-72635ff8d686-kube-api-access-bc68x\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:47:09.549041 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.549023    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dac2f7e-58e6-4494-944b-8fdc5ac94a82-plugins-config-volume\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:47:09.549041 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.549037    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dcef7b6-4351-4460-9cf1-54b699021513-maas-api-tls\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:47:09.680585 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.680553    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7758f49c-19ec-40f6-a7f3-ca9b01a98fd9" path="/var/lib/kubelet/pods/7758f49c-19ec-40f6-a7f3-ca9b01a98fd9/volumes"
Jul 29 19:47:09.747542 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.747507    2557 generic.go:356] "Generic (PLEG): container finished" podID="82c7f59c-6550-417f-b9e6-72635ff8d686" containerID="90e5beaefbe6ac90c1f349230a40d2a07669b71b38b17f74b308f9d0de7d5eba" exitCode=0
Jul 29 19:47:09.747665 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.747559    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4a71d55-56789b7844-n8zxl"
Jul 29 19:47:09.747665 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.747594    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4a71d55-56789b7844-n8zxl" event={"ID":"82c7f59c-6550-417f-b9e6-72635ff8d686","Type":"ContainerDied","Data":"90e5beaefbe6ac90c1f349230a40d2a07669b71b38b17f74b308f9d0de7d5eba"}
Jul 29 19:47:09.747665 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.747647    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4a71d55-56789b7844-n8zxl" event={"ID":"82c7f59c-6550-417f-b9e6-72635ff8d686","Type":"ContainerDied","Data":"bb6fbfd1e09c0948885df8a74949569bcc8e478485795be35084448cbf26d6bd"}
Jul 29 19:47:09.747799 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.747679    2557 scope.go:117] "RemoveContainer" containerID="90e5beaefbe6ac90c1f349230a40d2a07669b71b38b17f74b308f9d0de7d5eba"
Jul 29 19:47:09.749001 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.748957    2557 generic.go:356] "Generic (PLEG): container finished" podID="5dcef7b6-4351-4460-9cf1-54b699021513" containerID="fcf50ef3df7203004583e07489a43f07a63351f4bc76f88d7f98b4a2987100ff" exitCode=0
Jul 29 19:47:09.749103 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.749027    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5"
Jul 29 19:47:09.749103 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.749060    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5" event={"ID":"5dcef7b6-4351-4460-9cf1-54b699021513","Type":"ContainerDied","Data":"fcf50ef3df7203004583e07489a43f07a63351f4bc76f88d7f98b4a2987100ff"}
Jul 29 19:47:09.749294 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.749130    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5" event={"ID":"5dcef7b6-4351-4460-9cf1-54b699021513","Type":"ContainerDied","Data":"17320ed3eb6586675d7d6e6900b32bf577341dbbe682f31d246bd8c96b2f6733"}
Jul 29 19:47:09.751267 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.751040    2557 generic.go:356] "Generic (PLEG): container finished" podID="6dac2f7e-58e6-4494-944b-8fdc5ac94a82" containerID="969afa219c60b60f2ab8dcd2a3ce074b86bcbcad5381ef710d0a379965980842" exitCode=0
Jul 29 19:47:09.751267 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.751099    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a4a71d55-794d745b88-jf5tt" event={"ID":"6dac2f7e-58e6-4494-944b-8fdc5ac94a82","Type":"ContainerDied","Data":"969afa219c60b60f2ab8dcd2a3ce074b86bcbcad5381ef710d0a379965980842"}
Jul 29 19:47:09.751267 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.751118    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a4a71d55-794d745b88-jf5tt"
Jul 29 19:47:09.751267 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.751139    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a4a71d55-794d745b88-jf5tt" event={"ID":"6dac2f7e-58e6-4494-944b-8fdc5ac94a82","Type":"ContainerDied","Data":"cb77842efd2ee19b5a3f6d2aeb3572cbad0b4833d32d3bd16a22c3b42ae0869f"}
Jul 29 19:47:09.757084 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.757067    2557 scope.go:117] "RemoveContainer" containerID="90e5beaefbe6ac90c1f349230a40d2a07669b71b38b17f74b308f9d0de7d5eba"
Jul 29 19:47:09.757329 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:47:09.757300    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90e5beaefbe6ac90c1f349230a40d2a07669b71b38b17f74b308f9d0de7d5eba\": container with ID starting with 90e5beaefbe6ac90c1f349230a40d2a07669b71b38b17f74b308f9d0de7d5eba not found: ID does not exist" containerID="90e5beaefbe6ac90c1f349230a40d2a07669b71b38b17f74b308f9d0de7d5eba"
Jul 29 19:47:09.757379 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.757329    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90e5beaefbe6ac90c1f349230a40d2a07669b71b38b17f74b308f9d0de7d5eba"} err="failed to get container status \"90e5beaefbe6ac90c1f349230a40d2a07669b71b38b17f74b308f9d0de7d5eba\": rpc error: code = NotFound desc = could not find container \"90e5beaefbe6ac90c1f349230a40d2a07669b71b38b17f74b308f9d0de7d5eba\": container with ID starting with 90e5beaefbe6ac90c1f349230a40d2a07669b71b38b17f74b308f9d0de7d5eba not found: ID does not exist"
Jul 29 19:47:09.757379 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.757346    2557 scope.go:117] "RemoveContainer" containerID="fcf50ef3df7203004583e07489a43f07a63351f4bc76f88d7f98b4a2987100ff"
Jul 29 19:47:09.764955 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.764930    2557 scope.go:117] "RemoveContainer" containerID="fcf50ef3df7203004583e07489a43f07a63351f4bc76f88d7f98b4a2987100ff"
Jul 29 19:47:09.765383 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:47:09.765363    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcf50ef3df7203004583e07489a43f07a63351f4bc76f88d7f98b4a2987100ff\": container with ID starting with fcf50ef3df7203004583e07489a43f07a63351f4bc76f88d7f98b4a2987100ff not found: ID does not exist" containerID="fcf50ef3df7203004583e07489a43f07a63351f4bc76f88d7f98b4a2987100ff"
Jul 29 19:47:09.765468 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.765393    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcf50ef3df7203004583e07489a43f07a63351f4bc76f88d7f98b4a2987100ff"} err="failed to get container status \"fcf50ef3df7203004583e07489a43f07a63351f4bc76f88d7f98b4a2987100ff\": rpc error: code = NotFound desc = could not find container \"fcf50ef3df7203004583e07489a43f07a63351f4bc76f88d7f98b4a2987100ff\": container with ID starting with fcf50ef3df7203004583e07489a43f07a63351f4bc76f88d7f98b4a2987100ff not found: ID does not exist"
Jul 29 19:47:09.765468 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.765415    2557 scope.go:117] "RemoveContainer" containerID="969afa219c60b60f2ab8dcd2a3ce074b86bcbcad5381ef710d0a379965980842"
Jul 29 19:47:09.767077 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.767061    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a4a71d55-56789b7844-n8zxl"]
Jul 29 19:47:09.770971 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.770941    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a4a71d55-56789b7844-n8zxl"]
Jul 29 19:47:09.772891 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.772874    2557 scope.go:117] "RemoveContainer" containerID="969afa219c60b60f2ab8dcd2a3ce074b86bcbcad5381ef710d0a379965980842"
Jul 29 19:47:09.773163 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:47:09.773147    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"969afa219c60b60f2ab8dcd2a3ce074b86bcbcad5381ef710d0a379965980842\": container with ID starting with 969afa219c60b60f2ab8dcd2a3ce074b86bcbcad5381ef710d0a379965980842 not found: ID does not exist" containerID="969afa219c60b60f2ab8dcd2a3ce074b86bcbcad5381ef710d0a379965980842"
Jul 29 19:47:09.773213 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.773168    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"969afa219c60b60f2ab8dcd2a3ce074b86bcbcad5381ef710d0a379965980842"} err="failed to get container status \"969afa219c60b60f2ab8dcd2a3ce074b86bcbcad5381ef710d0a379965980842\": rpc error: code = NotFound desc = could not find container \"969afa219c60b60f2ab8dcd2a3ce074b86bcbcad5381ef710d0a379965980842\": container with ID starting with 969afa219c60b60f2ab8dcd2a3ce074b86bcbcad5381ef710d0a379965980842 not found: ID does not exist"
Jul 29 19:47:09.783611 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.783590    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5"]
Jul 29 19:47:09.790602 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.790582    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4a71d55-7b9c67bb84-6c2f5"]
Jul 29 19:47:09.810037 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.809985    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a4a71d55-794d745b88-jf5tt"]
Jul 29 19:47:09.817383 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:09.817364    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a4a71d55-794d745b88-jf5tt"]
Jul 29 19:47:11.678424 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:11.678391    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dcef7b6-4351-4460-9cf1-54b699021513" path="/var/lib/kubelet/pods/5dcef7b6-4351-4460-9cf1-54b699021513/volumes"
Jul 29 19:47:11.678807 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:11.678721    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dac2f7e-58e6-4494-944b-8fdc5ac94a82" path="/var/lib/kubelet/pods/6dac2f7e-58e6-4494-944b-8fdc5ac94a82/volumes"
Jul 29 19:47:11.679068 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:11.679053    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82c7f59c-6550-417f-b9e6-72635ff8d686" path="/var/lib/kubelet/pods/82c7f59c-6550-417f-b9e6-72635ff8d686/volumes"
Jul 29 19:47:14.420872 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:14.420837    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts"]
Jul 29 19:47:14.421394 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:14.421376    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dac2f7e-58e6-4494-944b-8fdc5ac94a82" containerName="payload-processing"
Jul 29 19:47:14.421443 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:14.421398    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dac2f7e-58e6-4494-944b-8fdc5ac94a82" containerName="payload-processing"
Jul 29 19:47:14.421443 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:14.421412    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dcef7b6-4351-4460-9cf1-54b699021513" containerName="maas-api"
Jul 29 19:47:14.421443 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:14.421422    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dcef7b6-4351-4460-9cf1-54b699021513" containerName="maas-api"
Jul 29 19:47:14.421443 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:14.421441    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82c7f59c-6550-417f-b9e6-72635ff8d686" containerName="payload-pre-processing"
Jul 29 19:47:14.421564 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:14.421450    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="82c7f59c-6550-417f-b9e6-72635ff8d686" containerName="payload-pre-processing"
Jul 29 19:47:14.421564 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:14.421463    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7758f49c-19ec-40f6-a7f3-ca9b01a98fd9" containerName="revoke-keys"
Jul 29 19:47:14.421564 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:14.421472    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7758f49c-19ec-40f6-a7f3-ca9b01a98fd9" containerName="revoke-keys"
Jul 29 19:47:14.421564 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:14.421549    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dac2f7e-58e6-4494-944b-8fdc5ac94a82" containerName="payload-processing"
Jul 29 19:47:14.421564 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:14.421559    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7758f49c-19ec-40f6-a7f3-ca9b01a98fd9" containerName="revoke-keys"
Jul 29 19:47:14.421757 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:14.421570    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="82c7f59c-6550-417f-b9e6-72635ff8d686" containerName="payload-pre-processing"
Jul 29 19:47:14.421757 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:14.421581    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dcef7b6-4351-4460-9cf1-54b699021513" containerName="maas-api"
Jul 29 19:47:14.426769 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:14.426748    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts"
Jul 29 19:47:14.430062 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:14.430035    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-23697b7f-serving-cert\""
Jul 29 19:47:14.430686 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:14.430662    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4dw6s\""
Jul 29 19:47:14.431890 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:14.431864    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts"]
Jul 29 19:47:14.493903 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:14.493852    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsdbf\" (UniqueName: \"kubernetes.io/projected/b8c6c876-407f-40e6-ba82-3149eba38bb4-kube-api-access-vsdbf\") pod \"maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts\" (UID: \"b8c6c876-407f-40e6-ba82-3149eba38bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts"
Jul 29 19:47:14.494090 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:14.493987    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c6c876-407f-40e6-ba82-3149eba38bb4-openshift-service-ca\") pod \"maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts\" (UID: \"b8c6c876-407f-40e6-ba82-3149eba38bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts"
Jul 29 19:47:14.494090 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:14.494055    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8c6c876-407f-40e6-ba82-3149eba38bb4-maas-api-tls\") pod \"maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts\" (UID: \"b8c6c876-407f-40e6-ba82-3149eba38bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts"
Jul 29 19:47:14.595135 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:14.595098    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c6c876-407f-40e6-ba82-3149eba38bb4-openshift-service-ca\") pod \"maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts\" (UID: \"b8c6c876-407f-40e6-ba82-3149eba38bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts"
Jul 29 19:47:14.595319 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:14.595169    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8c6c876-407f-40e6-ba82-3149eba38bb4-maas-api-tls\") pod \"maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts\" (UID: \"b8c6c876-407f-40e6-ba82-3149eba38bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts"
Jul 29 19:47:14.595319 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:14.595283    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsdbf\" (UniqueName: \"kubernetes.io/projected/b8c6c876-407f-40e6-ba82-3149eba38bb4-kube-api-access-vsdbf\") pod \"maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts\" (UID: \"b8c6c876-407f-40e6-ba82-3149eba38bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts"
Jul 29 19:47:14.595407 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:47:14.595335    2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-23697b7f-serving-cert: secret "maas-api-e2e-derive-23697b7f-serving-cert" not found
Jul 29 19:47:14.595457 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:47:14.595445    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8c6c876-407f-40e6-ba82-3149eba38bb4-maas-api-tls podName:b8c6c876-407f-40e6-ba82-3149eba38bb4 nodeName:}" failed. No retries permitted until 2026-07-29 19:47:15.095421643 +0000 UTC m=+1805.922920684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b8c6c876-407f-40e6-ba82-3149eba38bb4-maas-api-tls") pod "maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts" (UID: "b8c6c876-407f-40e6-ba82-3149eba38bb4") : secret "maas-api-e2e-derive-23697b7f-serving-cert" not found
Jul 29 19:47:14.595758 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:14.595740    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c6c876-407f-40e6-ba82-3149eba38bb4-openshift-service-ca\") pod \"maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts\" (UID: \"b8c6c876-407f-40e6-ba82-3149eba38bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts"
Jul 29 19:47:14.604646 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:14.604620    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsdbf\" (UniqueName: \"kubernetes.io/projected/b8c6c876-407f-40e6-ba82-3149eba38bb4-kube-api-access-vsdbf\") pod \"maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts\" (UID: \"b8c6c876-407f-40e6-ba82-3149eba38bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts"
Jul 29 19:47:15.098919 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.098879    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8c6c876-407f-40e6-ba82-3149eba38bb4-maas-api-tls\") pod \"maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts\" (UID: \"b8c6c876-407f-40e6-ba82-3149eba38bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts"
Jul 29 19:47:15.101080 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.101056    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-23697b7f-56fc68bc66-hqxd6"]
Jul 29 19:47:15.101601 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.101579    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8c6c876-407f-40e6-ba82-3149eba38bb4-maas-api-tls\") pod \"maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts\" (UID: \"b8c6c876-407f-40e6-ba82-3149eba38bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts"
Jul 29 19:47:15.109833 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.109810    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-23697b7f-56fc68bc66-hqxd6"
Jul 29 19:47:15.112689 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.112669    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-23697b7f-dockercfg-gphrx\""
Jul 29 19:47:15.112775 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.112669    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-23697b7f\""
Jul 29 19:47:15.118574 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.118550    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-23697b7f-56fc68bc66-hqxd6"]
Jul 29 19:47:15.145097 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.145071    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-23697b7f-57f74987b8-mng2n"]
Jul 29 19:47:15.148295 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.148281    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-23697b7f-57f74987b8-mng2n"
Jul 29 19:47:15.158939 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.158911    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-23697b7f-57f74987b8-mng2n"]
Jul 29 19:47:15.199976 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.199936    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bb9f3ae-0397-449a-8f66-49786798d75c-plugins-config-volume\") pod \"payload-processing-e2e-derive-23697b7f-56fc68bc66-hqxd6\" (UID: \"3bb9f3ae-0397-449a-8f66-49786798d75c\") " pod="openshift-ingress/payload-processing-e2e-derive-23697b7f-56fc68bc66-hqxd6"
Jul 29 19:47:15.200094 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.200055    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-23697b7f-57f74987b8-mng2n\" (UID: \"ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-23697b7f-57f74987b8-mng2n"
Jul 29 19:47:15.200149 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.200112    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs8bw\" (UniqueName: \"kubernetes.io/projected/ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d-kube-api-access-vs8bw\") pod \"payload-pre-processing-e2e-derive-23697b7f-57f74987b8-mng2n\" (UID: \"ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-23697b7f-57f74987b8-mng2n"
Jul 29 19:47:15.200149 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.200131    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg55s\" (UniqueName: \"kubernetes.io/projected/3bb9f3ae-0397-449a-8f66-49786798d75c-kube-api-access-xg55s\") pod \"payload-processing-e2e-derive-23697b7f-56fc68bc66-hqxd6\" (UID: \"3bb9f3ae-0397-449a-8f66-49786798d75c\") " pod="openshift-ingress/payload-processing-e2e-derive-23697b7f-56fc68bc66-hqxd6"
Jul 29 19:47:15.301304 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.301275    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs8bw\" (UniqueName: \"kubernetes.io/projected/ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d-kube-api-access-vs8bw\") pod \"payload-pre-processing-e2e-derive-23697b7f-57f74987b8-mng2n\" (UID: \"ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-23697b7f-57f74987b8-mng2n"
Jul 29 19:47:15.301304 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.301309    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg55s\" (UniqueName: \"kubernetes.io/projected/3bb9f3ae-0397-449a-8f66-49786798d75c-kube-api-access-xg55s\") pod \"payload-processing-e2e-derive-23697b7f-56fc68bc66-hqxd6\" (UID: \"3bb9f3ae-0397-449a-8f66-49786798d75c\") " pod="openshift-ingress/payload-processing-e2e-derive-23697b7f-56fc68bc66-hqxd6"
Jul 29 19:47:15.301540 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.301352    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bb9f3ae-0397-449a-8f66-49786798d75c-plugins-config-volume\") pod \"payload-processing-e2e-derive-23697b7f-56fc68bc66-hqxd6\" (UID: \"3bb9f3ae-0397-449a-8f66-49786798d75c\") " pod="openshift-ingress/payload-processing-e2e-derive-23697b7f-56fc68bc66-hqxd6"
Jul 29 19:47:15.301540 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.301431    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-23697b7f-57f74987b8-mng2n\" (UID: \"ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-23697b7f-57f74987b8-mng2n"
Jul 29 19:47:15.302128 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.302104    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-23697b7f-57f74987b8-mng2n\" (UID: \"ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-23697b7f-57f74987b8-mng2n"
Jul 29 19:47:15.302220 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.302142    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bb9f3ae-0397-449a-8f66-49786798d75c-plugins-config-volume\") pod \"payload-processing-e2e-derive-23697b7f-56fc68bc66-hqxd6\" (UID: \"3bb9f3ae-0397-449a-8f66-49786798d75c\") " pod="openshift-ingress/payload-processing-e2e-derive-23697b7f-56fc68bc66-hqxd6"
Jul 29 19:47:15.309875 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.309855    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs8bw\" (UniqueName: \"kubernetes.io/projected/ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d-kube-api-access-vs8bw\") pod \"payload-pre-processing-e2e-derive-23697b7f-57f74987b8-mng2n\" (UID: \"ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-23697b7f-57f74987b8-mng2n"
Jul 29 19:47:15.309998 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.309959    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg55s\" (UniqueName: \"kubernetes.io/projected/3bb9f3ae-0397-449a-8f66-49786798d75c-kube-api-access-xg55s\") pod \"payload-processing-e2e-derive-23697b7f-56fc68bc66-hqxd6\" (UID: \"3bb9f3ae-0397-449a-8f66-49786798d75c\") " pod="openshift-ingress/payload-processing-e2e-derive-23697b7f-56fc68bc66-hqxd6"
Jul 29 19:47:15.339740 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.339718    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts"
Jul 29 19:47:15.421947 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.421918    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-23697b7f-56fc68bc66-hqxd6"
Jul 29 19:47:15.458161 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.458131    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-23697b7f-57f74987b8-mng2n"
Jul 29 19:47:15.481933 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.481901    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts"]
Jul 29 19:47:15.484453 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:47:15.484384    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c6c876_407f_40e6_ba82_3149eba38bb4.slice/crio-591d3b67768e5b6437d94d45cd4ccea6f46f027a693ac16123f19186c7c55ecc WatchSource:0}: Error finding container 591d3b67768e5b6437d94d45cd4ccea6f46f027a693ac16123f19186c7c55ecc: Status 404 returned error can't find the container with id 591d3b67768e5b6437d94d45cd4ccea6f46f027a693ac16123f19186c7c55ecc
Jul 29 19:47:15.571387 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.571355    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-23697b7f-56fc68bc66-hqxd6"]
Jul 29 19:47:15.573757 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:47:15.573729    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb9f3ae_0397_449a_8f66_49786798d75c.slice/crio-d82c490cc12a9d18fb597796597126216d59249614e2cc7538e2be34319d92f3 WatchSource:0}: Error finding container d82c490cc12a9d18fb597796597126216d59249614e2cc7538e2be34319d92f3: Status 404 returned error can't find the container with id d82c490cc12a9d18fb597796597126216d59249614e2cc7538e2be34319d92f3
Jul 29 19:47:15.601445 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.601422    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-23697b7f-57f74987b8-mng2n"]
Jul 29 19:47:15.605139 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:47:15.605117    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff4667ba_d5c6_4187_9c5b_481b9e2dfd7d.slice/crio-3bb245f46cfc3398eb28ee0e67025eb311c0934f7cd90c3afc507009c3eca28a WatchSource:0}: Error finding container 3bb245f46cfc3398eb28ee0e67025eb311c0934f7cd90c3afc507009c3eca28a: Status 404 returned error can't find the container with id 3bb245f46cfc3398eb28ee0e67025eb311c0934f7cd90c3afc507009c3eca28a
Jul 29 19:47:15.785830 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.785790    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-23697b7f-57f74987b8-mng2n" event={"ID":"ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d","Type":"ContainerStarted","Data":"c7d2191a995d718d0d96f26d21a661b9170e33057e85bacbf620577619321f73"}
Jul 29 19:47:15.786034 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.785854    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-23697b7f-57f74987b8-mng2n" event={"ID":"ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d","Type":"ContainerStarted","Data":"3bb245f46cfc3398eb28ee0e67025eb311c0934f7cd90c3afc507009c3eca28a"}
Jul 29 19:47:15.786034 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.785985    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-23697b7f-57f74987b8-mng2n"
Jul 29 19:47:15.787526 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.787417    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-23697b7f-56fc68bc66-hqxd6" event={"ID":"3bb9f3ae-0397-449a-8f66-49786798d75c","Type":"ContainerStarted","Data":"3d44da157a161ed0eb1be44adf2424a098aed0d518be42d319143de83e6fd929"}
Jul 29 19:47:15.787526 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.787463    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-23697b7f-56fc68bc66-hqxd6" event={"ID":"3bb9f3ae-0397-449a-8f66-49786798d75c","Type":"ContainerStarted","Data":"d82c490cc12a9d18fb597796597126216d59249614e2cc7538e2be34319d92f3"}
Jul 29 19:47:15.788632 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.788610    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts" event={"ID":"b8c6c876-407f-40e6-ba82-3149eba38bb4","Type":"ContainerStarted","Data":"591d3b67768e5b6437d94d45cd4ccea6f46f027a693ac16123f19186c7c55ecc"}
Jul 29 19:47:15.805024 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.804956    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-23697b7f-57f74987b8-mng2n" podStartSLOduration=0.804942945 podStartE2EDuration="804.942945ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:47:15.803333503 +0000 UTC m=+1806.630832561" watchObservedRunningTime="2026-07-29 19:47:15.804942945 +0000 UTC m=+1806.632442059"
Jul 29 19:47:15.822488 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:15.822449    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-23697b7f-56fc68bc66-hqxd6" podStartSLOduration=0.822439018 podStartE2EDuration="822.439018ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:47:15.82068851 +0000 UTC m=+1806.648187601" watchObservedRunningTime="2026-07-29 19:47:15.822439018 +0000 UTC m=+1806.649938053"
Jul 29 19:47:16.795286 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:16.795247    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts" event={"ID":"b8c6c876-407f-40e6-ba82-3149eba38bb4","Type":"ContainerStarted","Data":"f27498db7161d8b24db9ad5c4bf7fcd384e84d749e99b1f1cba7bc52a0ecc1fb"}
Jul 29 19:47:16.795751 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:16.795578    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts"
Jul 29 19:47:16.795751 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:16.795633    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-23697b7f-56fc68bc66-hqxd6"
Jul 29 19:47:16.813111 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:16.813070    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts" podStartSLOduration=2.512314085 podStartE2EDuration="2.813055766s" totalImagesPullingTime="300.741681ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:47:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:47:16.812475906 +0000 UTC m=+1807.639974965" watchObservedRunningTime="2026-07-29 19:47:16.813055766 +0000 UTC m=+1807.640554821"
Jul 29 19:47:23.809458 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:23.809428    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts"
Jul 29 19:47:26.797668 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:26.797634    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-23697b7f-57f74987b8-mng2n"
Jul 29 19:47:27.802378 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:27.802347    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-23697b7f-56fc68bc66-hqxd6"
Jul 29 19:47:38.168314 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:38.168273    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-23697b7f-nl7ln"]
Jul 29 19:47:38.173137 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:38.173114    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-23697b7f-nl7ln"
Jul 29 19:47:38.176060 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:38.176036    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jsq2r\""
Jul 29 19:47:38.180034 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:38.180000    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-23697b7f-nl7ln"]
Jul 29 19:47:38.309442 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:38.309409    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d05ea0f1-137e-41ca-947b-fc5885f584f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-23697b7f-nl7ln\" (UID: \"d05ea0f1-137e-41ca-947b-fc5885f584f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-23697b7f-nl7ln"
Jul 29 19:47:38.410641 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:38.410611    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d05ea0f1-137e-41ca-947b-fc5885f584f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-23697b7f-nl7ln\" (UID: \"d05ea0f1-137e-41ca-947b-fc5885f584f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-23697b7f-nl7ln"
Jul 29 19:47:38.411264 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:38.411246    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d05ea0f1-137e-41ca-947b-fc5885f584f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-23697b7f-nl7ln\" (UID: \"d05ea0f1-137e-41ca-947b-fc5885f584f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-23697b7f-nl7ln"
Jul 29 19:47:38.484831 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:38.484753    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-23697b7f-nl7ln"
Jul 29 19:47:38.822449 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:38.822422    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-23697b7f-nl7ln"]
Jul 29 19:47:38.824829 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:47:38.824800    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd05ea0f1_137e_41ca_947b_fc5885f584f1.slice/crio-3137e4ff18ff7be6d5b3c8a3076926314c5af963722917bc5ef6af26ba014729 WatchSource:0}: Error finding container 3137e4ff18ff7be6d5b3c8a3076926314c5af963722917bc5ef6af26ba014729: Status 404 returned error can't find the container with id 3137e4ff18ff7be6d5b3c8a3076926314c5af963722917bc5ef6af26ba014729
Jul 29 19:47:38.905900 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:38.905867    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-23697b7f-nl7ln" event={"ID":"d05ea0f1-137e-41ca-947b-fc5885f584f1","Type":"ContainerStarted","Data":"3137e4ff18ff7be6d5b3c8a3076926314c5af963722917bc5ef6af26ba014729"}
Jul 29 19:47:39.911692 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:39.911652    2557 generic.go:356] "Generic (PLEG): container finished" podID="d05ea0f1-137e-41ca-947b-fc5885f584f1" containerID="1ec09ee8aa1c52df70298904b3adfc55077e50c182af2b9f5fbe31fa9dfe59bd" exitCode=0
Jul 29 19:47:39.912123 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:39.911730    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-23697b7f-nl7ln" event={"ID":"d05ea0f1-137e-41ca-947b-fc5885f584f1","Type":"ContainerDied","Data":"1ec09ee8aa1c52df70298904b3adfc55077e50c182af2b9f5fbe31fa9dfe59bd"}
Jul 29 19:47:41.049904 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:41.049879    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-23697b7f-nl7ln"
Jul 29 19:47:41.134205 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:41.134177    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d05ea0f1-137e-41ca-947b-fc5885f584f1-maas-api-service-ca\") pod \"d05ea0f1-137e-41ca-947b-fc5885f584f1\" (UID: \"d05ea0f1-137e-41ca-947b-fc5885f584f1\") "
Jul 29 19:47:41.134495 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:41.134471    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d05ea0f1-137e-41ca-947b-fc5885f584f1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d05ea0f1-137e-41ca-947b-fc5885f584f1" (UID: "d05ea0f1-137e-41ca-947b-fc5885f584f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:47:41.234817 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:41.234744    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d05ea0f1-137e-41ca-947b-fc5885f584f1-maas-api-service-ca\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:47:41.922149 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:41.922113    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-23697b7f-nl7ln" event={"ID":"d05ea0f1-137e-41ca-947b-fc5885f584f1","Type":"ContainerDied","Data":"3137e4ff18ff7be6d5b3c8a3076926314c5af963722917bc5ef6af26ba014729"}
Jul 29 19:47:41.922149 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:41.922132    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-23697b7f-nl7ln"
Jul 29 19:47:41.922356 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:41.922169    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3137e4ff18ff7be6d5b3c8a3076926314c5af963722917bc5ef6af26ba014729"
Jul 29 19:47:44.978130 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:44.978096    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts"]
Jul 29 19:47:44.978698 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:44.978305    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts" podUID="b8c6c876-407f-40e6-ba82-3149eba38bb4" containerName="maas-api" containerID="cri-o://f27498db7161d8b24db9ad5c4bf7fcd384e84d749e99b1f1cba7bc52a0ecc1fb" gracePeriod=30
Jul 29 19:47:45.051638 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.051602    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-23697b7f-56fc68bc66-hqxd6"]
Jul 29 19:47:45.051870 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.051845    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-23697b7f-56fc68bc66-hqxd6" podUID="3bb9f3ae-0397-449a-8f66-49786798d75c" containerName="payload-processing" containerID="cri-o://3d44da157a161ed0eb1be44adf2424a098aed0d518be42d319143de83e6fd929" gracePeriod=30
Jul 29 19:47:45.061398 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.061372    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-23697b7f-57f74987b8-mng2n"]
Jul 29 19:47:45.061603 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.061575    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-23697b7f-57f74987b8-mng2n" podUID="ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d" containerName="payload-pre-processing" containerID="cri-o://c7d2191a995d718d0d96f26d21a661b9170e33057e85bacbf620577619321f73" gracePeriod=30
Jul 29 19:47:45.259242 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.259208    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-23697b7f-nl7ln"]
Jul 29 19:47:45.265811 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.265787    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-23697b7f-nl7ln"]
Jul 29 19:47:45.316830 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.316780    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts"
Jul 29 19:47:45.343481 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.343459    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-23697b7f-56fc68bc66-hqxd6"
Jul 29 19:47:45.353760 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.353738    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-23697b7f-57f74987b8-mng2n"
Jul 29 19:47:45.369055 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.369027    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d-plugins-config-volume\") pod \"ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d\" (UID: \"ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d\") "
Jul 29 19:47:45.370106 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.369076    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsdbf\" (UniqueName: \"kubernetes.io/projected/b8c6c876-407f-40e6-ba82-3149eba38bb4-kube-api-access-vsdbf\") pod \"b8c6c876-407f-40e6-ba82-3149eba38bb4\" (UID: \"b8c6c876-407f-40e6-ba82-3149eba38bb4\") "
Jul 29 19:47:45.370106 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.369104    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bb9f3ae-0397-449a-8f66-49786798d75c-plugins-config-volume\") pod \"3bb9f3ae-0397-449a-8f66-49786798d75c\" (UID: \"3bb9f3ae-0397-449a-8f66-49786798d75c\") "
Jul 29 19:47:45.370106 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.369132    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8c6c876-407f-40e6-ba82-3149eba38bb4-maas-api-tls\") pod \"b8c6c876-407f-40e6-ba82-3149eba38bb4\" (UID: \"b8c6c876-407f-40e6-ba82-3149eba38bb4\") "
Jul 29 19:47:45.370106 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.369169    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c6c876-407f-40e6-ba82-3149eba38bb4-openshift-service-ca\") pod \"b8c6c876-407f-40e6-ba82-3149eba38bb4\" (UID: \"b8c6c876-407f-40e6-ba82-3149eba38bb4\") "
Jul 29 19:47:45.370106 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.369193    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vs8bw\" (UniqueName: \"kubernetes.io/projected/ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d-kube-api-access-vs8bw\") pod \"ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d\" (UID: \"ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d\") "
Jul 29 19:47:45.370106 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.369253    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xg55s\" (UniqueName: \"kubernetes.io/projected/3bb9f3ae-0397-449a-8f66-49786798d75c-kube-api-access-xg55s\") pod \"3bb9f3ae-0397-449a-8f66-49786798d75c\" (UID: \"3bb9f3ae-0397-449a-8f66-49786798d75c\") "
Jul 29 19:47:45.370106 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.369794    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bb9f3ae-0397-449a-8f66-49786798d75c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3bb9f3ae-0397-449a-8f66-49786798d75c" (UID: "3bb9f3ae-0397-449a-8f66-49786798d75c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:47:45.370490 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.370134    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8c6c876-407f-40e6-ba82-3149eba38bb4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b8c6c876-407f-40e6-ba82-3149eba38bb4" (UID: "b8c6c876-407f-40e6-ba82-3149eba38bb4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:47:45.370490 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.370429    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d" (UID: "ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:47:45.372661 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.372597    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8c6c876-407f-40e6-ba82-3149eba38bb4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b8c6c876-407f-40e6-ba82-3149eba38bb4" (UID: "b8c6c876-407f-40e6-ba82-3149eba38bb4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:47:45.373248 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.373212    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d-kube-api-access-vs8bw" (OuterVolumeSpecName: "kube-api-access-vs8bw") pod "ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d" (UID: "ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d"). InnerVolumeSpecName "kube-api-access-vs8bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:47:45.373347 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.373264    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c6c876-407f-40e6-ba82-3149eba38bb4-kube-api-access-vsdbf" (OuterVolumeSpecName: "kube-api-access-vsdbf") pod "b8c6c876-407f-40e6-ba82-3149eba38bb4" (UID: "b8c6c876-407f-40e6-ba82-3149eba38bb4"). InnerVolumeSpecName "kube-api-access-vsdbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:47:45.373347 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.373302    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb9f3ae-0397-449a-8f66-49786798d75c-kube-api-access-xg55s" (OuterVolumeSpecName: "kube-api-access-xg55s") pod "3bb9f3ae-0397-449a-8f66-49786798d75c" (UID: "3bb9f3ae-0397-449a-8f66-49786798d75c"). InnerVolumeSpecName "kube-api-access-xg55s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:47:45.470575 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.470537    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8c6c876-407f-40e6-ba82-3149eba38bb4-maas-api-tls\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:47:45.470575 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.470568    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c6c876-407f-40e6-ba82-3149eba38bb4-openshift-service-ca\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:47:45.470575 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.470578    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs8bw\" (UniqueName: \"kubernetes.io/projected/ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d-kube-api-access-vs8bw\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:47:45.470808 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.470587    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg55s\" (UniqueName: \"kubernetes.io/projected/3bb9f3ae-0397-449a-8f66-49786798d75c-kube-api-access-xg55s\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:47:45.470808 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.470595    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d-plugins-config-volume\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:47:45.470808 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.470606    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsdbf\" (UniqueName: \"kubernetes.io/projected/b8c6c876-407f-40e6-ba82-3149eba38bb4-kube-api-access-vsdbf\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:47:45.470808 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.470614    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bb9f3ae-0397-449a-8f66-49786798d75c-plugins-config-volume\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:47:45.678785 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.678748    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d05ea0f1-137e-41ca-947b-fc5885f584f1" path="/var/lib/kubelet/pods/d05ea0f1-137e-41ca-947b-fc5885f584f1/volumes"
Jul 29 19:47:45.940616 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.940528    2557 generic.go:356] "Generic (PLEG): container finished" podID="ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d" containerID="c7d2191a995d718d0d96f26d21a661b9170e33057e85bacbf620577619321f73" exitCode=0
Jul 29 19:47:45.940616 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.940570    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-23697b7f-57f74987b8-mng2n"
Jul 29 19:47:45.940857 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.940614    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-23697b7f-57f74987b8-mng2n" event={"ID":"ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d","Type":"ContainerDied","Data":"c7d2191a995d718d0d96f26d21a661b9170e33057e85bacbf620577619321f73"}
Jul 29 19:47:45.940857 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.940674    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-23697b7f-57f74987b8-mng2n" event={"ID":"ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d","Type":"ContainerDied","Data":"3bb245f46cfc3398eb28ee0e67025eb311c0934f7cd90c3afc507009c3eca28a"}
Jul 29 19:47:45.940857 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.940705    2557 scope.go:117] "RemoveContainer" containerID="c7d2191a995d718d0d96f26d21a661b9170e33057e85bacbf620577619321f73"
Jul 29 19:47:45.942089 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.942068    2557 generic.go:356] "Generic (PLEG): container finished" podID="3bb9f3ae-0397-449a-8f66-49786798d75c" containerID="3d44da157a161ed0eb1be44adf2424a098aed0d518be42d319143de83e6fd929" exitCode=0
Jul 29 19:47:45.942258 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.942109    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-23697b7f-56fc68bc66-hqxd6"
Jul 29 19:47:45.942258 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.942140    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-23697b7f-56fc68bc66-hqxd6" event={"ID":"3bb9f3ae-0397-449a-8f66-49786798d75c","Type":"ContainerDied","Data":"3d44da157a161ed0eb1be44adf2424a098aed0d518be42d319143de83e6fd929"}
Jul 29 19:47:45.942258 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.942181    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-23697b7f-56fc68bc66-hqxd6" event={"ID":"3bb9f3ae-0397-449a-8f66-49786798d75c","Type":"ContainerDied","Data":"d82c490cc12a9d18fb597796597126216d59249614e2cc7538e2be34319d92f3"}
Jul 29 19:47:45.943742 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.943722    2557 generic.go:356] "Generic (PLEG): container finished" podID="b8c6c876-407f-40e6-ba82-3149eba38bb4" containerID="f27498db7161d8b24db9ad5c4bf7fcd384e84d749e99b1f1cba7bc52a0ecc1fb" exitCode=0
Jul 29 19:47:45.943869 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.943770    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts"
Jul 29 19:47:45.943869 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.943800    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts" event={"ID":"b8c6c876-407f-40e6-ba82-3149eba38bb4","Type":"ContainerDied","Data":"f27498db7161d8b24db9ad5c4bf7fcd384e84d749e99b1f1cba7bc52a0ecc1fb"}
Jul 29 19:47:45.944000 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.943865    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts" event={"ID":"b8c6c876-407f-40e6-ba82-3149eba38bb4","Type":"ContainerDied","Data":"591d3b67768e5b6437d94d45cd4ccea6f46f027a693ac16123f19186c7c55ecc"}
Jul 29 19:47:45.950700 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.950689    2557 scope.go:117] "RemoveContainer" containerID="c7d2191a995d718d0d96f26d21a661b9170e33057e85bacbf620577619321f73"
Jul 29 19:47:45.950988 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:47:45.950948    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7d2191a995d718d0d96f26d21a661b9170e33057e85bacbf620577619321f73\": container with ID starting with c7d2191a995d718d0d96f26d21a661b9170e33057e85bacbf620577619321f73 not found: ID does not exist" containerID="c7d2191a995d718d0d96f26d21a661b9170e33057e85bacbf620577619321f73"
Jul 29 19:47:45.951113 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.950990    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7d2191a995d718d0d96f26d21a661b9170e33057e85bacbf620577619321f73"} err="failed to get container status \"c7d2191a995d718d0d96f26d21a661b9170e33057e85bacbf620577619321f73\": rpc error: code = NotFound desc = could not find container \"c7d2191a995d718d0d96f26d21a661b9170e33057e85bacbf620577619321f73\": container with ID starting with c7d2191a995d718d0d96f26d21a661b9170e33057e85bacbf620577619321f73 not found: ID does not exist"
Jul 29 19:47:45.951113 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.951004    2557 scope.go:117] "RemoveContainer" containerID="3d44da157a161ed0eb1be44adf2424a098aed0d518be42d319143de83e6fd929"
Jul 29 19:47:45.958324 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.958307    2557 scope.go:117] "RemoveContainer" containerID="3d44da157a161ed0eb1be44adf2424a098aed0d518be42d319143de83e6fd929"
Jul 29 19:47:45.958542 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:47:45.958525    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d44da157a161ed0eb1be44adf2424a098aed0d518be42d319143de83e6fd929\": container with ID starting with 3d44da157a161ed0eb1be44adf2424a098aed0d518be42d319143de83e6fd929 not found: ID does not exist" containerID="3d44da157a161ed0eb1be44adf2424a098aed0d518be42d319143de83e6fd929"
Jul 29 19:47:45.958587 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.958551    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d44da157a161ed0eb1be44adf2424a098aed0d518be42d319143de83e6fd929"} err="failed to get container status \"3d44da157a161ed0eb1be44adf2424a098aed0d518be42d319143de83e6fd929\": rpc error: code = NotFound desc = could not find container \"3d44da157a161ed0eb1be44adf2424a098aed0d518be42d319143de83e6fd929\": container with ID starting with 3d44da157a161ed0eb1be44adf2424a098aed0d518be42d319143de83e6fd929 not found: ID does not exist"
Jul 29 19:47:45.958587 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.958568    2557 scope.go:117] "RemoveContainer" containerID="f27498db7161d8b24db9ad5c4bf7fcd384e84d749e99b1f1cba7bc52a0ecc1fb"
Jul 29 19:47:45.967109 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.967086    2557 scope.go:117] "RemoveContainer" containerID="f27498db7161d8b24db9ad5c4bf7fcd384e84d749e99b1f1cba7bc52a0ecc1fb"
Jul 29 19:47:45.967427 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:47:45.967406    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f27498db7161d8b24db9ad5c4bf7fcd384e84d749e99b1f1cba7bc52a0ecc1fb\": container with ID starting with f27498db7161d8b24db9ad5c4bf7fcd384e84d749e99b1f1cba7bc52a0ecc1fb not found: ID does not exist" containerID="f27498db7161d8b24db9ad5c4bf7fcd384e84d749e99b1f1cba7bc52a0ecc1fb"
Jul 29 19:47:45.967493 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.967429    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts"]
Jul 29 19:47:45.967493 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.967432    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f27498db7161d8b24db9ad5c4bf7fcd384e84d749e99b1f1cba7bc52a0ecc1fb"} err="failed to get container status \"f27498db7161d8b24db9ad5c4bf7fcd384e84d749e99b1f1cba7bc52a0ecc1fb\": rpc error: code = NotFound desc = could not find container \"f27498db7161d8b24db9ad5c4bf7fcd384e84d749e99b1f1cba7bc52a0ecc1fb\": container with ID starting with f27498db7161d8b24db9ad5c4bf7fcd384e84d749e99b1f1cba7bc52a0ecc1fb not found: ID does not exist"
Jul 29 19:47:45.969097 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.969080    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-23697b7f-7ffcd8b9bf-758ts"]
Jul 29 19:47:45.980134 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.980110    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-23697b7f-57f74987b8-mng2n"]
Jul 29 19:47:45.983877 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.983856    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-23697b7f-57f74987b8-mng2n"]
Jul 29 19:47:45.994098 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:45.994079    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-23697b7f-56fc68bc66-hqxd6"]
Jul 29 19:47:46.001556 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:46.001536    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-23697b7f-56fc68bc66-hqxd6"]
Jul 29 19:47:47.679203 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:47.679166    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bb9f3ae-0397-449a-8f66-49786798d75c" path="/var/lib/kubelet/pods/3bb9f3ae-0397-449a-8f66-49786798d75c/volumes"
Jul 29 19:47:47.679565 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:47.679540    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8c6c876-407f-40e6-ba82-3149eba38bb4" path="/var/lib/kubelet/pods/b8c6c876-407f-40e6-ba82-3149eba38bb4/volumes"
Jul 29 19:47:47.679842 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:47:47.679830    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d" path="/var/lib/kubelet/pods/ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d/volumes"
Jul 29 19:49:39.829687 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:39.829648    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6"]
Jul 29 19:49:39.830223 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:39.830163    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d" containerName="payload-pre-processing"
Jul 29 19:49:39.830223 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:39.830183    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d" containerName="payload-pre-processing"
Jul 29 19:49:39.830223 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:39.830204    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bb9f3ae-0397-449a-8f66-49786798d75c" containerName="payload-processing"
Jul 29 19:49:39.830223 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:39.830212    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bb9f3ae-0397-449a-8f66-49786798d75c" containerName="payload-processing"
Jul 29 19:49:39.830445 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:39.830234    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d05ea0f1-137e-41ca-947b-fc5885f584f1" containerName="revoke-keys"
Jul 29 19:49:39.830445 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:39.830244    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d05ea0f1-137e-41ca-947b-fc5885f584f1" containerName="revoke-keys"
Jul 29 19:49:39.830445 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:39.830259    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8c6c876-407f-40e6-ba82-3149eba38bb4" containerName="maas-api"
Jul 29 19:49:39.830445 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:39.830269    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8c6c876-407f-40e6-ba82-3149eba38bb4" containerName="maas-api"
Jul 29 19:49:39.830445 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:39.830361    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8c6c876-407f-40e6-ba82-3149eba38bb4" containerName="maas-api"
Jul 29 19:49:39.830445 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:39.830375    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d05ea0f1-137e-41ca-947b-fc5885f584f1" containerName="revoke-keys"
Jul 29 19:49:39.830445 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:39.830386    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bb9f3ae-0397-449a-8f66-49786798d75c" containerName="payload-processing"
Jul 29 19:49:39.830445 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:39.830396    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff4667ba-d5c6-4187-9c5b-481b9e2dfd7d" containerName="payload-pre-processing"
Jul 29 19:49:39.833587 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:39.833557    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6"
Jul 29 19:49:39.836332 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:39.836314    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-20262e43-serving-cert\""
Jul 29 19:49:39.836490 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:39.836473    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4dw6s\""
Jul 29 19:49:39.848101 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:39.848076    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6"]
Jul 29 19:49:39.987060 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:39.987027    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69255464-8e45-426c-b73b-0d834090801b-openshift-service-ca\") pod \"maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6\" (UID: \"69255464-8e45-426c-b73b-0d834090801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6"
Jul 29 19:49:39.987275 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:39.987075    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69255464-8e45-426c-b73b-0d834090801b-maas-api-tls\") pod \"maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6\" (UID: \"69255464-8e45-426c-b73b-0d834090801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6"
Jul 29 19:49:39.987275 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:39.987224    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz628\" (UniqueName: \"kubernetes.io/projected/69255464-8e45-426c-b73b-0d834090801b-kube-api-access-gz628\") pod \"maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6\" (UID: \"69255464-8e45-426c-b73b-0d834090801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6"
Jul 29 19:49:40.088537 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:40.088451    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69255464-8e45-426c-b73b-0d834090801b-maas-api-tls\") pod \"maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6\" (UID: \"69255464-8e45-426c-b73b-0d834090801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6"
Jul 29 19:49:40.088537 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:40.088504    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz628\" (UniqueName: \"kubernetes.io/projected/69255464-8e45-426c-b73b-0d834090801b-kube-api-access-gz628\") pod \"maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6\" (UID: \"69255464-8e45-426c-b73b-0d834090801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6"
Jul 29 19:49:40.088731 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:40.088567    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69255464-8e45-426c-b73b-0d834090801b-openshift-service-ca\") pod \"maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6\" (UID: \"69255464-8e45-426c-b73b-0d834090801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6"
Jul 29 19:49:40.089266 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:40.089246    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69255464-8e45-426c-b73b-0d834090801b-openshift-service-ca\") pod \"maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6\" (UID: \"69255464-8e45-426c-b73b-0d834090801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6"
Jul 29 19:49:40.090926 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:40.090904    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69255464-8e45-426c-b73b-0d834090801b-maas-api-tls\") pod \"maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6\" (UID: \"69255464-8e45-426c-b73b-0d834090801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6"
Jul 29 19:49:40.102777 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:40.102748    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz628\" (UniqueName: \"kubernetes.io/projected/69255464-8e45-426c-b73b-0d834090801b-kube-api-access-gz628\") pod \"maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6\" (UID: \"69255464-8e45-426c-b73b-0d834090801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6"
Jul 29 19:49:40.143818 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:40.143794    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6"
Jul 29 19:49:40.286289 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:40.286259    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6"]
Jul 29 19:49:40.287530 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:49:40.287500    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69255464_8e45_426c_b73b_0d834090801b.slice/crio-88f198a6622873f4701442bd064cd677bd4b0533440b7c99ab6b0eca4e2b3d34 WatchSource:0}: Error finding container 88f198a6622873f4701442bd064cd677bd4b0533440b7c99ab6b0eca4e2b3d34: Status 404 returned error can't find the container with id 88f198a6622873f4701442bd064cd677bd4b0533440b7c99ab6b0eca4e2b3d34
Jul 29 19:49:40.486388 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:40.486349    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6" event={"ID":"69255464-8e45-426c-b73b-0d834090801b","Type":"ContainerStarted","Data":"88f198a6622873f4701442bd064cd677bd4b0533440b7c99ab6b0eca4e2b3d34"}
Jul 29 19:49:40.881347 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:40.881314    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-20262e43-8955bfdc4-l5f7m"]
Jul 29 19:49:40.884498 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:40.884479    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-20262e43-8955bfdc4-l5f7m"
Jul 29 19:49:40.887768 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:40.887746    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-20262e43\""
Jul 29 19:49:40.888535 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:40.888516    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-20262e43-dockercfg-rxshp\""
Jul 29 19:49:40.899008 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:40.898955    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-20262e43-8955bfdc4-l5f7m"]
Jul 29 19:49:40.984430 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:40.984395    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-20262e43-68fc64c8dd-lfztt"]
Jul 29 19:49:40.987774 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:40.987752    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-20262e43-68fc64c8dd-lfztt"
Jul 29 19:49:40.997356 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:40.997300    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8483bc2-dc0b-470b-b1df-ae39814872fb-plugins-config-volume\") pod \"payload-processing-e2e-mt-20262e43-8955bfdc4-l5f7m\" (UID: \"d8483bc2-dc0b-470b-b1df-ae39814872fb\") " pod="openshift-ingress/payload-processing-e2e-mt-20262e43-8955bfdc4-l5f7m"
Jul 29 19:49:40.997356 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:40.997329    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bsw2\" (UniqueName: \"kubernetes.io/projected/d8483bc2-dc0b-470b-b1df-ae39814872fb-kube-api-access-6bsw2\") pod \"payload-processing-e2e-mt-20262e43-8955bfdc4-l5f7m\" (UID: \"d8483bc2-dc0b-470b-b1df-ae39814872fb\") " pod="openshift-ingress/payload-processing-e2e-mt-20262e43-8955bfdc4-l5f7m"
Jul 29 19:49:41.004089 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:41.003151    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-20262e43-68fc64c8dd-lfztt"]
Jul 29 19:49:41.098030 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:41.097987    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7179fce-31d6-46b8-acae-d1bbccff47fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-20262e43-68fc64c8dd-lfztt\" (UID: \"c7179fce-31d6-46b8-acae-d1bbccff47fb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-20262e43-68fc64c8dd-lfztt"
Jul 29 19:49:41.098257 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:41.098074    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ddjl\" (UniqueName: \"kubernetes.io/projected/c7179fce-31d6-46b8-acae-d1bbccff47fb-kube-api-access-2ddjl\") pod \"payload-pre-processing-e2e-mt-20262e43-68fc64c8dd-lfztt\" (UID: \"c7179fce-31d6-46b8-acae-d1bbccff47fb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-20262e43-68fc64c8dd-lfztt"
Jul 29 19:49:41.098257 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:41.098127    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8483bc2-dc0b-470b-b1df-ae39814872fb-plugins-config-volume\") pod \"payload-processing-e2e-mt-20262e43-8955bfdc4-l5f7m\" (UID: \"d8483bc2-dc0b-470b-b1df-ae39814872fb\") " pod="openshift-ingress/payload-processing-e2e-mt-20262e43-8955bfdc4-l5f7m"
Jul 29 19:49:41.098257 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:41.098166    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bsw2\" (UniqueName: \"kubernetes.io/projected/d8483bc2-dc0b-470b-b1df-ae39814872fb-kube-api-access-6bsw2\") pod \"payload-processing-e2e-mt-20262e43-8955bfdc4-l5f7m\" (UID: \"d8483bc2-dc0b-470b-b1df-ae39814872fb\") " pod="openshift-ingress/payload-processing-e2e-mt-20262e43-8955bfdc4-l5f7m"
Jul 29 19:49:41.098842 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:41.098822    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8483bc2-dc0b-470b-b1df-ae39814872fb-plugins-config-volume\") pod \"payload-processing-e2e-mt-20262e43-8955bfdc4-l5f7m\" (UID: \"d8483bc2-dc0b-470b-b1df-ae39814872fb\") " pod="openshift-ingress/payload-processing-e2e-mt-20262e43-8955bfdc4-l5f7m"
Jul 29 19:49:41.107340 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:41.107321    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bsw2\" (UniqueName: \"kubernetes.io/projected/d8483bc2-dc0b-470b-b1df-ae39814872fb-kube-api-access-6bsw2\") pod \"payload-processing-e2e-mt-20262e43-8955bfdc4-l5f7m\" (UID: \"d8483bc2-dc0b-470b-b1df-ae39814872fb\") " pod="openshift-ingress/payload-processing-e2e-mt-20262e43-8955bfdc4-l5f7m"
Jul 29 19:49:41.194814 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:41.194779    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-20262e43-8955bfdc4-l5f7m"
Jul 29 19:49:41.198933 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:41.198912    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7179fce-31d6-46b8-acae-d1bbccff47fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-20262e43-68fc64c8dd-lfztt\" (UID: \"c7179fce-31d6-46b8-acae-d1bbccff47fb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-20262e43-68fc64c8dd-lfztt"
Jul 29 19:49:41.199048 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:41.198956    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ddjl\" (UniqueName: \"kubernetes.io/projected/c7179fce-31d6-46b8-acae-d1bbccff47fb-kube-api-access-2ddjl\") pod \"payload-pre-processing-e2e-mt-20262e43-68fc64c8dd-lfztt\" (UID: \"c7179fce-31d6-46b8-acae-d1bbccff47fb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-20262e43-68fc64c8dd-lfztt"
Jul 29 19:49:41.199591 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:41.199572    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7179fce-31d6-46b8-acae-d1bbccff47fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-20262e43-68fc64c8dd-lfztt\" (UID: \"c7179fce-31d6-46b8-acae-d1bbccff47fb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-20262e43-68fc64c8dd-lfztt"
Jul 29 19:49:41.207762 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:41.207740    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ddjl\" (UniqueName: \"kubernetes.io/projected/c7179fce-31d6-46b8-acae-d1bbccff47fb-kube-api-access-2ddjl\") pod \"payload-pre-processing-e2e-mt-20262e43-68fc64c8dd-lfztt\" (UID: \"c7179fce-31d6-46b8-acae-d1bbccff47fb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-20262e43-68fc64c8dd-lfztt"
Jul 29 19:49:41.298389 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:41.298358    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-20262e43-68fc64c8dd-lfztt"
Jul 29 19:49:41.322829 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:41.322799    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-20262e43-8955bfdc4-l5f7m"]
Jul 29 19:49:41.325475 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:49:41.325442    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8483bc2_dc0b_470b_b1df_ae39814872fb.slice/crio-c31f23bb2606746c47899ccc4cdfee4c8c84145fb0186bfa11a60ab46bf8f175 WatchSource:0}: Error finding container c31f23bb2606746c47899ccc4cdfee4c8c84145fb0186bfa11a60ab46bf8f175: Status 404 returned error can't find the container with id c31f23bb2606746c47899ccc4cdfee4c8c84145fb0186bfa11a60ab46bf8f175
Jul 29 19:49:41.438047 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:41.438020    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-20262e43-68fc64c8dd-lfztt"]
Jul 29 19:49:41.440009 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:49:41.439974    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7179fce_31d6_46b8_acae_d1bbccff47fb.slice/crio-ba777012e679fe370918b0bd893ce9c953406b850d09394ebcc0620034473a1f WatchSource:0}: Error finding container ba777012e679fe370918b0bd893ce9c953406b850d09394ebcc0620034473a1f: Status 404 returned error can't find the container with id ba777012e679fe370918b0bd893ce9c953406b850d09394ebcc0620034473a1f
Jul 29 19:49:41.492678 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:41.492646    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-20262e43-68fc64c8dd-lfztt" event={"ID":"c7179fce-31d6-46b8-acae-d1bbccff47fb","Type":"ContainerStarted","Data":"ba777012e679fe370918b0bd893ce9c953406b850d09394ebcc0620034473a1f"}
Jul 29 19:49:41.494412 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:41.494385    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6" event={"ID":"69255464-8e45-426c-b73b-0d834090801b","Type":"ContainerStarted","Data":"fb40bdd0f7c90dc017200f8a7116f421697dd3332ab0d58ddaf89e1f16ebb152"}
Jul 29 19:49:41.494542 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:41.494445    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6"
Jul 29 19:49:41.496064 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:41.495997    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-20262e43-8955bfdc4-l5f7m" event={"ID":"d8483bc2-dc0b-470b-b1df-ae39814872fb","Type":"ContainerStarted","Data":"a3355590f82958a6b47d2cf8b06441b8e66b897475da0036e2f0d03bccb0314d"}
Jul 29 19:49:41.496064 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:41.496054    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-20262e43-8955bfdc4-l5f7m" event={"ID":"d8483bc2-dc0b-470b-b1df-ae39814872fb","Type":"ContainerStarted","Data":"c31f23bb2606746c47899ccc4cdfee4c8c84145fb0186bfa11a60ab46bf8f175"}
Jul 29 19:49:41.496233 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:41.496084    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-20262e43-8955bfdc4-l5f7m"
Jul 29 19:49:41.512081 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:41.512040    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6" podStartSLOduration=2.069447964 podStartE2EDuration="2.512025381s" totalImagesPullingTime="442.577417ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:49:41.511394069 +0000 UTC m=+1952.338893129" watchObservedRunningTime="2026-07-29 19:49:41.512025381 +0000 UTC m=+1952.339524439"
Jul 29 19:49:41.528090 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:41.528014    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-20262e43-8955bfdc4-l5f7m" podStartSLOduration=1.528000645 podStartE2EDuration="1.528000645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:49:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:49:41.527776011 +0000 UTC m=+1952.355275071" watchObservedRunningTime="2026-07-29 19:49:41.528000645 +0000 UTC m=+1952.355499703"
Jul 29 19:49:42.501881 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:42.501839    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-20262e43-68fc64c8dd-lfztt" event={"ID":"c7179fce-31d6-46b8-acae-d1bbccff47fb","Type":"ContainerStarted","Data":"80096508bd39f495bd9a5ab3a0fcfc4c78d0d24087880f34f226707dcc4fbff4"}
Jul 29 19:49:43.506308 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:43.506270    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-20262e43-68fc64c8dd-lfztt"
Jul 29 19:49:47.507305 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:47.507275    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6"
Jul 29 19:49:47.526341 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:47.526286    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-20262e43-68fc64c8dd-lfztt" podStartSLOduration=7.526267226 podStartE2EDuration="7.526267226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:49:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:49:42.522664905 +0000 UTC m=+1953.350163982" watchObservedRunningTime="2026-07-29 19:49:47.526267226 +0000 UTC m=+1958.353766285"
Jul 29 19:49:52.504312 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:52.504280    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-20262e43-8955bfdc4-l5f7m"
Jul 29 19:49:54.513157 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:49:54.513124    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-20262e43-68fc64c8dd-lfztt"
Jul 29 19:50:11.125513 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:11.125477    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20262e43-r8nwc"]
Jul 29 19:50:11.129162 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:11.129145    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20262e43-r8nwc"
Jul 29 19:50:11.131886 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:50:11.131849    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-revoke-keys-e2e-mt-20262e43-r8nwc\" is forbidden: User \"system:node:ip-10-0-128-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-128-173.ec2.internal' and this object" podUID="82ff7b55-2303-4f5c-a6d5-b8b93aa050fc" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20262e43-r8nwc"
Jul 29 19:50:11.132721 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:11.132699    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jsq2r\""
Jul 29 19:50:11.142489 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:11.142464    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20262e43-r8nwc"]
Jul 29 19:50:11.269550 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:11.269514    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82ff7b55-2303-4f5c-a6d5-b8b93aa050fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-20262e43-r8nwc\" (UID: \"82ff7b55-2303-4f5c-a6d5-b8b93aa050fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20262e43-r8nwc"
Jul 29 19:50:11.370168 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:11.370132    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82ff7b55-2303-4f5c-a6d5-b8b93aa050fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-20262e43-r8nwc\" (UID: \"82ff7b55-2303-4f5c-a6d5-b8b93aa050fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20262e43-r8nwc"
Jul 29 19:50:11.370734 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:11.370715    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82ff7b55-2303-4f5c-a6d5-b8b93aa050fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-20262e43-r8nwc\" (UID: \"82ff7b55-2303-4f5c-a6d5-b8b93aa050fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20262e43-r8nwc"
Jul 29 19:50:11.438627 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:11.438589    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20262e43-r8nwc"
Jul 29 19:50:11.570077 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:11.570051    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20262e43-r8nwc"]
Jul 29 19:50:11.570677 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:50:11.570656    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ff7b55_2303_4f5c_a6d5_b8b93aa050fc.slice/crio-11de3e799ddc73fc8d18732064877e2ffc08bf1f9d949cb51eda1da6c820f923 WatchSource:0}: Error finding container 11de3e799ddc73fc8d18732064877e2ffc08bf1f9d949cb51eda1da6c820f923: Status 404 returned error can't find the container with id 11de3e799ddc73fc8d18732064877e2ffc08bf1f9d949cb51eda1da6c820f923
Jul 29 19:50:11.641091 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:11.641058    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20262e43-r8nwc" event={"ID":"82ff7b55-2303-4f5c-a6d5-b8b93aa050fc","Type":"ContainerStarted","Data":"11de3e799ddc73fc8d18732064877e2ffc08bf1f9d949cb51eda1da6c820f923"}
Jul 29 19:50:12.647905 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:12.647873    2557 generic.go:356] "Generic (PLEG): container finished" podID="82ff7b55-2303-4f5c-a6d5-b8b93aa050fc" containerID="6eb2bffff4eaa73438927a8eb350822dbc86109d8592ea043a5a355906c8daca" exitCode=0
Jul 29 19:50:12.648302 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:12.647958    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20262e43-r8nwc" event={"ID":"82ff7b55-2303-4f5c-a6d5-b8b93aa050fc","Type":"ContainerDied","Data":"6eb2bffff4eaa73438927a8eb350822dbc86109d8592ea043a5a355906c8daca"}
Jul 29 19:50:13.781884 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:13.781861    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20262e43-r8nwc"
Jul 29 19:50:13.893697 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:13.893669    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82ff7b55-2303-4f5c-a6d5-b8b93aa050fc-maas-api-service-ca\") pod \"82ff7b55-2303-4f5c-a6d5-b8b93aa050fc\" (UID: \"82ff7b55-2303-4f5c-a6d5-b8b93aa050fc\") "
Jul 29 19:50:13.894060 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:13.894038    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ff7b55-2303-4f5c-a6d5-b8b93aa050fc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "82ff7b55-2303-4f5c-a6d5-b8b93aa050fc" (UID: "82ff7b55-2303-4f5c-a6d5-b8b93aa050fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:50:13.994480 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:13.994400    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82ff7b55-2303-4f5c-a6d5-b8b93aa050fc-maas-api-service-ca\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:50:14.658900 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:14.658864    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20262e43-r8nwc" event={"ID":"82ff7b55-2303-4f5c-a6d5-b8b93aa050fc","Type":"ContainerDied","Data":"11de3e799ddc73fc8d18732064877e2ffc08bf1f9d949cb51eda1da6c820f923"}
Jul 29 19:50:14.658900 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:14.658898    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20262e43-r8nwc"
Jul 29 19:50:14.659131 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:14.658937    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11de3e799ddc73fc8d18732064877e2ffc08bf1f9d949cb51eda1da6c820f923"
Jul 29 19:50:21.738166 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:21.738131    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6"]
Jul 29 19:50:21.738655 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:21.738334    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6" podUID="69255464-8e45-426c-b73b-0d834090801b" containerName="maas-api" containerID="cri-o://fb40bdd0f7c90dc017200f8a7116f421697dd3332ab0d58ddaf89e1f16ebb152" gracePeriod=30
Jul 29 19:50:21.874626 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:21.874592    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-20262e43-8955bfdc4-l5f7m"]
Jul 29 19:50:21.875444 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:21.874824    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-20262e43-8955bfdc4-l5f7m" podUID="d8483bc2-dc0b-470b-b1df-ae39814872fb" containerName="payload-processing" containerID="cri-o://a3355590f82958a6b47d2cf8b06441b8e66b897475da0036e2f0d03bccb0314d" gracePeriod=30
Jul 29 19:50:21.883245 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:21.882841    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-20262e43-68fc64c8dd-lfztt"]
Jul 29 19:50:21.883245 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:21.883131    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-20262e43-68fc64c8dd-lfztt" podUID="c7179fce-31d6-46b8-acae-d1bbccff47fb" containerName="payload-pre-processing" containerID="cri-o://80096508bd39f495bd9a5ab3a0fcfc4c78d0d24087880f34f226707dcc4fbff4" gracePeriod=30
Jul 29 19:50:22.008784 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.008744    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6"
Jul 29 19:50:22.041662 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.041618    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20262e43-r8nwc"]
Jul 29 19:50:22.043255 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.043227    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20262e43-r8nwc"]
Jul 29 19:50:22.149280 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.149258    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-20262e43-8955bfdc4-l5f7m"
Jul 29 19:50:22.152182 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.152163    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-20262e43-68fc64c8dd-lfztt"
Jul 29 19:50:22.169079 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.169051    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gz628\" (UniqueName: \"kubernetes.io/projected/69255464-8e45-426c-b73b-0d834090801b-kube-api-access-gz628\") pod \"69255464-8e45-426c-b73b-0d834090801b\" (UID: \"69255464-8e45-426c-b73b-0d834090801b\") "
Jul 29 19:50:22.169267 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.169100    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69255464-8e45-426c-b73b-0d834090801b-maas-api-tls\") pod \"69255464-8e45-426c-b73b-0d834090801b\" (UID: \"69255464-8e45-426c-b73b-0d834090801b\") "
Jul 29 19:50:22.169267 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.169195    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69255464-8e45-426c-b73b-0d834090801b-openshift-service-ca\") pod \"69255464-8e45-426c-b73b-0d834090801b\" (UID: \"69255464-8e45-426c-b73b-0d834090801b\") "
Jul 29 19:50:22.169951 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.169720    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69255464-8e45-426c-b73b-0d834090801b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "69255464-8e45-426c-b73b-0d834090801b" (UID: "69255464-8e45-426c-b73b-0d834090801b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:50:22.171521 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.171496    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69255464-8e45-426c-b73b-0d834090801b-kube-api-access-gz628" (OuterVolumeSpecName: "kube-api-access-gz628") pod "69255464-8e45-426c-b73b-0d834090801b" (UID: "69255464-8e45-426c-b73b-0d834090801b"). InnerVolumeSpecName "kube-api-access-gz628". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:50:22.171627 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.171531    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69255464-8e45-426c-b73b-0d834090801b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "69255464-8e45-426c-b73b-0d834090801b" (UID: "69255464-8e45-426c-b73b-0d834090801b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:50:22.270107 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.270032    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7179fce-31d6-46b8-acae-d1bbccff47fb-plugins-config-volume\") pod \"c7179fce-31d6-46b8-acae-d1bbccff47fb\" (UID: \"c7179fce-31d6-46b8-acae-d1bbccff47fb\") "
Jul 29 19:50:22.270107 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.270073    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bsw2\" (UniqueName: \"kubernetes.io/projected/d8483bc2-dc0b-470b-b1df-ae39814872fb-kube-api-access-6bsw2\") pod \"d8483bc2-dc0b-470b-b1df-ae39814872fb\" (UID: \"d8483bc2-dc0b-470b-b1df-ae39814872fb\") "
Jul 29 19:50:22.270309 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.270117    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8483bc2-dc0b-470b-b1df-ae39814872fb-plugins-config-volume\") pod \"d8483bc2-dc0b-470b-b1df-ae39814872fb\" (UID: \"d8483bc2-dc0b-470b-b1df-ae39814872fb\") "
Jul 29 19:50:22.270309 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.270149    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ddjl\" (UniqueName: \"kubernetes.io/projected/c7179fce-31d6-46b8-acae-d1bbccff47fb-kube-api-access-2ddjl\") pod \"c7179fce-31d6-46b8-acae-d1bbccff47fb\" (UID: \"c7179fce-31d6-46b8-acae-d1bbccff47fb\") "
Jul 29 19:50:22.270385 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.270367    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69255464-8e45-426c-b73b-0d834090801b-maas-api-tls\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:50:22.270419 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.270387    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69255464-8e45-426c-b73b-0d834090801b-openshift-service-ca\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:50:22.270419 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.270401    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz628\" (UniqueName: \"kubernetes.io/projected/69255464-8e45-426c-b73b-0d834090801b-kube-api-access-gz628\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:50:22.270493 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.270450    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8483bc2-dc0b-470b-b1df-ae39814872fb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d8483bc2-dc0b-470b-b1df-ae39814872fb" (UID: "d8483bc2-dc0b-470b-b1df-ae39814872fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:50:22.270493 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.270450    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7179fce-31d6-46b8-acae-d1bbccff47fb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c7179fce-31d6-46b8-acae-d1bbccff47fb" (UID: "c7179fce-31d6-46b8-acae-d1bbccff47fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:50:22.272167 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.272145    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7179fce-31d6-46b8-acae-d1bbccff47fb-kube-api-access-2ddjl" (OuterVolumeSpecName: "kube-api-access-2ddjl") pod "c7179fce-31d6-46b8-acae-d1bbccff47fb" (UID: "c7179fce-31d6-46b8-acae-d1bbccff47fb"). InnerVolumeSpecName "kube-api-access-2ddjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:50:22.272238 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.272208    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8483bc2-dc0b-470b-b1df-ae39814872fb-kube-api-access-6bsw2" (OuterVolumeSpecName: "kube-api-access-6bsw2") pod "d8483bc2-dc0b-470b-b1df-ae39814872fb" (UID: "d8483bc2-dc0b-470b-b1df-ae39814872fb"). InnerVolumeSpecName "kube-api-access-6bsw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:50:22.371192 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.371156    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bsw2\" (UniqueName: \"kubernetes.io/projected/d8483bc2-dc0b-470b-b1df-ae39814872fb-kube-api-access-6bsw2\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:50:22.371192 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.371187    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8483bc2-dc0b-470b-b1df-ae39814872fb-plugins-config-volume\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:50:22.371192 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.371198    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ddjl\" (UniqueName: \"kubernetes.io/projected/c7179fce-31d6-46b8-acae-d1bbccff47fb-kube-api-access-2ddjl\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:50:22.371412 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.371207    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7179fce-31d6-46b8-acae-d1bbccff47fb-plugins-config-volume\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:50:22.696095 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.696047    2557 generic.go:356] "Generic (PLEG): container finished" podID="c7179fce-31d6-46b8-acae-d1bbccff47fb" containerID="80096508bd39f495bd9a5ab3a0fcfc4c78d0d24087880f34f226707dcc4fbff4" exitCode=0
Jul 29 19:50:22.696095 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.696082    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-20262e43-68fc64c8dd-lfztt" event={"ID":"c7179fce-31d6-46b8-acae-d1bbccff47fb","Type":"ContainerDied","Data":"80096508bd39f495bd9a5ab3a0fcfc4c78d0d24087880f34f226707dcc4fbff4"}
Jul 29 19:50:22.696337 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.696101    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-20262e43-68fc64c8dd-lfztt"
Jul 29 19:50:22.696337 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.696137    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-20262e43-68fc64c8dd-lfztt" event={"ID":"c7179fce-31d6-46b8-acae-d1bbccff47fb","Type":"ContainerDied","Data":"ba777012e679fe370918b0bd893ce9c953406b850d09394ebcc0620034473a1f"}
Jul 29 19:50:22.696337 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.696165    2557 scope.go:117] "RemoveContainer" containerID="80096508bd39f495bd9a5ab3a0fcfc4c78d0d24087880f34f226707dcc4fbff4"
Jul 29 19:50:22.697472 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.697450    2557 generic.go:356] "Generic (PLEG): container finished" podID="69255464-8e45-426c-b73b-0d834090801b" containerID="fb40bdd0f7c90dc017200f8a7116f421697dd3332ab0d58ddaf89e1f16ebb152" exitCode=0
Jul 29 19:50:22.697577 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.697475    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6" event={"ID":"69255464-8e45-426c-b73b-0d834090801b","Type":"ContainerDied","Data":"fb40bdd0f7c90dc017200f8a7116f421697dd3332ab0d58ddaf89e1f16ebb152"}
Jul 29 19:50:22.697577 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.697490    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6"
Jul 29 19:50:22.697577 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.697522    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6" event={"ID":"69255464-8e45-426c-b73b-0d834090801b","Type":"ContainerDied","Data":"88f198a6622873f4701442bd064cd677bd4b0533440b7c99ab6b0eca4e2b3d34"}
Jul 29 19:50:22.698809 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.698786    2557 generic.go:356] "Generic (PLEG): container finished" podID="d8483bc2-dc0b-470b-b1df-ae39814872fb" containerID="a3355590f82958a6b47d2cf8b06441b8e66b897475da0036e2f0d03bccb0314d" exitCode=0
Jul 29 19:50:22.698905 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.698826    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-20262e43-8955bfdc4-l5f7m"
Jul 29 19:50:22.698905 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.698856    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-20262e43-8955bfdc4-l5f7m" event={"ID":"d8483bc2-dc0b-470b-b1df-ae39814872fb","Type":"ContainerDied","Data":"a3355590f82958a6b47d2cf8b06441b8e66b897475da0036e2f0d03bccb0314d"}
Jul 29 19:50:22.698905 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.698887    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-20262e43-8955bfdc4-l5f7m" event={"ID":"d8483bc2-dc0b-470b-b1df-ae39814872fb","Type":"ContainerDied","Data":"c31f23bb2606746c47899ccc4cdfee4c8c84145fb0186bfa11a60ab46bf8f175"}
Jul 29 19:50:22.706065 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.706041    2557 scope.go:117] "RemoveContainer" containerID="80096508bd39f495bd9a5ab3a0fcfc4c78d0d24087880f34f226707dcc4fbff4"
Jul 29 19:50:22.706377 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:50:22.706353    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80096508bd39f495bd9a5ab3a0fcfc4c78d0d24087880f34f226707dcc4fbff4\": container with ID starting with 80096508bd39f495bd9a5ab3a0fcfc4c78d0d24087880f34f226707dcc4fbff4 not found: ID does not exist" containerID="80096508bd39f495bd9a5ab3a0fcfc4c78d0d24087880f34f226707dcc4fbff4"
Jul 29 19:50:22.706442 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.706387    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80096508bd39f495bd9a5ab3a0fcfc4c78d0d24087880f34f226707dcc4fbff4"} err="failed to get container status \"80096508bd39f495bd9a5ab3a0fcfc4c78d0d24087880f34f226707dcc4fbff4\": rpc error: code = NotFound desc = could not find container \"80096508bd39f495bd9a5ab3a0fcfc4c78d0d24087880f34f226707dcc4fbff4\": container with ID starting with 80096508bd39f495bd9a5ab3a0fcfc4c78d0d24087880f34f226707dcc4fbff4 not found: ID does not exist"
Jul 29 19:50:22.706442 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.706409    2557 scope.go:117] "RemoveContainer" containerID="fb40bdd0f7c90dc017200f8a7116f421697dd3332ab0d58ddaf89e1f16ebb152"
Jul 29 19:50:22.715059 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.715042    2557 scope.go:117] "RemoveContainer" containerID="fb40bdd0f7c90dc017200f8a7116f421697dd3332ab0d58ddaf89e1f16ebb152"
Jul 29 19:50:22.715314 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:50:22.715295    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb40bdd0f7c90dc017200f8a7116f421697dd3332ab0d58ddaf89e1f16ebb152\": container with ID starting with fb40bdd0f7c90dc017200f8a7116f421697dd3332ab0d58ddaf89e1f16ebb152 not found: ID does not exist" containerID="fb40bdd0f7c90dc017200f8a7116f421697dd3332ab0d58ddaf89e1f16ebb152"
Jul 29 19:50:22.715366 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.715324    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb40bdd0f7c90dc017200f8a7116f421697dd3332ab0d58ddaf89e1f16ebb152"} err="failed to get container status \"fb40bdd0f7c90dc017200f8a7116f421697dd3332ab0d58ddaf89e1f16ebb152\": rpc error: code = NotFound desc = could not find container \"fb40bdd0f7c90dc017200f8a7116f421697dd3332ab0d58ddaf89e1f16ebb152\": container with ID starting with fb40bdd0f7c90dc017200f8a7116f421697dd3332ab0d58ddaf89e1f16ebb152 not found: ID does not exist"
Jul 29 19:50:22.715366 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.715341    2557 scope.go:117] "RemoveContainer" containerID="a3355590f82958a6b47d2cf8b06441b8e66b897475da0036e2f0d03bccb0314d"
Jul 29 19:50:22.724742 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.724654    2557 scope.go:117] "RemoveContainer" containerID="a3355590f82958a6b47d2cf8b06441b8e66b897475da0036e2f0d03bccb0314d"
Jul 29 19:50:22.725152 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:50:22.725092    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3355590f82958a6b47d2cf8b06441b8e66b897475da0036e2f0d03bccb0314d\": container with ID starting with a3355590f82958a6b47d2cf8b06441b8e66b897475da0036e2f0d03bccb0314d not found: ID does not exist" containerID="a3355590f82958a6b47d2cf8b06441b8e66b897475da0036e2f0d03bccb0314d"
Jul 29 19:50:22.725152 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.725121    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3355590f82958a6b47d2cf8b06441b8e66b897475da0036e2f0d03bccb0314d"} err="failed to get container status \"a3355590f82958a6b47d2cf8b06441b8e66b897475da0036e2f0d03bccb0314d\": rpc error: code = NotFound desc = could not find container \"a3355590f82958a6b47d2cf8b06441b8e66b897475da0036e2f0d03bccb0314d\": container with ID starting with a3355590f82958a6b47d2cf8b06441b8e66b897475da0036e2f0d03bccb0314d not found: ID does not exist"
Jul 29 19:50:22.726690 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.726671    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-20262e43-68fc64c8dd-lfztt"]
Jul 29 19:50:22.732015 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.731995    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-20262e43-68fc64c8dd-lfztt"]
Jul 29 19:50:22.743079 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.743058    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6"]
Jul 29 19:50:22.746937 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.746913    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-20262e43-5d844fcff9-4nbb6"]
Jul 29 19:50:22.757550 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.757528    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-20262e43-8955bfdc4-l5f7m"]
Jul 29 19:50:22.760984 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:22.760953    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-20262e43-8955bfdc4-l5f7m"]
Jul 29 19:50:23.678638 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:23.678604    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69255464-8e45-426c-b73b-0d834090801b" path="/var/lib/kubelet/pods/69255464-8e45-426c-b73b-0d834090801b/volumes"
Jul 29 19:50:23.679004 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:23.678988    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82ff7b55-2303-4f5c-a6d5-b8b93aa050fc" path="/var/lib/kubelet/pods/82ff7b55-2303-4f5c-a6d5-b8b93aa050fc/volumes"
Jul 29 19:50:23.679350 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:23.679337    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7179fce-31d6-46b8-acae-d1bbccff47fb" path="/var/lib/kubelet/pods/c7179fce-31d6-46b8-acae-d1bbccff47fb/volumes"
Jul 29 19:50:23.679690 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:23.679676    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8483bc2-dc0b-470b-b1df-ae39814872fb" path="/var/lib/kubelet/pods/d8483bc2-dc0b-470b-b1df-ae39814872fb/volumes"
Jul 29 19:50:53.118165 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.118130    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr"]
Jul 29 19:50:53.120630 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.118607    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7179fce-31d6-46b8-acae-d1bbccff47fb" containerName="payload-pre-processing"
Jul 29 19:50:53.120630 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.118622    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7179fce-31d6-46b8-acae-d1bbccff47fb" containerName="payload-pre-processing"
Jul 29 19:50:53.120630 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.118639    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82ff7b55-2303-4f5c-a6d5-b8b93aa050fc" containerName="revoke-keys"
Jul 29 19:50:53.120630 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.118649    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="82ff7b55-2303-4f5c-a6d5-b8b93aa050fc" containerName="revoke-keys"
Jul 29 19:50:53.120630 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.118679    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69255464-8e45-426c-b73b-0d834090801b" containerName="maas-api"
Jul 29 19:50:53.120630 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.118688    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="69255464-8e45-426c-b73b-0d834090801b" containerName="maas-api"
Jul 29 19:50:53.120630 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.118697    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8483bc2-dc0b-470b-b1df-ae39814872fb" containerName="payload-processing"
Jul 29 19:50:53.120630 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.118706    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8483bc2-dc0b-470b-b1df-ae39814872fb" containerName="payload-processing"
Jul 29 19:50:53.120630 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.118785    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="82ff7b55-2303-4f5c-a6d5-b8b93aa050fc" containerName="revoke-keys"
Jul 29 19:50:53.120630 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.118798    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7179fce-31d6-46b8-acae-d1bbccff47fb" containerName="payload-pre-processing"
Jul 29 19:50:53.120630 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.118807    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="69255464-8e45-426c-b73b-0d834090801b" containerName="maas-api"
Jul 29 19:50:53.120630 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.118818    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8483bc2-dc0b-470b-b1df-ae39814872fb" containerName="payload-processing"
Jul 29 19:50:53.121953 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.121936    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr"
Jul 29 19:50:53.125120 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.125095    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2d31776a-serving-cert\""
Jul 29 19:50:53.125267 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.125135    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4dw6s\""
Jul 29 19:50:53.134079 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.134014    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr"]
Jul 29 19:50:53.242213 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.242180    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b34c407-bf77-44f7-bd80-f066d27686d6-maas-api-tls\") pod \"maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr\" (UID: \"0b34c407-bf77-44f7-bd80-f066d27686d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr"
Jul 29 19:50:53.242380 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.242237    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b34c407-bf77-44f7-bd80-f066d27686d6-openshift-service-ca\") pod \"maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr\" (UID: \"0b34c407-bf77-44f7-bd80-f066d27686d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr"
Jul 29 19:50:53.242380 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.242293    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b27cq\" (UniqueName: \"kubernetes.io/projected/0b34c407-bf77-44f7-bd80-f066d27686d6-kube-api-access-b27cq\") pod \"maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr\" (UID: \"0b34c407-bf77-44f7-bd80-f066d27686d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr"
Jul 29 19:50:53.343844 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.343801    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b34c407-bf77-44f7-bd80-f066d27686d6-maas-api-tls\") pod \"maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr\" (UID: \"0b34c407-bf77-44f7-bd80-f066d27686d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr"
Jul 29 19:50:53.344036 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.343875    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b34c407-bf77-44f7-bd80-f066d27686d6-openshift-service-ca\") pod \"maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr\" (UID: \"0b34c407-bf77-44f7-bd80-f066d27686d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr"
Jul 29 19:50:53.344036 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.343906    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b27cq\" (UniqueName: \"kubernetes.io/projected/0b34c407-bf77-44f7-bd80-f066d27686d6-kube-api-access-b27cq\") pod \"maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr\" (UID: \"0b34c407-bf77-44f7-bd80-f066d27686d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr"
Jul 29 19:50:53.344036 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:50:53.343940    2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-2d31776a-serving-cert: secret "maas-api-e2e-mt-2d31776a-serving-cert" not found
Jul 29 19:50:53.344179 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:50:53.344053    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b34c407-bf77-44f7-bd80-f066d27686d6-maas-api-tls podName:0b34c407-bf77-44f7-bd80-f066d27686d6 nodeName:}" failed. No retries permitted until 2026-07-29 19:50:53.844035818 +0000 UTC m=+2024.671534855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0b34c407-bf77-44f7-bd80-f066d27686d6-maas-api-tls") pod "maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr" (UID: "0b34c407-bf77-44f7-bd80-f066d27686d6") : secret "maas-api-e2e-mt-2d31776a-serving-cert" not found
Jul 29 19:50:53.344634 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.344607    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b34c407-bf77-44f7-bd80-f066d27686d6-openshift-service-ca\") pod \"maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr\" (UID: \"0b34c407-bf77-44f7-bd80-f066d27686d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr"
Jul 29 19:50:53.358889 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.358859    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b27cq\" (UniqueName: \"kubernetes.io/projected/0b34c407-bf77-44f7-bd80-f066d27686d6-kube-api-access-b27cq\") pod \"maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr\" (UID: \"0b34c407-bf77-44f7-bd80-f066d27686d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr"
Jul 29 19:50:53.848211 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.848180    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b34c407-bf77-44f7-bd80-f066d27686d6-maas-api-tls\") pod \"maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr\" (UID: \"0b34c407-bf77-44f7-bd80-f066d27686d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr"
Jul 29 19:50:53.850573 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.850551    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b34c407-bf77-44f7-bd80-f066d27686d6-maas-api-tls\") pod \"maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr\" (UID: \"0b34c407-bf77-44f7-bd80-f066d27686d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr"
Jul 29 19:50:53.892552 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.892518    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2d31776a-5d864dcb74-bkz9z"]
Jul 29 19:50:53.895974 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.895949    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2d31776a-5d864dcb74-bkz9z"
Jul 29 19:50:53.899580 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.899561    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2d31776a-dockercfg-d8fvp\""
Jul 29 19:50:53.899689 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.899602    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2d31776a\""
Jul 29 19:50:53.907756 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.907732    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2d31776a-5d864dcb74-bkz9z"]
Jul 29 19:50:53.948881 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.948838    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5316aa0-e7b8-4955-b412-2f2673bd007f-plugins-config-volume\") pod \"payload-processing-e2e-mt-2d31776a-5d864dcb74-bkz9z\" (UID: \"a5316aa0-e7b8-4955-b412-2f2673bd007f\") " pod="openshift-ingress/payload-processing-e2e-mt-2d31776a-5d864dcb74-bkz9z"
Jul 29 19:50:53.948881 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.948893    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlz4j\" (UniqueName: \"kubernetes.io/projected/a5316aa0-e7b8-4955-b412-2f2673bd007f-kube-api-access-qlz4j\") pod \"payload-processing-e2e-mt-2d31776a-5d864dcb74-bkz9z\" (UID: \"a5316aa0-e7b8-4955-b412-2f2673bd007f\") " pod="openshift-ingress/payload-processing-e2e-mt-2d31776a-5d864dcb74-bkz9z"
Jul 29 19:50:53.971188 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.971153    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2d31776a-5ddb44597b-m7wqb"]
Jul 29 19:50:53.974317 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.974302    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2d31776a-5ddb44597b-m7wqb"
Jul 29 19:50:53.988741 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:53.988715    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2d31776a-5ddb44597b-m7wqb"]
Jul 29 19:50:54.032337 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:54.032308    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr"
Jul 29 19:50:54.049588 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:54.049553    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxvb9\" (UniqueName: \"kubernetes.io/projected/5a758cfe-7d56-4c39-be42-6a7dfad9bec3-kube-api-access-dxvb9\") pod \"payload-pre-processing-e2e-mt-2d31776a-5ddb44597b-m7wqb\" (UID: \"5a758cfe-7d56-4c39-be42-6a7dfad9bec3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2d31776a-5ddb44597b-m7wqb"
Jul 29 19:50:54.049711 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:54.049604    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5316aa0-e7b8-4955-b412-2f2673bd007f-plugins-config-volume\") pod \"payload-processing-e2e-mt-2d31776a-5d864dcb74-bkz9z\" (UID: \"a5316aa0-e7b8-4955-b412-2f2673bd007f\") " pod="openshift-ingress/payload-processing-e2e-mt-2d31776a-5d864dcb74-bkz9z"
Jul 29 19:50:54.049711 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:54.049651    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlz4j\" (UniqueName: \"kubernetes.io/projected/a5316aa0-e7b8-4955-b412-2f2673bd007f-kube-api-access-qlz4j\") pod \"payload-processing-e2e-mt-2d31776a-5d864dcb74-bkz9z\" (UID: \"a5316aa0-e7b8-4955-b412-2f2673bd007f\") " pod="openshift-ingress/payload-processing-e2e-mt-2d31776a-5d864dcb74-bkz9z"
Jul 29 19:50:54.049812 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:54.049745    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a758cfe-7d56-4c39-be42-6a7dfad9bec3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2d31776a-5ddb44597b-m7wqb\" (UID: \"5a758cfe-7d56-4c39-be42-6a7dfad9bec3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2d31776a-5ddb44597b-m7wqb"
Jul 29 19:50:54.050526 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:54.050496    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5316aa0-e7b8-4955-b412-2f2673bd007f-plugins-config-volume\") pod \"payload-processing-e2e-mt-2d31776a-5d864dcb74-bkz9z\" (UID: \"a5316aa0-e7b8-4955-b412-2f2673bd007f\") " pod="openshift-ingress/payload-processing-e2e-mt-2d31776a-5d864dcb74-bkz9z"
Jul 29 19:50:54.058920 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:54.058899    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlz4j\" (UniqueName: \"kubernetes.io/projected/a5316aa0-e7b8-4955-b412-2f2673bd007f-kube-api-access-qlz4j\") pod \"payload-processing-e2e-mt-2d31776a-5d864dcb74-bkz9z\" (UID: \"a5316aa0-e7b8-4955-b412-2f2673bd007f\") " pod="openshift-ingress/payload-processing-e2e-mt-2d31776a-5d864dcb74-bkz9z"
Jul 29 19:50:54.150751 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:54.150718    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a758cfe-7d56-4c39-be42-6a7dfad9bec3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2d31776a-5ddb44597b-m7wqb\" (UID: \"5a758cfe-7d56-4c39-be42-6a7dfad9bec3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2d31776a-5ddb44597b-m7wqb"
Jul 29 19:50:54.151238 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:54.150772    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxvb9\" (UniqueName: \"kubernetes.io/projected/5a758cfe-7d56-4c39-be42-6a7dfad9bec3-kube-api-access-dxvb9\") pod \"payload-pre-processing-e2e-mt-2d31776a-5ddb44597b-m7wqb\" (UID: \"5a758cfe-7d56-4c39-be42-6a7dfad9bec3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2d31776a-5ddb44597b-m7wqb"
Jul 29 19:50:54.151555 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:54.151532    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a758cfe-7d56-4c39-be42-6a7dfad9bec3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2d31776a-5ddb44597b-m7wqb\" (UID: \"5a758cfe-7d56-4c39-be42-6a7dfad9bec3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2d31776a-5ddb44597b-m7wqb"
Jul 29 19:50:54.162384 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:54.162361    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxvb9\" (UniqueName: \"kubernetes.io/projected/5a758cfe-7d56-4c39-be42-6a7dfad9bec3-kube-api-access-dxvb9\") pod \"payload-pre-processing-e2e-mt-2d31776a-5ddb44597b-m7wqb\" (UID: \"5a758cfe-7d56-4c39-be42-6a7dfad9bec3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2d31776a-5ddb44597b-m7wqb"
Jul 29 19:50:54.201342 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:54.201317    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr"]
Jul 29 19:50:54.203692 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:50:54.203665    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b34c407_bf77_44f7_bd80_f066d27686d6.slice/crio-77f9e9ff43d99cf31a85689f9dfe182a5888fb09019a79f95a2cd84d7ea36b8c WatchSource:0}: Error finding container 77f9e9ff43d99cf31a85689f9dfe182a5888fb09019a79f95a2cd84d7ea36b8c: Status 404 returned error can't find the container with id 77f9e9ff43d99cf31a85689f9dfe182a5888fb09019a79f95a2cd84d7ea36b8c
Jul 29 19:50:54.205224 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:54.205205    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 19:50:54.206827 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:54.206807    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2d31776a-5d864dcb74-bkz9z"
Jul 29 19:50:54.284515 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:54.284486    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2d31776a-5ddb44597b-m7wqb"
Jul 29 19:50:54.339326 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:54.339293    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2d31776a-5d864dcb74-bkz9z"]
Jul 29 19:50:54.343236 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:50:54.343198    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5316aa0_e7b8_4955_b412_2f2673bd007f.slice/crio-dab49e5d74a5cbb09882a7abf56a4977ed9ad95a90ee66a536340c82fc825f95 WatchSource:0}: Error finding container dab49e5d74a5cbb09882a7abf56a4977ed9ad95a90ee66a536340c82fc825f95: Status 404 returned error can't find the container with id dab49e5d74a5cbb09882a7abf56a4977ed9ad95a90ee66a536340c82fc825f95
Jul 29 19:50:54.421353 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:54.421331    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2d31776a-5ddb44597b-m7wqb"]
Jul 29 19:50:54.423557 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:50:54.423525    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a758cfe_7d56_4c39_be42_6a7dfad9bec3.slice/crio-7589ed682a2acdbc1e43b4858486e76cb92c08017c7ec89fc03e291e957d6e44 WatchSource:0}: Error finding container 7589ed682a2acdbc1e43b4858486e76cb92c08017c7ec89fc03e291e957d6e44: Status 404 returned error can't find the container with id 7589ed682a2acdbc1e43b4858486e76cb92c08017c7ec89fc03e291e957d6e44
Jul 29 19:50:54.856476 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:54.856432    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2d31776a-5d864dcb74-bkz9z" event={"ID":"a5316aa0-e7b8-4955-b412-2f2673bd007f","Type":"ContainerStarted","Data":"bfdc48514eec05c6fbf62b6b76639bf1719d24d729ab14130f46f679bc3501f3"}
Jul 29 19:50:54.856662 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:54.856493    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2d31776a-5d864dcb74-bkz9z" event={"ID":"a5316aa0-e7b8-4955-b412-2f2673bd007f","Type":"ContainerStarted","Data":"dab49e5d74a5cbb09882a7abf56a4977ed9ad95a90ee66a536340c82fc825f95"}
Jul 29 19:50:54.856662 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:54.856535    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-2d31776a-5d864dcb74-bkz9z"
Jul 29 19:50:54.858254 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:54.858226    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr" event={"ID":"0b34c407-bf77-44f7-bd80-f066d27686d6","Type":"ContainerStarted","Data":"1d33b532554b4860017016ca77107820df2dda3bdad3f42534ba0590cce506d8"}
Jul 29 19:50:54.858386 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:54.858284    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr"
Jul 29 19:50:54.858386 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:54.858309    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr" event={"ID":"0b34c407-bf77-44f7-bd80-f066d27686d6","Type":"ContainerStarted","Data":"77f9e9ff43d99cf31a85689f9dfe182a5888fb09019a79f95a2cd84d7ea36b8c"}
Jul 29 19:50:54.859775 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:54.859726    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2d31776a-5ddb44597b-m7wqb" event={"ID":"5a758cfe-7d56-4c39-be42-6a7dfad9bec3","Type":"ContainerStarted","Data":"28080d5f8fc303e59a47f693e1160e9523f2094dbdce15ad215437db7b77515e"}
Jul 29 19:50:54.859775 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:54.859763    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2d31776a-5ddb44597b-m7wqb" event={"ID":"5a758cfe-7d56-4c39-be42-6a7dfad9bec3","Type":"ContainerStarted","Data":"7589ed682a2acdbc1e43b4858486e76cb92c08017c7ec89fc03e291e957d6e44"}
Jul 29 19:50:54.860206 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:54.859801    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2d31776a-5ddb44597b-m7wqb"
Jul 29 19:50:54.877856 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:54.877816    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-2d31776a-5d864dcb74-bkz9z" podStartSLOduration=1.877803208 podStartE2EDuration="1.877803208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:50:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:50:54.877113726 +0000 UTC m=+2025.704612786" watchObservedRunningTime="2026-07-29 19:50:54.877803208 +0000 UTC m=+2025.705302315"
Jul 29 19:50:54.900655 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:54.900613    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr" podStartSLOduration=1.353628545 podStartE2EDuration="1.900599323s" totalImagesPullingTime="546.970778ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:50:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:50:54.899150568 +0000 UTC m=+2025.726649627" watchObservedRunningTime="2026-07-29 19:50:54.900599323 +0000 UTC m=+2025.728098401"
Jul 29 19:50:54.917946 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:50:54.917903    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-2d31776a-5ddb44597b-m7wqb" podStartSLOduration=1.9178915010000002 podStartE2EDuration="1.917891501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:50:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:50:54.917218973 +0000 UTC m=+2025.744718031" watchObservedRunningTime="2026-07-29 19:50:54.917891501 +0000 UTC m=+2025.745390559"
Jul 29 19:51:00.870633 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:00.870594    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr"
Jul 29 19:51:05.867863 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:05.867827    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-2d31776a-5d864dcb74-bkz9z"
Jul 29 19:51:05.868332 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:05.867922    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2d31776a-5ddb44597b-m7wqb"
Jul 29 19:51:21.934149 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:21.934108    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf"]
Jul 29 19:51:21.938080 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:21.938061    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf"
Jul 29 19:51:21.941519 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:21.941498    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2d31776a\"/\"openshift-service-ca.crt\""
Jul 29 19:51:21.942793 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:21.942770    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2d31776a\"/\"e2e-lifecycle-model-2d31776a-kserve-self-signed-certs\""
Jul 29 19:51:21.942935 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:21.942918    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2d31776a\"/\"default-dockercfg-m8p24\""
Jul 29 19:51:21.943223 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:21.943208    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2d31776a\"/\"kube-root-ca.crt\""
Jul 29 19:51:21.949709 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:21.949688    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf"]
Jul 29 19:51:21.995137 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:21.995110    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbvgp\" (UniqueName: \"kubernetes.io/projected/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-kube-api-access-pbvgp\") pod \"e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf\" (UID: \"2e7acf16-796d-4d9f-97bc-0af0ea9f938e\") " pod="ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf"
Jul 29 19:51:21.995298 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:21.995143    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-tmp-dir\") pod \"e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf\" (UID: \"2e7acf16-796d-4d9f-97bc-0af0ea9f938e\") " pod="ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf"
Jul 29 19:51:21.995298 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:21.995175    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-model-cache\") pod \"e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf\" (UID: \"2e7acf16-796d-4d9f-97bc-0af0ea9f938e\") " pod="ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf"
Jul 29 19:51:21.995298 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:21.995246    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-home\") pod \"e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf\" (UID: \"2e7acf16-796d-4d9f-97bc-0af0ea9f938e\") " pod="ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf"
Jul 29 19:51:21.995298 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:21.995294    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-dshm\") pod \"e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf\" (UID: \"2e7acf16-796d-4d9f-97bc-0af0ea9f938e\") " pod="ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf"
Jul 29 19:51:21.995456 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:21.995311    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-tls-certs\") pod \"e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf\" (UID: \"2e7acf16-796d-4d9f-97bc-0af0ea9f938e\") " pod="ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf"
Jul 29 19:51:22.095935 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:22.095904    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbvgp\" (UniqueName: \"kubernetes.io/projected/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-kube-api-access-pbvgp\") pod \"e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf\" (UID: \"2e7acf16-796d-4d9f-97bc-0af0ea9f938e\") " pod="ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf"
Jul 29 19:51:22.095935 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:22.095940    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-tmp-dir\") pod \"e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf\" (UID: \"2e7acf16-796d-4d9f-97bc-0af0ea9f938e\") " pod="ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf"
Jul 29 19:51:22.096204 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:22.095986    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-model-cache\") pod \"e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf\" (UID: \"2e7acf16-796d-4d9f-97bc-0af0ea9f938e\") " pod="ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf"
Jul 29 19:51:22.096204 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:22.096017    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-home\") pod \"e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf\" (UID: \"2e7acf16-796d-4d9f-97bc-0af0ea9f938e\") " pod="ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf"
Jul 29 19:51:22.096204 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:22.096044    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-dshm\") pod \"e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf\" (UID: \"2e7acf16-796d-4d9f-97bc-0af0ea9f938e\") " pod="ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf"
Jul 29 19:51:22.096204 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:22.096059    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-tls-certs\") pod \"e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf\" (UID: \"2e7acf16-796d-4d9f-97bc-0af0ea9f938e\") " pod="ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf"
Jul 29 19:51:22.096426 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:22.096376    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-tmp-dir\") pod \"e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf\" (UID: \"2e7acf16-796d-4d9f-97bc-0af0ea9f938e\") " pod="ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf"
Jul 29 19:51:22.096478 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:22.096437    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-home\") pod \"e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf\" (UID: \"2e7acf16-796d-4d9f-97bc-0af0ea9f938e\") " pod="ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf"
Jul 29 19:51:22.096540 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:22.096508    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-model-cache\") pod \"e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf\" (UID: \"2e7acf16-796d-4d9f-97bc-0af0ea9f938e\") " pod="ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf"
Jul 29 19:51:22.098371 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:22.098347    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-dshm\") pod \"e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf\" (UID: \"2e7acf16-796d-4d9f-97bc-0af0ea9f938e\") " pod="ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf"
Jul 29 19:51:22.098744 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:22.098724    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-tls-certs\") pod \"e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf\" (UID: \"2e7acf16-796d-4d9f-97bc-0af0ea9f938e\") " pod="ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf"
Jul 29 19:51:22.104083 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:22.104065    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbvgp\" (UniqueName: \"kubernetes.io/projected/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-kube-api-access-pbvgp\") pod \"e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf\" (UID: \"2e7acf16-796d-4d9f-97bc-0af0ea9f938e\") " pod="ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf"
Jul 29 19:51:22.247598 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:22.247509    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf"
Jul 29 19:51:22.382129 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:22.382101    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf"]
Jul 29 19:51:22.384320 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:51:22.384292    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e7acf16_796d_4d9f_97bc_0af0ea9f938e.slice/crio-d95e42b4529b6fcc1d71e9a62989c1d8c641342585877f60842878bf7971fa2e WatchSource:0}: Error finding container d95e42b4529b6fcc1d71e9a62989c1d8c641342585877f60842878bf7971fa2e: Status 404 returned error can't find the container with id d95e42b4529b6fcc1d71e9a62989c1d8c641342585877f60842878bf7971fa2e
Jul 29 19:51:22.997884 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:22.997845    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf" event={"ID":"2e7acf16-796d-4d9f-97bc-0af0ea9f938e","Type":"ContainerStarted","Data":"653799dd4575567d615b6b03c66a34e3dab4cef26cb7f01ea005f7a1771ecfc3"}
Jul 29 19:51:22.997884 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:22.997889    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf" event={"ID":"2e7acf16-796d-4d9f-97bc-0af0ea9f938e","Type":"ContainerStarted","Data":"d95e42b4529b6fcc1d71e9a62989c1d8c641342585877f60842878bf7971fa2e"}
Jul 29 19:51:23.018206 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:23.018159    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf" podStartSLOduration=2.018144727 podStartE2EDuration="2.018144727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:51:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:51:23.016049414 +0000 UTC m=+2053.843548491" watchObservedRunningTime="2026-07-29 19:51:23.018144727 +0000 UTC m=+2053.845643784"
Jul 29 19:51:32.248028 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:32.247928    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf"
Jul 29 19:51:32.248028 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:32.248003    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf"
Jul 29 19:51:32.260528 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:32.260506    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf"
Jul 29 19:51:33.055837 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:33.055806    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf"
Jul 29 19:51:45.971589 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:45.971552    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d31776a-fstwb"]
Jul 29 19:51:45.975031 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:45.975013    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d31776a-fstwb"
Jul 29 19:51:45.979160 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:45.979138    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jsq2r\""
Jul 29 19:51:45.983721 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:45.983691    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d31776a-fstwb"]
Jul 29 19:51:46.110375 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:46.110335    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82931199-556a-45cf-8c70-0e0709bdef1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2d31776a-fstwb\" (UID: \"82931199-556a-45cf-8c70-0e0709bdef1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d31776a-fstwb"
Jul 29 19:51:46.211104 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:46.211066    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82931199-556a-45cf-8c70-0e0709bdef1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2d31776a-fstwb\" (UID: \"82931199-556a-45cf-8c70-0e0709bdef1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d31776a-fstwb"
Jul 29 19:51:46.211684 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:46.211666    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82931199-556a-45cf-8c70-0e0709bdef1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2d31776a-fstwb\" (UID: \"82931199-556a-45cf-8c70-0e0709bdef1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d31776a-fstwb"
Jul 29 19:51:46.288299 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:46.288210    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d31776a-fstwb"
Jul 29 19:51:46.423517 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:46.423488    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d31776a-fstwb"]
Jul 29 19:51:46.425116 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:51:46.425091    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82931199_556a_45cf_8c70_0e0709bdef1a.slice/crio-cef049426930f26541a46990210b61315922fac08bf94c35aa9079d310fbd48f WatchSource:0}: Error finding container cef049426930f26541a46990210b61315922fac08bf94c35aa9079d310fbd48f: Status 404 returned error can't find the container with id cef049426930f26541a46990210b61315922fac08bf94c35aa9079d310fbd48f
Jul 29 19:51:47.112658 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:47.112569    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d31776a-fstwb" event={"ID":"82931199-556a-45cf-8c70-0e0709bdef1a","Type":"ContainerStarted","Data":"027c507ac2b7fc1fe05f471cb81e0dfce99fb251c3db76866dfc8c00fcb16f07"}
Jul 29 19:51:47.112658 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:47.112627    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d31776a-fstwb" event={"ID":"82931199-556a-45cf-8c70-0e0709bdef1a","Type":"ContainerStarted","Data":"cef049426930f26541a46990210b61315922fac08bf94c35aa9079d310fbd48f"}
Jul 29 19:51:47.131616 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:47.131565    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d31776a-fstwb" podStartSLOduration=2.131550466 podStartE2EDuration="2.131550466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:51:47.130678512 +0000 UTC m=+2077.958177571" watchObservedRunningTime="2026-07-29 19:51:47.131550466 +0000 UTC m=+2077.959049524"
Jul 29 19:51:48.119064 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:48.119031    2557 generic.go:356] "Generic (PLEG): container finished" podID="82931199-556a-45cf-8c70-0e0709bdef1a" containerID="027c507ac2b7fc1fe05f471cb81e0dfce99fb251c3db76866dfc8c00fcb16f07" exitCode=0
Jul 29 19:51:48.119064 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:48.119067    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d31776a-fstwb" event={"ID":"82931199-556a-45cf-8c70-0e0709bdef1a","Type":"ContainerDied","Data":"027c507ac2b7fc1fe05f471cb81e0dfce99fb251c3db76866dfc8c00fcb16f07"}
Jul 29 19:51:49.253730 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:49.253705    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d31776a-fstwb"
Jul 29 19:51:49.341100 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:49.341069    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82931199-556a-45cf-8c70-0e0709bdef1a-maas-api-service-ca\") pod \"82931199-556a-45cf-8c70-0e0709bdef1a\" (UID: \"82931199-556a-45cf-8c70-0e0709bdef1a\") "
Jul 29 19:51:49.341426 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:49.341402    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82931199-556a-45cf-8c70-0e0709bdef1a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "82931199-556a-45cf-8c70-0e0709bdef1a" (UID: "82931199-556a-45cf-8c70-0e0709bdef1a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:51:49.442222 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:49.442192    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82931199-556a-45cf-8c70-0e0709bdef1a-maas-api-service-ca\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:51:50.129860 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:50.129828    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d31776a-fstwb"
Jul 29 19:51:50.130053 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:50.129831    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d31776a-fstwb" event={"ID":"82931199-556a-45cf-8c70-0e0709bdef1a","Type":"ContainerDied","Data":"cef049426930f26541a46990210b61315922fac08bf94c35aa9079d310fbd48f"}
Jul 29 19:51:50.130053 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:50.129980    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cef049426930f26541a46990210b61315922fac08bf94c35aa9079d310fbd48f"
Jul 29 19:51:56.472505 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:56.472470    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr"]
Jul 29 19:51:56.472953 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:56.472723    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr" podUID="0b34c407-bf77-44f7-bd80-f066d27686d6" containerName="maas-api" containerID="cri-o://1d33b532554b4860017016ca77107820df2dda3bdad3f42534ba0590cce506d8" gracePeriod=30
Jul 29 19:51:56.541791 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:56.541756    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2d31776a-5d864dcb74-bkz9z"]
Jul 29 19:51:56.542082 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:56.542030    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-2d31776a-5d864dcb74-bkz9z" podUID="a5316aa0-e7b8-4955-b412-2f2673bd007f" containerName="payload-processing" containerID="cri-o://bfdc48514eec05c6fbf62b6b76639bf1719d24d729ab14130f46f679bc3501f3" gracePeriod=30
Jul 29 19:51:56.547300 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:56.547259    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2d31776a-5ddb44597b-m7wqb"]
Jul 29 19:51:56.547549 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:56.547502    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-2d31776a-5ddb44597b-m7wqb" podUID="5a758cfe-7d56-4c39-be42-6a7dfad9bec3" containerName="payload-pre-processing" containerID="cri-o://28080d5f8fc303e59a47f693e1160e9523f2094dbdce15ad215437db7b77515e" gracePeriod=30
Jul 29 19:51:56.715645 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:56.715580    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d31776a-fstwb"]
Jul 29 19:51:56.717470 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:56.717133    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d31776a-fstwb"]
Jul 29 19:51:56.797926 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:56.797899    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr"
Jul 29 19:51:56.827555 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:56.827526    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2d31776a-5ddb44597b-m7wqb"
Jul 29 19:51:56.835117 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:56.835095    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2d31776a-5d864dcb74-bkz9z"
Jul 29 19:51:56.910605 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:56.910562    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b27cq\" (UniqueName: \"kubernetes.io/projected/0b34c407-bf77-44f7-bd80-f066d27686d6-kube-api-access-b27cq\") pod \"0b34c407-bf77-44f7-bd80-f066d27686d6\" (UID: \"0b34c407-bf77-44f7-bd80-f066d27686d6\") "
Jul 29 19:51:56.910796 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:56.910645    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxvb9\" (UniqueName: \"kubernetes.io/projected/5a758cfe-7d56-4c39-be42-6a7dfad9bec3-kube-api-access-dxvb9\") pod \"5a758cfe-7d56-4c39-be42-6a7dfad9bec3\" (UID: \"5a758cfe-7d56-4c39-be42-6a7dfad9bec3\") "
Jul 29 19:51:56.910796 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:56.910714    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b34c407-bf77-44f7-bd80-f066d27686d6-maas-api-tls\") pod \"0b34c407-bf77-44f7-bd80-f066d27686d6\" (UID: \"0b34c407-bf77-44f7-bd80-f066d27686d6\") "
Jul 29 19:51:56.910796 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:56.910762    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a758cfe-7d56-4c39-be42-6a7dfad9bec3-plugins-config-volume\") pod \"5a758cfe-7d56-4c39-be42-6a7dfad9bec3\" (UID: \"5a758cfe-7d56-4c39-be42-6a7dfad9bec3\") "
Jul 29 19:51:56.910796 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:56.910793    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b34c407-bf77-44f7-bd80-f066d27686d6-openshift-service-ca\") pod \"0b34c407-bf77-44f7-bd80-f066d27686d6\" (UID: \"0b34c407-bf77-44f7-bd80-f066d27686d6\") "
Jul 29 19:51:56.911233 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:56.911200    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a758cfe-7d56-4c39-be42-6a7dfad9bec3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5a758cfe-7d56-4c39-be42-6a7dfad9bec3" (UID: "5a758cfe-7d56-4c39-be42-6a7dfad9bec3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:51:56.911233 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:56.911224    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b34c407-bf77-44f7-bd80-f066d27686d6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0b34c407-bf77-44f7-bd80-f066d27686d6" (UID: "0b34c407-bf77-44f7-bd80-f066d27686d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:51:56.912692 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:56.912670    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b34c407-bf77-44f7-bd80-f066d27686d6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0b34c407-bf77-44f7-bd80-f066d27686d6" (UID: "0b34c407-bf77-44f7-bd80-f066d27686d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:51:56.913192 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:56.913174    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a758cfe-7d56-4c39-be42-6a7dfad9bec3-kube-api-access-dxvb9" (OuterVolumeSpecName: "kube-api-access-dxvb9") pod "5a758cfe-7d56-4c39-be42-6a7dfad9bec3" (UID: "5a758cfe-7d56-4c39-be42-6a7dfad9bec3"). InnerVolumeSpecName "kube-api-access-dxvb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:51:56.913192 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:56.913182    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b34c407-bf77-44f7-bd80-f066d27686d6-kube-api-access-b27cq" (OuterVolumeSpecName: "kube-api-access-b27cq") pod "0b34c407-bf77-44f7-bd80-f066d27686d6" (UID: "0b34c407-bf77-44f7-bd80-f066d27686d6"). InnerVolumeSpecName "kube-api-access-b27cq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:51:57.011764 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.011679    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlz4j\" (UniqueName: \"kubernetes.io/projected/a5316aa0-e7b8-4955-b412-2f2673bd007f-kube-api-access-qlz4j\") pod \"a5316aa0-e7b8-4955-b412-2f2673bd007f\" (UID: \"a5316aa0-e7b8-4955-b412-2f2673bd007f\") "
Jul 29 19:51:57.011764 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.011747    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5316aa0-e7b8-4955-b412-2f2673bd007f-plugins-config-volume\") pod \"a5316aa0-e7b8-4955-b412-2f2673bd007f\" (UID: \"a5316aa0-e7b8-4955-b412-2f2673bd007f\") "
Jul 29 19:51:57.012004 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.011929    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxvb9\" (UniqueName: \"kubernetes.io/projected/5a758cfe-7d56-4c39-be42-6a7dfad9bec3-kube-api-access-dxvb9\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:51:57.012004 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.011942    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b34c407-bf77-44f7-bd80-f066d27686d6-maas-api-tls\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:51:57.012004 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.011951    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a758cfe-7d56-4c39-be42-6a7dfad9bec3-plugins-config-volume\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:51:57.012004 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.011974    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b34c407-bf77-44f7-bd80-f066d27686d6-openshift-service-ca\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:51:57.012004 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.011988    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b27cq\" (UniqueName: \"kubernetes.io/projected/0b34c407-bf77-44f7-bd80-f066d27686d6-kube-api-access-b27cq\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:51:57.012264 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.012176    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5316aa0-e7b8-4955-b412-2f2673bd007f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a5316aa0-e7b8-4955-b412-2f2673bd007f" (UID: "a5316aa0-e7b8-4955-b412-2f2673bd007f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:51:57.013607 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.013587    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5316aa0-e7b8-4955-b412-2f2673bd007f-kube-api-access-qlz4j" (OuterVolumeSpecName: "kube-api-access-qlz4j") pod "a5316aa0-e7b8-4955-b412-2f2673bd007f" (UID: "a5316aa0-e7b8-4955-b412-2f2673bd007f"). InnerVolumeSpecName "kube-api-access-qlz4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:51:57.113287 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.113249    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5316aa0-e7b8-4955-b412-2f2673bd007f-plugins-config-volume\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:51:57.113287 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.113279    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlz4j\" (UniqueName: \"kubernetes.io/projected/a5316aa0-e7b8-4955-b412-2f2673bd007f-kube-api-access-qlz4j\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:51:57.167392 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.167355    2557 generic.go:356] "Generic (PLEG): container finished" podID="0b34c407-bf77-44f7-bd80-f066d27686d6" containerID="1d33b532554b4860017016ca77107820df2dda3bdad3f42534ba0590cce506d8" exitCode=0
Jul 29 19:51:57.167553 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.167402    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr"
Jul 29 19:51:57.167553 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.167444    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr" event={"ID":"0b34c407-bf77-44f7-bd80-f066d27686d6","Type":"ContainerDied","Data":"1d33b532554b4860017016ca77107820df2dda3bdad3f42534ba0590cce506d8"}
Jul 29 19:51:57.167553 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.167510    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr" event={"ID":"0b34c407-bf77-44f7-bd80-f066d27686d6","Type":"ContainerDied","Data":"77f9e9ff43d99cf31a85689f9dfe182a5888fb09019a79f95a2cd84d7ea36b8c"}
Jul 29 19:51:57.167553 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.167548    2557 scope.go:117] "RemoveContainer" containerID="1d33b532554b4860017016ca77107820df2dda3bdad3f42534ba0590cce506d8"
Jul 29 19:51:57.168747 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.168725    2557 generic.go:356] "Generic (PLEG): container finished" podID="5a758cfe-7d56-4c39-be42-6a7dfad9bec3" containerID="28080d5f8fc303e59a47f693e1160e9523f2094dbdce15ad215437db7b77515e" exitCode=0
Jul 29 19:51:57.168836 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.168766    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2d31776a-5ddb44597b-m7wqb"
Jul 29 19:51:57.168836 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.168815    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2d31776a-5ddb44597b-m7wqb" event={"ID":"5a758cfe-7d56-4c39-be42-6a7dfad9bec3","Type":"ContainerDied","Data":"28080d5f8fc303e59a47f693e1160e9523f2094dbdce15ad215437db7b77515e"}
Jul 29 19:51:57.168932 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.168862    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2d31776a-5ddb44597b-m7wqb" event={"ID":"5a758cfe-7d56-4c39-be42-6a7dfad9bec3","Type":"ContainerDied","Data":"7589ed682a2acdbc1e43b4858486e76cb92c08017c7ec89fc03e291e957d6e44"}
Jul 29 19:51:57.170311 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.170284    2557 generic.go:356] "Generic (PLEG): container finished" podID="a5316aa0-e7b8-4955-b412-2f2673bd007f" containerID="bfdc48514eec05c6fbf62b6b76639bf1719d24d729ab14130f46f679bc3501f3" exitCode=0
Jul 29 19:51:57.170436 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.170323    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2d31776a-5d864dcb74-bkz9z" event={"ID":"a5316aa0-e7b8-4955-b412-2f2673bd007f","Type":"ContainerDied","Data":"bfdc48514eec05c6fbf62b6b76639bf1719d24d729ab14130f46f679bc3501f3"}
Jul 29 19:51:57.170436 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.170332    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2d31776a-5d864dcb74-bkz9z"
Jul 29 19:51:57.170436 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.170354    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2d31776a-5d864dcb74-bkz9z" event={"ID":"a5316aa0-e7b8-4955-b412-2f2673bd007f","Type":"ContainerDied","Data":"dab49e5d74a5cbb09882a7abf56a4977ed9ad95a90ee66a536340c82fc825f95"}
Jul 29 19:51:57.176839 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.176819    2557 scope.go:117] "RemoveContainer" containerID="1d33b532554b4860017016ca77107820df2dda3bdad3f42534ba0590cce506d8"
Jul 29 19:51:57.177126 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:51:57.177105    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d33b532554b4860017016ca77107820df2dda3bdad3f42534ba0590cce506d8\": container with ID starting with 1d33b532554b4860017016ca77107820df2dda3bdad3f42534ba0590cce506d8 not found: ID does not exist" containerID="1d33b532554b4860017016ca77107820df2dda3bdad3f42534ba0590cce506d8"
Jul 29 19:51:57.177196 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.177132    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d33b532554b4860017016ca77107820df2dda3bdad3f42534ba0590cce506d8"} err="failed to get container status \"1d33b532554b4860017016ca77107820df2dda3bdad3f42534ba0590cce506d8\": rpc error: code = NotFound desc = could not find container \"1d33b532554b4860017016ca77107820df2dda3bdad3f42534ba0590cce506d8\": container with ID starting with 1d33b532554b4860017016ca77107820df2dda3bdad3f42534ba0590cce506d8 not found: ID does not exist"
Jul 29 19:51:57.177196 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.177150    2557 scope.go:117] "RemoveContainer" containerID="28080d5f8fc303e59a47f693e1160e9523f2094dbdce15ad215437db7b77515e"
Jul 29 19:51:57.186130 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.186109    2557 scope.go:117] "RemoveContainer" containerID="28080d5f8fc303e59a47f693e1160e9523f2094dbdce15ad215437db7b77515e"
Jul 29 19:51:57.186365 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:51:57.186348    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28080d5f8fc303e59a47f693e1160e9523f2094dbdce15ad215437db7b77515e\": container with ID starting with 28080d5f8fc303e59a47f693e1160e9523f2094dbdce15ad215437db7b77515e not found: ID does not exist" containerID="28080d5f8fc303e59a47f693e1160e9523f2094dbdce15ad215437db7b77515e"
Jul 29 19:51:57.186424 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.186370    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28080d5f8fc303e59a47f693e1160e9523f2094dbdce15ad215437db7b77515e"} err="failed to get container status \"28080d5f8fc303e59a47f693e1160e9523f2094dbdce15ad215437db7b77515e\": rpc error: code = NotFound desc = could not find container \"28080d5f8fc303e59a47f693e1160e9523f2094dbdce15ad215437db7b77515e\": container with ID starting with 28080d5f8fc303e59a47f693e1160e9523f2094dbdce15ad215437db7b77515e not found: ID does not exist"
Jul 29 19:51:57.186424 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.186386    2557 scope.go:117] "RemoveContainer" containerID="bfdc48514eec05c6fbf62b6b76639bf1719d24d729ab14130f46f679bc3501f3"
Jul 29 19:51:57.194201 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.194179    2557 scope.go:117] "RemoveContainer" containerID="bfdc48514eec05c6fbf62b6b76639bf1719d24d729ab14130f46f679bc3501f3"
Jul 29 19:51:57.194476 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:51:57.194455    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfdc48514eec05c6fbf62b6b76639bf1719d24d729ab14130f46f679bc3501f3\": container with ID starting with bfdc48514eec05c6fbf62b6b76639bf1719d24d729ab14130f46f679bc3501f3 not found: ID does not exist" containerID="bfdc48514eec05c6fbf62b6b76639bf1719d24d729ab14130f46f679bc3501f3"
Jul 29 19:51:57.194552 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.194486    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfdc48514eec05c6fbf62b6b76639bf1719d24d729ab14130f46f679bc3501f3"} err="failed to get container status \"bfdc48514eec05c6fbf62b6b76639bf1719d24d729ab14130f46f679bc3501f3\": rpc error: code = NotFound desc = could not find container \"bfdc48514eec05c6fbf62b6b76639bf1719d24d729ab14130f46f679bc3501f3\": container with ID starting with bfdc48514eec05c6fbf62b6b76639bf1719d24d729ab14130f46f679bc3501f3 not found: ID does not exist"
Jul 29 19:51:57.196012 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.195990    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2d31776a-5ddb44597b-m7wqb"]
Jul 29 19:51:57.200118 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.200095    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2d31776a-5ddb44597b-m7wqb"]
Jul 29 19:51:57.210457 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.210436    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2d31776a-5d864dcb74-bkz9z"]
Jul 29 19:51:57.214508 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.214488    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2d31776a-5d864dcb74-bkz9z"]
Jul 29 19:51:57.224655 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.224633    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr"]
Jul 29 19:51:57.231277 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.231256    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2d31776a-588b75b8c6-6tsmr"]
Jul 29 19:51:57.680814 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.680779    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b34c407-bf77-44f7-bd80-f066d27686d6" path="/var/lib/kubelet/pods/0b34c407-bf77-44f7-bd80-f066d27686d6/volumes"
Jul 29 19:51:57.681209 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.681177    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a758cfe-7d56-4c39-be42-6a7dfad9bec3" path="/var/lib/kubelet/pods/5a758cfe-7d56-4c39-be42-6a7dfad9bec3/volumes"
Jul 29 19:51:57.681452 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.681439    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82931199-556a-45cf-8c70-0e0709bdef1a" path="/var/lib/kubelet/pods/82931199-556a-45cf-8c70-0e0709bdef1a/volumes"
Jul 29 19:51:57.681750 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:51:57.681738    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5316aa0-e7b8-4955-b412-2f2673bd007f" path="/var/lib/kubelet/pods/a5316aa0-e7b8-4955-b412-2f2673bd007f/volumes"
Jul 29 19:52:04.939556 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:52:04.939522    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf"]
Jul 29 19:52:12.274527 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:52:12.274494    2557 scope.go:117] "RemoveContainer" containerID="9c8068b5a68b192400f4470aec4c68f1c6d7654edd2b6d50c05ebb26744217a2"
Jul 29 19:52:19.960176 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:52:19.960121    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf" podUID="2e7acf16-796d-4d9f-97bc-0af0ea9f938e" containerName="main" containerID="cri-o://653799dd4575567d615b6b03c66a34e3dab4cef26cb7f01ea005f7a1771ecfc3" gracePeriod=45
Jul 29 19:52:20.214713 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:52:20.214661    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf"
Jul 29 19:52:20.289390 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:52:20.289359    2557 generic.go:356] "Generic (PLEG): container finished" podID="2e7acf16-796d-4d9f-97bc-0af0ea9f938e" containerID="653799dd4575567d615b6b03c66a34e3dab4cef26cb7f01ea005f7a1771ecfc3" exitCode=0
Jul 29 19:52:20.289554 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:52:20.289402    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf"
Jul 29 19:52:20.289554 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:52:20.289406    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf" event={"ID":"2e7acf16-796d-4d9f-97bc-0af0ea9f938e","Type":"ContainerDied","Data":"653799dd4575567d615b6b03c66a34e3dab4cef26cb7f01ea005f7a1771ecfc3"}
Jul 29 19:52:20.289554 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:52:20.289522    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf" event={"ID":"2e7acf16-796d-4d9f-97bc-0af0ea9f938e","Type":"ContainerDied","Data":"d95e42b4529b6fcc1d71e9a62989c1d8c641342585877f60842878bf7971fa2e"}
Jul 29 19:52:20.289554 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:52:20.289546    2557 scope.go:117] "RemoveContainer" containerID="653799dd4575567d615b6b03c66a34e3dab4cef26cb7f01ea005f7a1771ecfc3"
Jul 29 19:52:20.297760 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:52:20.297743    2557 scope.go:117] "RemoveContainer" containerID="653799dd4575567d615b6b03c66a34e3dab4cef26cb7f01ea005f7a1771ecfc3"
Jul 29 19:52:20.298000 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:52:20.297980    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"653799dd4575567d615b6b03c66a34e3dab4cef26cb7f01ea005f7a1771ecfc3\": container with ID starting with 653799dd4575567d615b6b03c66a34e3dab4cef26cb7f01ea005f7a1771ecfc3 not found: ID does not exist" containerID="653799dd4575567d615b6b03c66a34e3dab4cef26cb7f01ea005f7a1771ecfc3"
Jul 29 19:52:20.298066 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:52:20.298007    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"653799dd4575567d615b6b03c66a34e3dab4cef26cb7f01ea005f7a1771ecfc3"} err="failed to get container status \"653799dd4575567d615b6b03c66a34e3dab4cef26cb7f01ea005f7a1771ecfc3\": rpc error: code = NotFound desc = could not find container \"653799dd4575567d615b6b03c66a34e3dab4cef26cb7f01ea005f7a1771ecfc3\": container with ID starting with 653799dd4575567d615b6b03c66a34e3dab4cef26cb7f01ea005f7a1771ecfc3 not found: ID does not exist"
Jul 29 19:52:20.326944 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:52:20.326914    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-dshm\") pod \"2e7acf16-796d-4d9f-97bc-0af0ea9f938e\" (UID: \"2e7acf16-796d-4d9f-97bc-0af0ea9f938e\") "
Jul 29 19:52:20.327095 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:52:20.327038    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-home\") pod \"2e7acf16-796d-4d9f-97bc-0af0ea9f938e\" (UID: \"2e7acf16-796d-4d9f-97bc-0af0ea9f938e\") "
Jul 29 19:52:20.327161 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:52:20.327093    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-tmp-dir\") pod \"2e7acf16-796d-4d9f-97bc-0af0ea9f938e\" (UID: \"2e7acf16-796d-4d9f-97bc-0af0ea9f938e\") "
Jul 29 19:52:20.327161 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:52:20.327131    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbvgp\" (UniqueName: \"kubernetes.io/projected/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-kube-api-access-pbvgp\") pod \"2e7acf16-796d-4d9f-97bc-0af0ea9f938e\" (UID: \"2e7acf16-796d-4d9f-97bc-0af0ea9f938e\") "
Jul 29 19:52:20.327281 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:52:20.327159    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-tls-certs\") pod \"2e7acf16-796d-4d9f-97bc-0af0ea9f938e\" (UID: \"2e7acf16-796d-4d9f-97bc-0af0ea9f938e\") "
Jul 29 19:52:20.327341 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:52:20.327289    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-home" (OuterVolumeSpecName: "home") pod "2e7acf16-796d-4d9f-97bc-0af0ea9f938e" (UID: "2e7acf16-796d-4d9f-97bc-0af0ea9f938e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:52:20.327341 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:52:20.327309    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-model-cache\") pod \"2e7acf16-796d-4d9f-97bc-0af0ea9f938e\" (UID: \"2e7acf16-796d-4d9f-97bc-0af0ea9f938e\") "
Jul 29 19:52:20.327449 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:52:20.327405    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2e7acf16-796d-4d9f-97bc-0af0ea9f938e" (UID: "2e7acf16-796d-4d9f-97bc-0af0ea9f938e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:52:20.327516 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:52:20.327495    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-model-cache" (OuterVolumeSpecName: "model-cache") pod "2e7acf16-796d-4d9f-97bc-0af0ea9f938e" (UID: "2e7acf16-796d-4d9f-97bc-0af0ea9f938e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:52:20.327639 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:52:20.327622    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-model-cache\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:52:20.327639 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:52:20.327640    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-home\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:52:20.327772 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:52:20.327649    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-tmp-dir\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:52:20.329167 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:52:20.329140    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-dshm" (OuterVolumeSpecName: "dshm") pod "2e7acf16-796d-4d9f-97bc-0af0ea9f938e" (UID: "2e7acf16-796d-4d9f-97bc-0af0ea9f938e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:52:20.329259 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:52:20.329206    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e7acf16-796d-4d9f-97bc-0af0ea9f938e" (UID: "2e7acf16-796d-4d9f-97bc-0af0ea9f938e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:52:20.329311 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:52:20.329256    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-kube-api-access-pbvgp" (OuterVolumeSpecName: "kube-api-access-pbvgp") pod "2e7acf16-796d-4d9f-97bc-0af0ea9f938e" (UID: "2e7acf16-796d-4d9f-97bc-0af0ea9f938e"). InnerVolumeSpecName "kube-api-access-pbvgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:52:20.428092 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:52:20.428056    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbvgp\" (UniqueName: \"kubernetes.io/projected/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-kube-api-access-pbvgp\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:52:20.428092 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:52:20.428087    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-tls-certs\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:52:20.428092 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:52:20.428096    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e7acf16-796d-4d9f-97bc-0af0ea9f938e-dshm\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:52:20.612793 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:52:20.612761    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf"]
Jul 29 19:52:20.617388 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:52:20.617360    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-2d31776a/e2e-lifecycle-model-2d31776a-kserve-755dd6cbbc-2bhcf"]
Jul 29 19:52:21.678803 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:52:21.678766    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e7acf16-796d-4d9f-97bc-0af0ea9f938e" path="/var/lib/kubelet/pods/2e7acf16-796d-4d9f-97bc-0af0ea9f938e/volumes"
Jul 29 19:53:12.303240 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:53:12.303213    2557 scope.go:117] "RemoveContainer" containerID="fa1e0ec5bcacf35325b5d6ad97ed6f391a05e8d68c241869e509e2f8ba8e5e41"
Jul 29 19:53:12.311159 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:53:12.311131    2557 scope.go:117] "RemoveContainer" containerID="ff2a160835ad6473af28772261f0c487991516a30f91f78705249c43e9205a80"
Jul 29 19:54:02.144087 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.144041    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c"]
Jul 29 19:54:02.144563 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.144407    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a758cfe-7d56-4c39-be42-6a7dfad9bec3" containerName="payload-pre-processing"
Jul 29 19:54:02.144563 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.144419    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a758cfe-7d56-4c39-be42-6a7dfad9bec3" containerName="payload-pre-processing"
Jul 29 19:54:02.144563 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.144430    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82931199-556a-45cf-8c70-0e0709bdef1a" containerName="revoke-keys"
Jul 29 19:54:02.144563 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.144437    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="82931199-556a-45cf-8c70-0e0709bdef1a" containerName="revoke-keys"
Jul 29 19:54:02.144563 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.144443    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e7acf16-796d-4d9f-97bc-0af0ea9f938e" containerName="main"
Jul 29 19:54:02.144563 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.144449    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e7acf16-796d-4d9f-97bc-0af0ea9f938e" containerName="main"
Jul 29 19:54:02.144563 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.144462    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b34c407-bf77-44f7-bd80-f066d27686d6" containerName="maas-api"
Jul 29 19:54:02.144563 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.144467    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b34c407-bf77-44f7-bd80-f066d27686d6" containerName="maas-api"
Jul 29 19:54:02.144563 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.144480    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5316aa0-e7b8-4955-b412-2f2673bd007f" containerName="payload-processing"
Jul 29 19:54:02.144563 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.144485    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5316aa0-e7b8-4955-b412-2f2673bd007f" containerName="payload-processing"
Jul 29 19:54:02.144563 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.144535    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="82931199-556a-45cf-8c70-0e0709bdef1a" containerName="revoke-keys"
Jul 29 19:54:02.144563 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.144542    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5316aa0-e7b8-4955-b412-2f2673bd007f" containerName="payload-processing"
Jul 29 19:54:02.144563 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.144549    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a758cfe-7d56-4c39-be42-6a7dfad9bec3" containerName="payload-pre-processing"
Jul 29 19:54:02.144563 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.144556    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b34c407-bf77-44f7-bd80-f066d27686d6" containerName="maas-api"
Jul 29 19:54:02.144563 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.144562    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e7acf16-796d-4d9f-97bc-0af0ea9f938e" containerName="main"
Jul 29 19:54:02.147518 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.147502    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c"
Jul 29 19:54:02.150229 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.150207    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4dw6s\""
Jul 29 19:54:02.150354 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.150227    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-06a5f2-serving-cert\""
Jul 29 19:54:02.158266 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.158212    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c"]
Jul 29 19:54:02.306531 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.306503    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a87fc48e-7f52-48e6-9c3c-92216470881d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c\" (UID: \"a87fc48e-7f52-48e6-9c3c-92216470881d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c"
Jul 29 19:54:02.306703 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.306552    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjcgp\" (UniqueName: \"kubernetes.io/projected/a87fc48e-7f52-48e6-9c3c-92216470881d-kube-api-access-sjcgp\") pod \"maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c\" (UID: \"a87fc48e-7f52-48e6-9c3c-92216470881d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c"
Jul 29 19:54:02.306703 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.306614    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a87fc48e-7f52-48e6-9c3c-92216470881d-maas-api-tls\") pod \"maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c\" (UID: \"a87fc48e-7f52-48e6-9c3c-92216470881d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c"
Jul 29 19:54:02.407260 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.407179    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a87fc48e-7f52-48e6-9c3c-92216470881d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c\" (UID: \"a87fc48e-7f52-48e6-9c3c-92216470881d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c"
Jul 29 19:54:02.407260 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.407244    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjcgp\" (UniqueName: \"kubernetes.io/projected/a87fc48e-7f52-48e6-9c3c-92216470881d-kube-api-access-sjcgp\") pod \"maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c\" (UID: \"a87fc48e-7f52-48e6-9c3c-92216470881d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c"
Jul 29 19:54:02.407440 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.407291    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a87fc48e-7f52-48e6-9c3c-92216470881d-maas-api-tls\") pod \"maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c\" (UID: \"a87fc48e-7f52-48e6-9c3c-92216470881d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c"
Jul 29 19:54:02.407941 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.407921    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a87fc48e-7f52-48e6-9c3c-92216470881d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c\" (UID: \"a87fc48e-7f52-48e6-9c3c-92216470881d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c"
Jul 29 19:54:02.409708 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.409690    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a87fc48e-7f52-48e6-9c3c-92216470881d-maas-api-tls\") pod \"maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c\" (UID: \"a87fc48e-7f52-48e6-9c3c-92216470881d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c"
Jul 29 19:54:02.415942 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.415917    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjcgp\" (UniqueName: \"kubernetes.io/projected/a87fc48e-7f52-48e6-9c3c-92216470881d-kube-api-access-sjcgp\") pod \"maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c\" (UID: \"a87fc48e-7f52-48e6-9c3c-92216470881d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c"
Jul 29 19:54:02.460251 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.460226    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c"
Jul 29 19:54:02.595802 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.595777    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c"]
Jul 29 19:54:02.598426 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:54:02.598397    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda87fc48e_7f52_48e6_9c3c_92216470881d.slice/crio-9036960391c7c616114900165857c36557caf89186293528aea310799ea1b72c WatchSource:0}: Error finding container 9036960391c7c616114900165857c36557caf89186293528aea310799ea1b72c: Status 404 returned error can't find the container with id 9036960391c7c616114900165857c36557caf89186293528aea310799ea1b72c
Jul 29 19:54:02.762202 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.762107    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c" event={"ID":"a87fc48e-7f52-48e6-9c3c-92216470881d","Type":"ContainerStarted","Data":"9036960391c7c616114900165857c36557caf89186293528aea310799ea1b72c"}
Jul 29 19:54:02.810078 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.810038    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-06a5f2-5bc74bc4c4-l6dgh"]
Jul 29 19:54:02.816133 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.815076    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-06a5f2-5bc74bc4c4-l6dgh"
Jul 29 19:54:02.818224 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.818193    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-06a5f2-dockercfg-cqbbw\""
Jul 29 19:54:02.818816 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.818793    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-06a5f2\""
Jul 29 19:54:02.824224 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.824198    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-06a5f2-5bc74bc4c4-l6dgh"]
Jul 29 19:54:02.860002 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.859959    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-06a5f2-7fd7464688-t4pxl"]
Jul 29 19:54:02.864088 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.864067    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-06a5f2-7fd7464688-t4pxl"
Jul 29 19:54:02.874605 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.874581    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-06a5f2-7fd7464688-t4pxl"]
Jul 29 19:54:02.912070 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.912034    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps7lb\" (UniqueName: \"kubernetes.io/projected/0965cf59-dee6-4391-a0f6-e84d807677b6-kube-api-access-ps7lb\") pod \"payload-processing-e2e-inf-a-06a5f2-5bc74bc4c4-l6dgh\" (UID: \"0965cf59-dee6-4391-a0f6-e84d807677b6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-06a5f2-5bc74bc4c4-l6dgh"
Jul 29 19:54:02.912219 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:02.912156    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0965cf59-dee6-4391-a0f6-e84d807677b6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-06a5f2-5bc74bc4c4-l6dgh\" (UID: \"0965cf59-dee6-4391-a0f6-e84d807677b6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-06a5f2-5bc74bc4c4-l6dgh"
Jul 29 19:54:03.013488 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:03.013396    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0965cf59-dee6-4391-a0f6-e84d807677b6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-06a5f2-5bc74bc4c4-l6dgh\" (UID: \"0965cf59-dee6-4391-a0f6-e84d807677b6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-06a5f2-5bc74bc4c4-l6dgh"
Jul 29 19:54:03.013488 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:03.013462    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-06a5f2-7fd7464688-t4pxl\" (UID: \"a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-06a5f2-7fd7464688-t4pxl"
Jul 29 19:54:03.013488 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:03.013490    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps7lb\" (UniqueName: \"kubernetes.io/projected/0965cf59-dee6-4391-a0f6-e84d807677b6-kube-api-access-ps7lb\") pod \"payload-processing-e2e-inf-a-06a5f2-5bc74bc4c4-l6dgh\" (UID: \"0965cf59-dee6-4391-a0f6-e84d807677b6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-06a5f2-5bc74bc4c4-l6dgh"
Jul 29 19:54:03.013777 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:03.013563    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n5pz\" (UniqueName: \"kubernetes.io/projected/a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a-kube-api-access-7n5pz\") pod \"payload-pre-processing-e2e-inf-a-06a5f2-7fd7464688-t4pxl\" (UID: \"a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-06a5f2-7fd7464688-t4pxl"
Jul 29 19:54:03.014230 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:03.014205    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0965cf59-dee6-4391-a0f6-e84d807677b6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-06a5f2-5bc74bc4c4-l6dgh\" (UID: \"0965cf59-dee6-4391-a0f6-e84d807677b6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-06a5f2-5bc74bc4c4-l6dgh"
Jul 29 19:54:03.023748 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:03.023726    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps7lb\" (UniqueName: \"kubernetes.io/projected/0965cf59-dee6-4391-a0f6-e84d807677b6-kube-api-access-ps7lb\") pod \"payload-processing-e2e-inf-a-06a5f2-5bc74bc4c4-l6dgh\" (UID: \"0965cf59-dee6-4391-a0f6-e84d807677b6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-06a5f2-5bc74bc4c4-l6dgh"
Jul 29 19:54:03.115063 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:03.115018    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-06a5f2-7fd7464688-t4pxl\" (UID: \"a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-06a5f2-7fd7464688-t4pxl"
Jul 29 19:54:03.115246 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:03.115115    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n5pz\" (UniqueName: \"kubernetes.io/projected/a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a-kube-api-access-7n5pz\") pod \"payload-pre-processing-e2e-inf-a-06a5f2-7fd7464688-t4pxl\" (UID: \"a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-06a5f2-7fd7464688-t4pxl"
Jul 29 19:54:03.115932 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:03.115910    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-06a5f2-7fd7464688-t4pxl\" (UID: \"a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-06a5f2-7fd7464688-t4pxl"
Jul 29 19:54:03.123232 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:03.123202    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n5pz\" (UniqueName: \"kubernetes.io/projected/a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a-kube-api-access-7n5pz\") pod \"payload-pre-processing-e2e-inf-a-06a5f2-7fd7464688-t4pxl\" (UID: \"a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-06a5f2-7fd7464688-t4pxl"
Jul 29 19:54:03.129012 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:03.128995    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-06a5f2-5bc74bc4c4-l6dgh"
Jul 29 19:54:03.175284 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:03.175252    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-06a5f2-7fd7464688-t4pxl"
Jul 29 19:54:03.271319 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:03.271224    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-06a5f2-5bc74bc4c4-l6dgh"]
Jul 29 19:54:03.275670 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:54:03.274335    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0965cf59_dee6_4391_a0f6_e84d807677b6.slice/crio-cc4066be92212483fdaf61c6aef08ccf9d474153d5f03558724c4ed7fc030d63 WatchSource:0}: Error finding container cc4066be92212483fdaf61c6aef08ccf9d474153d5f03558724c4ed7fc030d63: Status 404 returned error can't find the container with id cc4066be92212483fdaf61c6aef08ccf9d474153d5f03558724c4ed7fc030d63
Jul 29 19:54:03.325503 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:03.325479    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-06a5f2-7fd7464688-t4pxl"]
Jul 29 19:54:03.328236 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:54:03.328211    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda244e8b8_0262_4e7e_b705_1bcfc8a7dd5a.slice/crio-4e52c91bcc461687e2882f229cc0911b8b3ab2e61e6c0040935544c744bddd16 WatchSource:0}: Error finding container 4e52c91bcc461687e2882f229cc0911b8b3ab2e61e6c0040935544c744bddd16: Status 404 returned error can't find the container with id 4e52c91bcc461687e2882f229cc0911b8b3ab2e61e6c0040935544c744bddd16
Jul 29 19:54:03.769846 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:03.769812    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c" event={"ID":"a87fc48e-7f52-48e6-9c3c-92216470881d","Type":"ContainerStarted","Data":"1aa966efbb811226ac666ee9fd190ec0b3be4ebf880641e5245bdf0de112f700"}
Jul 29 19:54:03.770057 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:03.769919    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c"
Jul 29 19:54:03.771696 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:03.771658    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-06a5f2-7fd7464688-t4pxl" event={"ID":"a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a","Type":"ContainerStarted","Data":"fda984b0a63f8eabaaabebeccbccb1294bd5c046f654781a69800496b2a773ba"}
Jul 29 19:54:03.771852 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:03.771723    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-06a5f2-7fd7464688-t4pxl"
Jul 29 19:54:03.771852 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:03.771748    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-06a5f2-7fd7464688-t4pxl" event={"ID":"a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a","Type":"ContainerStarted","Data":"4e52c91bcc461687e2882f229cc0911b8b3ab2e61e6c0040935544c744bddd16"}
Jul 29 19:54:03.773434 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:03.773408    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-06a5f2-5bc74bc4c4-l6dgh" event={"ID":"0965cf59-dee6-4391-a0f6-e84d807677b6","Type":"ContainerStarted","Data":"5f5a8c96f00e85490ef442e3ecd3d99b11a5d95aa5d08314255fd67ef0fb0be1"}
Jul 29 19:54:03.773546 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:03.773456    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-06a5f2-5bc74bc4c4-l6dgh" event={"ID":"0965cf59-dee6-4391-a0f6-e84d807677b6","Type":"ContainerStarted","Data":"cc4066be92212483fdaf61c6aef08ccf9d474153d5f03558724c4ed7fc030d63"}
Jul 29 19:54:03.773546 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:03.773510    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-06a5f2-5bc74bc4c4-l6dgh"
Jul 29 19:54:03.789455 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:03.789411    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c" podStartSLOduration=0.96659753 podStartE2EDuration="1.789398719s" totalImagesPullingTime="822.801189ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:54:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:54:03.787334058 +0000 UTC m=+2214.614833107" watchObservedRunningTime="2026-07-29 19:54:03.789398719 +0000 UTC m=+2214.616897777"
Jul 29 19:54:03.804712 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:03.804665    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-06a5f2-5bc74bc4c4-l6dgh" podStartSLOduration=1.804650472 podStartE2EDuration="1.804650472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:54:03.803449698 +0000 UTC m=+2214.630948759" watchObservedRunningTime="2026-07-29 19:54:03.804650472 +0000 UTC m=+2214.632149529"
Jul 29 19:54:03.819954 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:03.819913    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-06a5f2-7fd7464688-t4pxl" podStartSLOduration=1.819900977 podStartE2EDuration="1.819900977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:54:03.818925938 +0000 UTC m=+2214.646424997" watchObservedRunningTime="2026-07-29 19:54:03.819900977 +0000 UTC m=+2214.647400013"
Jul 29 19:54:09.783312 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:09.783286    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c"
Jul 29 19:54:12.335315 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:12.335284    2557 scope.go:117] "RemoveContainer" containerID="1ec09ee8aa1c52df70298904b3adfc55077e50c182af2b9f5fbe31fa9dfe59bd"
Jul 29 19:54:14.780186 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:14.780154    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-06a5f2-5bc74bc4c4-l6dgh"
Jul 29 19:54:14.780639 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:14.780530    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-06a5f2-7fd7464688-t4pxl"
Jul 29 19:54:30.199004 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:30.198953    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9"]
Jul 29 19:54:30.203121 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:30.203095    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9"
Jul 29 19:54:30.205666 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:30.205642    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-1d9a27-serving-cert\""
Jul 29 19:54:30.213682 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:30.213652    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9"]
Jul 29 19:54:30.258676 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:30.258638    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47626a9a-94cb-47bc-a34d-288f743d9b04-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9\" (UID: \"47626a9a-94cb-47bc-a34d-288f743d9b04\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9"
Jul 29 19:54:30.258676 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:30.258682    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47626a9a-94cb-47bc-a34d-288f743d9b04-maas-api-tls\") pod \"maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9\" (UID: \"47626a9a-94cb-47bc-a34d-288f743d9b04\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9"
Jul 29 19:54:30.258876 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:30.258814    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt89r\" (UniqueName: \"kubernetes.io/projected/47626a9a-94cb-47bc-a34d-288f743d9b04-kube-api-access-kt89r\") pod \"maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9\" (UID: \"47626a9a-94cb-47bc-a34d-288f743d9b04\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9"
Jul 29 19:54:30.359891 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:30.359860    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt89r\" (UniqueName: \"kubernetes.io/projected/47626a9a-94cb-47bc-a34d-288f743d9b04-kube-api-access-kt89r\") pod \"maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9\" (UID: \"47626a9a-94cb-47bc-a34d-288f743d9b04\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9"
Jul 29 19:54:30.360119 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:30.359909    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47626a9a-94cb-47bc-a34d-288f743d9b04-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9\" (UID: \"47626a9a-94cb-47bc-a34d-288f743d9b04\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9"
Jul 29 19:54:30.360119 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:30.359947    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47626a9a-94cb-47bc-a34d-288f743d9b04-maas-api-tls\") pod \"maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9\" (UID: \"47626a9a-94cb-47bc-a34d-288f743d9b04\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9"
Jul 29 19:54:30.360620 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:30.360600    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47626a9a-94cb-47bc-a34d-288f743d9b04-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9\" (UID: \"47626a9a-94cb-47bc-a34d-288f743d9b04\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9"
Jul 29 19:54:30.362387 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:30.362361    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47626a9a-94cb-47bc-a34d-288f743d9b04-maas-api-tls\") pod \"maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9\" (UID: \"47626a9a-94cb-47bc-a34d-288f743d9b04\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9"
Jul 29 19:54:30.368413 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:30.368386    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt89r\" (UniqueName: \"kubernetes.io/projected/47626a9a-94cb-47bc-a34d-288f743d9b04-kube-api-access-kt89r\") pod \"maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9\" (UID: \"47626a9a-94cb-47bc-a34d-288f743d9b04\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9"
Jul 29 19:54:30.515640 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:30.515556    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9"
Jul 29 19:54:30.674747 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:30.674716    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9"]
Jul 29 19:54:30.676100 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:54:30.676075    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47626a9a_94cb_47bc_a34d_288f743d9b04.slice/crio-6afbdbacc570462e791d6bf6f0004c5551e0a083b6a824b148b265844a9ecb61 WatchSource:0}: Error finding container 6afbdbacc570462e791d6bf6f0004c5551e0a083b6a824b148b265844a9ecb61: Status 404 returned error can't find the container with id 6afbdbacc570462e791d6bf6f0004c5551e0a083b6a824b148b265844a9ecb61
Jul 29 19:54:30.899337 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:30.899303    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9" event={"ID":"47626a9a-94cb-47bc-a34d-288f743d9b04","Type":"ContainerStarted","Data":"6afbdbacc570462e791d6bf6f0004c5551e0a083b6a824b148b265844a9ecb61"}
Jul 29 19:54:30.998107 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:30.998073    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1d9a27-7ff67d944d-wn4k4"]
Jul 29 19:54:31.002608 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:31.002569    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1d9a27-f6579896d-mpknn"]
Jul 29 19:54:31.002786 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:31.002702    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d9a27-7ff67d944d-wn4k4"
Jul 29 19:54:31.005838 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:31.005819    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1d9a27-f6579896d-mpknn"
Jul 29 19:54:31.006152 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:31.006132    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-1d9a27\""
Jul 29 19:54:31.008432 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:31.008417    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-1d9a27-dockercfg-b7q92\""
Jul 29 19:54:31.015827 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:31.015797    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1d9a27-7ff67d944d-wn4k4"]
Jul 29 19:54:31.020393 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:31.020371    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1d9a27-f6579896d-mpknn"]
Jul 29 19:54:31.065441 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:31.065415    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfc7j\" (UniqueName: \"kubernetes.io/projected/29567083-0305-4296-8baa-30330c968658-kube-api-access-qfc7j\") pod \"payload-processing-e2e-inf-b-1d9a27-f6579896d-mpknn\" (UID: \"29567083-0305-4296-8baa-30330c968658\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1d9a27-f6579896d-mpknn"
Jul 29 19:54:31.065597 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:31.065457    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pcwb\" (UniqueName: \"kubernetes.io/projected/a1d6f940-f312-422c-b951-7acb7df5b318-kube-api-access-2pcwb\") pod \"payload-pre-processing-e2e-inf-b-1d9a27-7ff67d944d-wn4k4\" (UID: \"a1d6f940-f312-422c-b951-7acb7df5b318\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d9a27-7ff67d944d-wn4k4"
Jul 29 19:54:31.065597 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:31.065514    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29567083-0305-4296-8baa-30330c968658-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1d9a27-f6579896d-mpknn\" (UID: \"29567083-0305-4296-8baa-30330c968658\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1d9a27-f6579896d-mpknn"
Jul 29 19:54:31.065597 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:31.065557    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1d6f940-f312-422c-b951-7acb7df5b318-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1d9a27-7ff67d944d-wn4k4\" (UID: \"a1d6f940-f312-422c-b951-7acb7df5b318\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d9a27-7ff67d944d-wn4k4"
Jul 29 19:54:31.166815 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:31.166727    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfc7j\" (UniqueName: \"kubernetes.io/projected/29567083-0305-4296-8baa-30330c968658-kube-api-access-qfc7j\") pod \"payload-processing-e2e-inf-b-1d9a27-f6579896d-mpknn\" (UID: \"29567083-0305-4296-8baa-30330c968658\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1d9a27-f6579896d-mpknn"
Jul 29 19:54:31.166815 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:31.166780    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pcwb\" (UniqueName: \"kubernetes.io/projected/a1d6f940-f312-422c-b951-7acb7df5b318-kube-api-access-2pcwb\") pod \"payload-pre-processing-e2e-inf-b-1d9a27-7ff67d944d-wn4k4\" (UID: \"a1d6f940-f312-422c-b951-7acb7df5b318\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d9a27-7ff67d944d-wn4k4"
Jul 29 19:54:31.167080 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:31.166829    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29567083-0305-4296-8baa-30330c968658-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1d9a27-f6579896d-mpknn\" (UID: \"29567083-0305-4296-8baa-30330c968658\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1d9a27-f6579896d-mpknn"
Jul 29 19:54:31.167080 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:31.166855    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1d6f940-f312-422c-b951-7acb7df5b318-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1d9a27-7ff67d944d-wn4k4\" (UID: \"a1d6f940-f312-422c-b951-7acb7df5b318\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d9a27-7ff67d944d-wn4k4"
Jul 29 19:54:31.167618 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:31.167594    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29567083-0305-4296-8baa-30330c968658-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1d9a27-f6579896d-mpknn\" (UID: \"29567083-0305-4296-8baa-30330c968658\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1d9a27-f6579896d-mpknn"
Jul 29 19:54:31.167697 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:31.167616    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1d6f940-f312-422c-b951-7acb7df5b318-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1d9a27-7ff67d944d-wn4k4\" (UID: \"a1d6f940-f312-422c-b951-7acb7df5b318\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d9a27-7ff67d944d-wn4k4"
Jul 29 19:54:31.175342 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:31.175320    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pcwb\" (UniqueName: \"kubernetes.io/projected/a1d6f940-f312-422c-b951-7acb7df5b318-kube-api-access-2pcwb\") pod \"payload-pre-processing-e2e-inf-b-1d9a27-7ff67d944d-wn4k4\" (UID: \"a1d6f940-f312-422c-b951-7acb7df5b318\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d9a27-7ff67d944d-wn4k4"
Jul 29 19:54:31.175430 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:31.175326    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfc7j\" (UniqueName: \"kubernetes.io/projected/29567083-0305-4296-8baa-30330c968658-kube-api-access-qfc7j\") pod \"payload-processing-e2e-inf-b-1d9a27-f6579896d-mpknn\" (UID: \"29567083-0305-4296-8baa-30330c968658\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1d9a27-f6579896d-mpknn"
Jul 29 19:54:31.316181 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:31.316156    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d9a27-7ff67d944d-wn4k4"
Jul 29 19:54:31.323281 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:31.323260    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1d9a27-f6579896d-mpknn"
Jul 29 19:54:31.459829 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:31.459527    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1d9a27-7ff67d944d-wn4k4"]
Jul 29 19:54:31.461570 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:54:31.461537    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1d6f940_f312_422c_b951_7acb7df5b318.slice/crio-3ab41fc47be173bb0016f750d794c7b3d38d25b602f2817662c6c5ddd96d820e WatchSource:0}: Error finding container 3ab41fc47be173bb0016f750d794c7b3d38d25b602f2817662c6c5ddd96d820e: Status 404 returned error can't find the container with id 3ab41fc47be173bb0016f750d794c7b3d38d25b602f2817662c6c5ddd96d820e
Jul 29 19:54:31.698783 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:31.698707    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1d9a27-f6579896d-mpknn"]
Jul 29 19:54:31.701990 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:54:31.701944    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29567083_0305_4296_8baa_30330c968658.slice/crio-6c03ea98944fb6db58d94a9434f7ddbafbd51fc9676491a0a928e23408c72ef8 WatchSource:0}: Error finding container 6c03ea98944fb6db58d94a9434f7ddbafbd51fc9676491a0a928e23408c72ef8: Status 404 returned error can't find the container with id 6c03ea98944fb6db58d94a9434f7ddbafbd51fc9676491a0a928e23408c72ef8
Jul 29 19:54:31.905563 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:31.905520    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d9a27-7ff67d944d-wn4k4" event={"ID":"a1d6f940-f312-422c-b951-7acb7df5b318","Type":"ContainerStarted","Data":"afac48069b8cecc488128f64a81d2f7b1452c99058ba65d09400024a4223ecd0"}
Jul 29 19:54:31.905737 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:31.905581    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d9a27-7ff67d944d-wn4k4" event={"ID":"a1d6f940-f312-422c-b951-7acb7df5b318","Type":"ContainerStarted","Data":"3ab41fc47be173bb0016f750d794c7b3d38d25b602f2817662c6c5ddd96d820e"}
Jul 29 19:54:31.905737 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:31.905606    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d9a27-7ff67d944d-wn4k4"
Jul 29 19:54:31.906980 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:31.906945    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1d9a27-f6579896d-mpknn" event={"ID":"29567083-0305-4296-8baa-30330c968658","Type":"ContainerStarted","Data":"240fc6e911344750f4de1a36be0bbaa800494775a814177eb7b5e6e999dbeb54"}
Jul 29 19:54:31.907104 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:31.907001    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1d9a27-f6579896d-mpknn" event={"ID":"29567083-0305-4296-8baa-30330c968658","Type":"ContainerStarted","Data":"6c03ea98944fb6db58d94a9434f7ddbafbd51fc9676491a0a928e23408c72ef8"}
Jul 29 19:54:31.907104 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:31.907044    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1d9a27-f6579896d-mpknn"
Jul 29 19:54:31.908437 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:31.908416    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9" event={"ID":"47626a9a-94cb-47bc-a34d-288f743d9b04","Type":"ContainerStarted","Data":"82197994cd77fc2bc8ae00e5e175eff7815a4d6bff6f97d6f101e0efb91f303b"}
Jul 29 19:54:31.908549 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:31.908530    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9"
Jul 29 19:54:31.924797 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:31.924748    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d9a27-7ff67d944d-wn4k4" podStartSLOduration=1.924736082 podStartE2EDuration="1.924736082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:54:31.922757043 +0000 UTC m=+2242.750256100" watchObservedRunningTime="2026-07-29 19:54:31.924736082 +0000 UTC m=+2242.752235140"
Jul 29 19:54:31.943686 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:31.943643    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-1d9a27-f6579896d-mpknn" podStartSLOduration=1.94363113 podStartE2EDuration="1.94363113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:54:31.942177351 +0000 UTC m=+2242.769676409" watchObservedRunningTime="2026-07-29 19:54:31.94363113 +0000 UTC m=+2242.771130188"
Jul 29 19:54:31.962573 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:31.962487    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9" podStartSLOduration=1.362046991 podStartE2EDuration="1.962469503s" totalImagesPullingTime="600.422512ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:54:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:54:31.960064314 +0000 UTC m=+2242.787563374" watchObservedRunningTime="2026-07-29 19:54:31.962469503 +0000 UTC m=+2242.789968562"
Jul 29 19:54:37.918981 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:37.918946    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9"
Jul 29 19:54:42.914517 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:42.914475    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d9a27-7ff67d944d-wn4k4" podUID="a1d6f940-f312-422c-b951-7acb7df5b318" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.64:9004: connect: connection refused"
Jul 29 19:54:42.915105 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:42.915027    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1d9a27-f6579896d-mpknn"
Jul 29 19:54:52.913896 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:54:52.913864    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d9a27-7ff67d944d-wn4k4"
Jul 29 19:55:00.137081 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:00.137036    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8"]
Jul 29 19:55:00.141223 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:00.141199    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8"
Jul 29 19:55:00.145320 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:00.145290    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-06a5f2\"/\"default-dockercfg-gm6x4\""
Jul 29 19:55:00.145320 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:00.145312    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-06a5f2\"/\"kube-root-ca.crt\""
Jul 29 19:55:00.145522 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:00.145337    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-06a5f2\"/\"test-model-06a5f2-kserve-self-signed-certs\""
Jul 29 19:55:00.145522 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:00.145364    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-06a5f2\"/\"openshift-service-ca.crt\""
Jul 29 19:55:00.154354 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:00.154324    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8"]
Jul 29 19:55:00.328745 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:00.328704    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2e72342-4853-4a90-ae14-a57bae1441fb-home\") pod \"test-model-06a5f2-kserve-7c94d68589-tsfp8\" (UID: \"d2e72342-4853-4a90-ae14-a57bae1441fb\") " pod="ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8"
Jul 29 19:55:00.328934 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:00.328756    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2e72342-4853-4a90-ae14-a57bae1441fb-model-cache\") pod \"test-model-06a5f2-kserve-7c94d68589-tsfp8\" (UID: \"d2e72342-4853-4a90-ae14-a57bae1441fb\") " pod="ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8"
Jul 29 19:55:00.328934 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:00.328786    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2e72342-4853-4a90-ae14-a57bae1441fb-tmp-dir\") pod \"test-model-06a5f2-kserve-7c94d68589-tsfp8\" (UID: \"d2e72342-4853-4a90-ae14-a57bae1441fb\") " pod="ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8"
Jul 29 19:55:00.328934 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:00.328827    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svbzf\" (UniqueName: \"kubernetes.io/projected/d2e72342-4853-4a90-ae14-a57bae1441fb-kube-api-access-svbzf\") pod \"test-model-06a5f2-kserve-7c94d68589-tsfp8\" (UID: \"d2e72342-4853-4a90-ae14-a57bae1441fb\") " pod="ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8"
Jul 29 19:55:00.328934 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:00.328911    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2e72342-4853-4a90-ae14-a57bae1441fb-dshm\") pod \"test-model-06a5f2-kserve-7c94d68589-tsfp8\" (UID: \"d2e72342-4853-4a90-ae14-a57bae1441fb\") " pod="ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8"
Jul 29 19:55:00.329097 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:00.328951    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2e72342-4853-4a90-ae14-a57bae1441fb-tls-certs\") pod \"test-model-06a5f2-kserve-7c94d68589-tsfp8\" (UID: \"d2e72342-4853-4a90-ae14-a57bae1441fb\") " pod="ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8"
Jul 29 19:55:00.429579 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:00.429476    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2e72342-4853-4a90-ae14-a57bae1441fb-dshm\") pod \"test-model-06a5f2-kserve-7c94d68589-tsfp8\" (UID: \"d2e72342-4853-4a90-ae14-a57bae1441fb\") " pod="ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8"
Jul 29 19:55:00.429579 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:00.429531    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2e72342-4853-4a90-ae14-a57bae1441fb-tls-certs\") pod \"test-model-06a5f2-kserve-7c94d68589-tsfp8\" (UID: \"d2e72342-4853-4a90-ae14-a57bae1441fb\") " pod="ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8"
Jul 29 19:55:00.429789 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:00.429591    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2e72342-4853-4a90-ae14-a57bae1441fb-home\") pod \"test-model-06a5f2-kserve-7c94d68589-tsfp8\" (UID: \"d2e72342-4853-4a90-ae14-a57bae1441fb\") " pod="ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8"
Jul 29 19:55:00.429789 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:00.429618    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2e72342-4853-4a90-ae14-a57bae1441fb-model-cache\") pod \"test-model-06a5f2-kserve-7c94d68589-tsfp8\" (UID: \"d2e72342-4853-4a90-ae14-a57bae1441fb\") " pod="ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8"
Jul 29 19:55:00.429789 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:00.429636    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2e72342-4853-4a90-ae14-a57bae1441fb-tmp-dir\") pod \"test-model-06a5f2-kserve-7c94d68589-tsfp8\" (UID: \"d2e72342-4853-4a90-ae14-a57bae1441fb\") " pod="ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8"
Jul 29 19:55:00.429789 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:00.429677    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svbzf\" (UniqueName: \"kubernetes.io/projected/d2e72342-4853-4a90-ae14-a57bae1441fb-kube-api-access-svbzf\") pod \"test-model-06a5f2-kserve-7c94d68589-tsfp8\" (UID: \"d2e72342-4853-4a90-ae14-a57bae1441fb\") " pod="ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8"
Jul 29 19:55:00.430066 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:00.430040    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2e72342-4853-4a90-ae14-a57bae1441fb-home\") pod \"test-model-06a5f2-kserve-7c94d68589-tsfp8\" (UID: \"d2e72342-4853-4a90-ae14-a57bae1441fb\") " pod="ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8"
Jul 29 19:55:00.430066 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:00.430057    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2e72342-4853-4a90-ae14-a57bae1441fb-model-cache\") pod \"test-model-06a5f2-kserve-7c94d68589-tsfp8\" (UID: \"d2e72342-4853-4a90-ae14-a57bae1441fb\") " pod="ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8"
Jul 29 19:55:00.430188 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:00.430087    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2e72342-4853-4a90-ae14-a57bae1441fb-tmp-dir\") pod \"test-model-06a5f2-kserve-7c94d68589-tsfp8\" (UID: \"d2e72342-4853-4a90-ae14-a57bae1441fb\") " pod="ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8"
Jul 29 19:55:00.431851 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:00.431833    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2e72342-4853-4a90-ae14-a57bae1441fb-dshm\") pod \"test-model-06a5f2-kserve-7c94d68589-tsfp8\" (UID: \"d2e72342-4853-4a90-ae14-a57bae1441fb\") " pod="ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8"
Jul 29 19:55:00.432273 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:00.432252    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2e72342-4853-4a90-ae14-a57bae1441fb-tls-certs\") pod \"test-model-06a5f2-kserve-7c94d68589-tsfp8\" (UID: \"d2e72342-4853-4a90-ae14-a57bae1441fb\") " pod="ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8"
Jul 29 19:55:00.439228 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:00.439204    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svbzf\" (UniqueName: \"kubernetes.io/projected/d2e72342-4853-4a90-ae14-a57bae1441fb-kube-api-access-svbzf\") pod \"test-model-06a5f2-kserve-7c94d68589-tsfp8\" (UID: \"d2e72342-4853-4a90-ae14-a57bae1441fb\") " pod="ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8"
Jul 29 19:55:00.451210 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:00.451187    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8"
Jul 29 19:55:00.587741 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:00.587713    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8"]
Jul 29 19:55:00.589892 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:55:00.589858    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2e72342_4853_4a90_ae14_a57bae1441fb.slice/crio-9fd6f9ca089331f31a608f1992f49966b3ba68e7b0daa28dd3b9824c15ff2fe2 WatchSource:0}: Error finding container 9fd6f9ca089331f31a608f1992f49966b3ba68e7b0daa28dd3b9824c15ff2fe2: Status 404 returned error can't find the container with id 9fd6f9ca089331f31a608f1992f49966b3ba68e7b0daa28dd3b9824c15ff2fe2
Jul 29 19:55:01.060798 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:01.060760    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8" event={"ID":"d2e72342-4853-4a90-ae14-a57bae1441fb","Type":"ContainerStarted","Data":"a55f47b0a354055ba4b0e6d4d7394cabb940694b7cd108b920eec0b80d1d5af6"}
Jul 29 19:55:01.060798 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:01.060805    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8" event={"ID":"d2e72342-4853-4a90-ae14-a57bae1441fb","Type":"ContainerStarted","Data":"9fd6f9ca089331f31a608f1992f49966b3ba68e7b0daa28dd3b9824c15ff2fe2"}
Jul 29 19:55:01.082660 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:01.082585    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8" podStartSLOduration=1.082566902 podStartE2EDuration="1.082566902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:55:01.081187478 +0000 UTC m=+2271.908686536" watchObservedRunningTime="2026-07-29 19:55:01.082566902 +0000 UTC m=+2271.910065961"
Jul 29 19:55:10.451659 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:10.451602    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8"
Jul 29 19:55:10.451659 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:10.451669    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8"
Jul 29 19:55:10.464359 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:10.464291    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8"
Jul 29 19:55:11.128002 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:11.127405    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8"
Jul 29 19:55:44.246509 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:44.246476    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h"]
Jul 29 19:55:44.252616 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:44.252579    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h"
Jul 29 19:55:44.257833 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:44.257797    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1d9a27\"/\"openshift-service-ca.crt\""
Jul 29 19:55:44.257833 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:44.257821    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1d9a27\"/\"test-model-1d9a27-kserve-self-signed-certs\""
Jul 29 19:55:44.258085 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:44.257884    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1d9a27\"/\"default-dockercfg-55pl6\""
Jul 29 19:55:44.258085 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:44.258022    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1d9a27\"/\"kube-root-ca.crt\""
Jul 29 19:55:44.262207 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:44.262184    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h"]
Jul 29 19:55:44.431263 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:44.431216    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fff1b56-4ab6-4e24-bb66-17449d173461-tls-certs\") pod \"test-model-1d9a27-kserve-7476fccdf-ql94h\" (UID: \"5fff1b56-4ab6-4e24-bb66-17449d173461\") " pod="ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h"
Jul 29 19:55:44.431459 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:44.431295    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fff1b56-4ab6-4e24-bb66-17449d173461-home\") pod \"test-model-1d9a27-kserve-7476fccdf-ql94h\" (UID: \"5fff1b56-4ab6-4e24-bb66-17449d173461\") " pod="ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h"
Jul 29 19:55:44.431459 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:44.431334    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87t9w\" (UniqueName: \"kubernetes.io/projected/5fff1b56-4ab6-4e24-bb66-17449d173461-kube-api-access-87t9w\") pod \"test-model-1d9a27-kserve-7476fccdf-ql94h\" (UID: \"5fff1b56-4ab6-4e24-bb66-17449d173461\") " pod="ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h"
Jul 29 19:55:44.431552 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:44.431477    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fff1b56-4ab6-4e24-bb66-17449d173461-tmp-dir\") pod \"test-model-1d9a27-kserve-7476fccdf-ql94h\" (UID: \"5fff1b56-4ab6-4e24-bb66-17449d173461\") " pod="ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h"
Jul 29 19:55:44.431552 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:44.431498    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fff1b56-4ab6-4e24-bb66-17449d173461-dshm\") pod \"test-model-1d9a27-kserve-7476fccdf-ql94h\" (UID: \"5fff1b56-4ab6-4e24-bb66-17449d173461\") " pod="ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h"
Jul 29 19:55:44.431552 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:44.431514    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fff1b56-4ab6-4e24-bb66-17449d173461-model-cache\") pod \"test-model-1d9a27-kserve-7476fccdf-ql94h\" (UID: \"5fff1b56-4ab6-4e24-bb66-17449d173461\") " pod="ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h"
Jul 29 19:55:44.532095 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:44.531999    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fff1b56-4ab6-4e24-bb66-17449d173461-tmp-dir\") pod \"test-model-1d9a27-kserve-7476fccdf-ql94h\" (UID: \"5fff1b56-4ab6-4e24-bb66-17449d173461\") " pod="ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h"
Jul 29 19:55:44.532095 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:44.532044    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fff1b56-4ab6-4e24-bb66-17449d173461-dshm\") pod \"test-model-1d9a27-kserve-7476fccdf-ql94h\" (UID: \"5fff1b56-4ab6-4e24-bb66-17449d173461\") " pod="ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h"
Jul 29 19:55:44.532095 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:44.532065    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fff1b56-4ab6-4e24-bb66-17449d173461-model-cache\") pod \"test-model-1d9a27-kserve-7476fccdf-ql94h\" (UID: \"5fff1b56-4ab6-4e24-bb66-17449d173461\") " pod="ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h"
Jul 29 19:55:44.532095 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:44.532089    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fff1b56-4ab6-4e24-bb66-17449d173461-tls-certs\") pod \"test-model-1d9a27-kserve-7476fccdf-ql94h\" (UID: \"5fff1b56-4ab6-4e24-bb66-17449d173461\") " pod="ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h"
Jul 29 19:55:44.532454 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:44.532116    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fff1b56-4ab6-4e24-bb66-17449d173461-home\") pod \"test-model-1d9a27-kserve-7476fccdf-ql94h\" (UID: \"5fff1b56-4ab6-4e24-bb66-17449d173461\") " pod="ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h"
Jul 29 19:55:44.532454 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:44.532145    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87t9w\" (UniqueName: \"kubernetes.io/projected/5fff1b56-4ab6-4e24-bb66-17449d173461-kube-api-access-87t9w\") pod \"test-model-1d9a27-kserve-7476fccdf-ql94h\" (UID: \"5fff1b56-4ab6-4e24-bb66-17449d173461\") " pod="ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h"
Jul 29 19:55:44.532454 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:44.532435    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fff1b56-4ab6-4e24-bb66-17449d173461-tmp-dir\") pod \"test-model-1d9a27-kserve-7476fccdf-ql94h\" (UID: \"5fff1b56-4ab6-4e24-bb66-17449d173461\") " pod="ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h"
Jul 29 19:55:44.532602 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:44.532496    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fff1b56-4ab6-4e24-bb66-17449d173461-model-cache\") pod \"test-model-1d9a27-kserve-7476fccdf-ql94h\" (UID: \"5fff1b56-4ab6-4e24-bb66-17449d173461\") " pod="ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h"
Jul 29 19:55:44.532644 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:44.532615    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fff1b56-4ab6-4e24-bb66-17449d173461-home\") pod \"test-model-1d9a27-kserve-7476fccdf-ql94h\" (UID: \"5fff1b56-4ab6-4e24-bb66-17449d173461\") " pod="ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h"
Jul 29 19:55:44.534401 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:44.534377    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fff1b56-4ab6-4e24-bb66-17449d173461-dshm\") pod \"test-model-1d9a27-kserve-7476fccdf-ql94h\" (UID: \"5fff1b56-4ab6-4e24-bb66-17449d173461\") " pod="ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h"
Jul 29 19:55:44.534724 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:44.534701    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fff1b56-4ab6-4e24-bb66-17449d173461-tls-certs\") pod \"test-model-1d9a27-kserve-7476fccdf-ql94h\" (UID: \"5fff1b56-4ab6-4e24-bb66-17449d173461\") " pod="ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h"
Jul 29 19:55:44.540566 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:44.540541    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87t9w\" (UniqueName: \"kubernetes.io/projected/5fff1b56-4ab6-4e24-bb66-17449d173461-kube-api-access-87t9w\") pod \"test-model-1d9a27-kserve-7476fccdf-ql94h\" (UID: \"5fff1b56-4ab6-4e24-bb66-17449d173461\") " pod="ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h"
Jul 29 19:55:44.563916 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:44.563892    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h"
Jul 29 19:55:44.704487 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:44.704458    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h"]
Jul 29 19:55:44.706597 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:55:44.706568    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fff1b56_4ab6_4e24_bb66_17449d173461.slice/crio-dc10d4e8ea089332bf15033ae493fa156341e81c19e41f3c8fa8f4d1ae18bb2c WatchSource:0}: Error finding container dc10d4e8ea089332bf15033ae493fa156341e81c19e41f3c8fa8f4d1ae18bb2c: Status 404 returned error can't find the container with id dc10d4e8ea089332bf15033ae493fa156341e81c19e41f3c8fa8f4d1ae18bb2c
Jul 29 19:55:45.282884 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:45.282846    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h" event={"ID":"5fff1b56-4ab6-4e24-bb66-17449d173461","Type":"ContainerStarted","Data":"ff014c6e863d7d966dd277500f559f67206729eb025c5eb0057fc0ac97f4bda3"}
Jul 29 19:55:45.282884 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:45.282890    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h" event={"ID":"5fff1b56-4ab6-4e24-bb66-17449d173461","Type":"ContainerStarted","Data":"dc10d4e8ea089332bf15033ae493fa156341e81c19e41f3c8fa8f4d1ae18bb2c"}
Jul 29 19:55:45.304509 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:45.304456    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h" podStartSLOduration=1.304439928 podStartE2EDuration="1.304439928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:55:45.301568953 +0000 UTC m=+2316.129068024" watchObservedRunningTime="2026-07-29 19:55:45.304439928 +0000 UTC m=+2316.131938985"
Jul 29 19:55:54.564604 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:54.564565    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h"
Jul 29 19:55:54.567154 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:54.564620    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h"
Jul 29 19:55:54.577328 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:54.577310    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h"
Jul 29 19:55:55.343600 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:55:55.343567    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h"
Jul 29 19:56:12.364048 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:12.364016    2557 scope.go:117] "RemoveContainer" containerID="6eb2bffff4eaa73438927a8eb350822dbc86109d8592ea043a5a355906c8daca"
Jul 29 19:56:34.083484 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:34.083449    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8"]
Jul 29 19:56:34.980094 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:34.980061    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h"]
Jul 29 19:56:45.543496 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:45.543459    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c"]
Jul 29 19:56:45.543910 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:45.543677    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c" podUID="a87fc48e-7f52-48e6-9c3c-92216470881d" containerName="maas-api" containerID="cri-o://1aa966efbb811226ac666ee9fd190ec0b3be4ebf880641e5245bdf0de112f700" gracePeriod=30
Jul 29 19:56:45.601408 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:45.601376    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-06a5f2-5bc74bc4c4-l6dgh"]
Jul 29 19:56:45.601594 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:45.601569    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-06a5f2-5bc74bc4c4-l6dgh" podUID="0965cf59-dee6-4391-a0f6-e84d807677b6" containerName="payload-processing" containerID="cri-o://5f5a8c96f00e85490ef442e3ecd3d99b11a5d95aa5d08314255fd67ef0fb0be1" gracePeriod=30
Jul 29 19:56:45.614121 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:45.614095    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-06a5f2-7fd7464688-t4pxl"]
Jul 29 19:56:45.614381 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:45.614332    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-06a5f2-7fd7464688-t4pxl" podUID="a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a" containerName="payload-pre-processing" containerID="cri-o://fda984b0a63f8eabaaabebeccbccb1294bd5c046f654781a69800496b2a773ba" gracePeriod=30
Jul 29 19:56:45.900790 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:45.900766    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-06a5f2-5bc74bc4c4-l6dgh"
Jul 29 19:56:45.903937 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:45.903914    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-06a5f2-7fd7464688-t4pxl"
Jul 29 19:56:45.987628 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:45.987597    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c"
Jul 29 19:56:45.996504 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:45.996477    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ps7lb\" (UniqueName: \"kubernetes.io/projected/0965cf59-dee6-4391-a0f6-e84d807677b6-kube-api-access-ps7lb\") pod \"0965cf59-dee6-4391-a0f6-e84d807677b6\" (UID: \"0965cf59-dee6-4391-a0f6-e84d807677b6\") "
Jul 29 19:56:45.996635 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:45.996544    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0965cf59-dee6-4391-a0f6-e84d807677b6-plugins-config-volume\") pod \"0965cf59-dee6-4391-a0f6-e84d807677b6\" (UID: \"0965cf59-dee6-4391-a0f6-e84d807677b6\") "
Jul 29 19:56:45.996687 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:45.996639    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a-plugins-config-volume\") pod \"a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a\" (UID: \"a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a\") "
Jul 29 19:56:45.996738 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:45.996691    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7n5pz\" (UniqueName: \"kubernetes.io/projected/a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a-kube-api-access-7n5pz\") pod \"a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a\" (UID: \"a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a\") "
Jul 29 19:56:45.996995 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:45.996955    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0965cf59-dee6-4391-a0f6-e84d807677b6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0965cf59-dee6-4391-a0f6-e84d807677b6" (UID: "0965cf59-dee6-4391-a0f6-e84d807677b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:56:45.996995 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:45.996988    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a" (UID: "a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:56:45.998581 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:45.998561    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0965cf59-dee6-4391-a0f6-e84d807677b6-kube-api-access-ps7lb" (OuterVolumeSpecName: "kube-api-access-ps7lb") pod "0965cf59-dee6-4391-a0f6-e84d807677b6" (UID: "0965cf59-dee6-4391-a0f6-e84d807677b6"). InnerVolumeSpecName "kube-api-access-ps7lb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:56:45.998726 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:45.998705    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a-kube-api-access-7n5pz" (OuterVolumeSpecName: "kube-api-access-7n5pz") pod "a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a" (UID: "a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a"). InnerVolumeSpecName "kube-api-access-7n5pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:56:46.097897 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.097807    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a87fc48e-7f52-48e6-9c3c-92216470881d-openshift-service-ca\") pod \"a87fc48e-7f52-48e6-9c3c-92216470881d\" (UID: \"a87fc48e-7f52-48e6-9c3c-92216470881d\") "
Jul 29 19:56:46.097897 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.097892    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a87fc48e-7f52-48e6-9c3c-92216470881d-maas-api-tls\") pod \"a87fc48e-7f52-48e6-9c3c-92216470881d\" (UID: \"a87fc48e-7f52-48e6-9c3c-92216470881d\") "
Jul 29 19:56:46.098132 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.097940    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjcgp\" (UniqueName: \"kubernetes.io/projected/a87fc48e-7f52-48e6-9c3c-92216470881d-kube-api-access-sjcgp\") pod \"a87fc48e-7f52-48e6-9c3c-92216470881d\" (UID: \"a87fc48e-7f52-48e6-9c3c-92216470881d\") "
Jul 29 19:56:46.098168 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.098160    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0965cf59-dee6-4391-a0f6-e84d807677b6-plugins-config-volume\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:56:46.098209 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.098174    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a-plugins-config-volume\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:56:46.098209 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.098185    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n5pz\" (UniqueName: \"kubernetes.io/projected/a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a-kube-api-access-7n5pz\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:56:46.098209 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.098193    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps7lb\" (UniqueName: \"kubernetes.io/projected/0965cf59-dee6-4391-a0f6-e84d807677b6-kube-api-access-ps7lb\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:56:46.098312 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.098220    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a87fc48e-7f52-48e6-9c3c-92216470881d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a87fc48e-7f52-48e6-9c3c-92216470881d" (UID: "a87fc48e-7f52-48e6-9c3c-92216470881d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:56:46.100111 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.100080    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a87fc48e-7f52-48e6-9c3c-92216470881d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a87fc48e-7f52-48e6-9c3c-92216470881d" (UID: "a87fc48e-7f52-48e6-9c3c-92216470881d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:56:46.100223 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.100131    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a87fc48e-7f52-48e6-9c3c-92216470881d-kube-api-access-sjcgp" (OuterVolumeSpecName: "kube-api-access-sjcgp") pod "a87fc48e-7f52-48e6-9c3c-92216470881d" (UID: "a87fc48e-7f52-48e6-9c3c-92216470881d"). InnerVolumeSpecName "kube-api-access-sjcgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:56:46.199110 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.199079    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjcgp\" (UniqueName: \"kubernetes.io/projected/a87fc48e-7f52-48e6-9c3c-92216470881d-kube-api-access-sjcgp\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:56:46.199110 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.199104    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a87fc48e-7f52-48e6-9c3c-92216470881d-openshift-service-ca\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:56:46.199110 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.199116    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a87fc48e-7f52-48e6-9c3c-92216470881d-maas-api-tls\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:56:46.583642 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.583603    2557 generic.go:356] "Generic (PLEG): container finished" podID="a87fc48e-7f52-48e6-9c3c-92216470881d" containerID="1aa966efbb811226ac666ee9fd190ec0b3be4ebf880641e5245bdf0de112f700" exitCode=0
Jul 29 19:56:46.584106 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.583652    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c"
Jul 29 19:56:46.584106 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.583653    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c" event={"ID":"a87fc48e-7f52-48e6-9c3c-92216470881d","Type":"ContainerDied","Data":"1aa966efbb811226ac666ee9fd190ec0b3be4ebf880641e5245bdf0de112f700"}
Jul 29 19:56:46.584106 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.583709    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c" event={"ID":"a87fc48e-7f52-48e6-9c3c-92216470881d","Type":"ContainerDied","Data":"9036960391c7c616114900165857c36557caf89186293528aea310799ea1b72c"}
Jul 29 19:56:46.584106 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.583736    2557 scope.go:117] "RemoveContainer" containerID="1aa966efbb811226ac666ee9fd190ec0b3be4ebf880641e5245bdf0de112f700"
Jul 29 19:56:46.585100 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.585078    2557 generic.go:356] "Generic (PLEG): container finished" podID="a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a" containerID="fda984b0a63f8eabaaabebeccbccb1294bd5c046f654781a69800496b2a773ba" exitCode=0
Jul 29 19:56:46.585171 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.585116    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-06a5f2-7fd7464688-t4pxl"
Jul 29 19:56:46.585218 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.585164    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-06a5f2-7fd7464688-t4pxl" event={"ID":"a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a","Type":"ContainerDied","Data":"fda984b0a63f8eabaaabebeccbccb1294bd5c046f654781a69800496b2a773ba"}
Jul 29 19:56:46.585275 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.585217    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-06a5f2-7fd7464688-t4pxl" event={"ID":"a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a","Type":"ContainerDied","Data":"4e52c91bcc461687e2882f229cc0911b8b3ab2e61e6c0040935544c744bddd16"}
Jul 29 19:56:46.586621 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.586582    2557 generic.go:356] "Generic (PLEG): container finished" podID="0965cf59-dee6-4391-a0f6-e84d807677b6" containerID="5f5a8c96f00e85490ef442e3ecd3d99b11a5d95aa5d08314255fd67ef0fb0be1" exitCode=0
Jul 29 19:56:46.586797 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.586638    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-06a5f2-5bc74bc4c4-l6dgh"
Jul 29 19:56:46.587022 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.586639    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-06a5f2-5bc74bc4c4-l6dgh" event={"ID":"0965cf59-dee6-4391-a0f6-e84d807677b6","Type":"ContainerDied","Data":"5f5a8c96f00e85490ef442e3ecd3d99b11a5d95aa5d08314255fd67ef0fb0be1"}
Jul 29 19:56:46.587228 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.587193    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-06a5f2-5bc74bc4c4-l6dgh" event={"ID":"0965cf59-dee6-4391-a0f6-e84d807677b6","Type":"ContainerDied","Data":"cc4066be92212483fdaf61c6aef08ccf9d474153d5f03558724c4ed7fc030d63"}
Jul 29 19:56:46.594818 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.594790    2557 scope.go:117] "RemoveContainer" containerID="1aa966efbb811226ac666ee9fd190ec0b3be4ebf880641e5245bdf0de112f700"
Jul 29 19:56:46.599102 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:56:46.599075    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aa966efbb811226ac666ee9fd190ec0b3be4ebf880641e5245bdf0de112f700\": container with ID starting with 1aa966efbb811226ac666ee9fd190ec0b3be4ebf880641e5245bdf0de112f700 not found: ID does not exist" containerID="1aa966efbb811226ac666ee9fd190ec0b3be4ebf880641e5245bdf0de112f700"
Jul 29 19:56:46.599201 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.599114    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aa966efbb811226ac666ee9fd190ec0b3be4ebf880641e5245bdf0de112f700"} err="failed to get container status \"1aa966efbb811226ac666ee9fd190ec0b3be4ebf880641e5245bdf0de112f700\": rpc error: code = NotFound desc = could not find container \"1aa966efbb811226ac666ee9fd190ec0b3be4ebf880641e5245bdf0de112f700\": container with ID starting with 1aa966efbb811226ac666ee9fd190ec0b3be4ebf880641e5245bdf0de112f700 not found: ID does not exist"
Jul 29 19:56:46.599201 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.599138    2557 scope.go:117] "RemoveContainer" containerID="fda984b0a63f8eabaaabebeccbccb1294bd5c046f654781a69800496b2a773ba"
Jul 29 19:56:46.607760 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.607743    2557 scope.go:117] "RemoveContainer" containerID="fda984b0a63f8eabaaabebeccbccb1294bd5c046f654781a69800496b2a773ba"
Jul 29 19:56:46.608086 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:56:46.608065    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fda984b0a63f8eabaaabebeccbccb1294bd5c046f654781a69800496b2a773ba\": container with ID starting with fda984b0a63f8eabaaabebeccbccb1294bd5c046f654781a69800496b2a773ba not found: ID does not exist" containerID="fda984b0a63f8eabaaabebeccbccb1294bd5c046f654781a69800496b2a773ba"
Jul 29 19:56:46.608151 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.608092    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fda984b0a63f8eabaaabebeccbccb1294bd5c046f654781a69800496b2a773ba"} err="failed to get container status \"fda984b0a63f8eabaaabebeccbccb1294bd5c046f654781a69800496b2a773ba\": rpc error: code = NotFound desc = could not find container \"fda984b0a63f8eabaaabebeccbccb1294bd5c046f654781a69800496b2a773ba\": container with ID starting with fda984b0a63f8eabaaabebeccbccb1294bd5c046f654781a69800496b2a773ba not found: ID does not exist"
Jul 29 19:56:46.608151 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.608109    2557 scope.go:117] "RemoveContainer" containerID="5f5a8c96f00e85490ef442e3ecd3d99b11a5d95aa5d08314255fd67ef0fb0be1"
Jul 29 19:56:46.617242 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.617220    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c"]
Jul 29 19:56:46.622088 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.622068    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-06a5f2-77ccc685c5-9dx4c"]
Jul 29 19:56:46.629352 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.629263    2557 scope.go:117] "RemoveContainer" containerID="5f5a8c96f00e85490ef442e3ecd3d99b11a5d95aa5d08314255fd67ef0fb0be1"
Jul 29 19:56:46.630524 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:56:46.630501    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f5a8c96f00e85490ef442e3ecd3d99b11a5d95aa5d08314255fd67ef0fb0be1\": container with ID starting with 5f5a8c96f00e85490ef442e3ecd3d99b11a5d95aa5d08314255fd67ef0fb0be1 not found: ID does not exist" containerID="5f5a8c96f00e85490ef442e3ecd3d99b11a5d95aa5d08314255fd67ef0fb0be1"
Jul 29 19:56:46.630611 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.630528    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f5a8c96f00e85490ef442e3ecd3d99b11a5d95aa5d08314255fd67ef0fb0be1"} err="failed to get container status \"5f5a8c96f00e85490ef442e3ecd3d99b11a5d95aa5d08314255fd67ef0fb0be1\": rpc error: code = NotFound desc = could not find container \"5f5a8c96f00e85490ef442e3ecd3d99b11a5d95aa5d08314255fd67ef0fb0be1\": container with ID starting with 5f5a8c96f00e85490ef442e3ecd3d99b11a5d95aa5d08314255fd67ef0fb0be1 not found: ID does not exist"
Jul 29 19:56:46.633815 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.633794    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-06a5f2-7fd7464688-t4pxl"]
Jul 29 19:56:46.638269 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.638248    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-06a5f2-7fd7464688-t4pxl"]
Jul 29 19:56:46.649742 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.649722    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-06a5f2-5bc74bc4c4-l6dgh"]
Jul 29 19:56:46.655052 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:46.655032    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-06a5f2-5bc74bc4c4-l6dgh"]
Jul 29 19:56:47.678770 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:47.678734    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0965cf59-dee6-4391-a0f6-e84d807677b6" path="/var/lib/kubelet/pods/0965cf59-dee6-4391-a0f6-e84d807677b6/volumes"
Jul 29 19:56:47.679147 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:47.679132    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a" path="/var/lib/kubelet/pods/a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a/volumes"
Jul 29 19:56:47.679501 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:47.679486    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a87fc48e-7f52-48e6-9c3c-92216470881d" path="/var/lib/kubelet/pods/a87fc48e-7f52-48e6-9c3c-92216470881d/volumes"
Jul 29 19:56:49.102486 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:49.102437    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8" podUID="d2e72342-4853-4a90-ae14-a57bae1441fb" containerName="main" containerID="cri-o://a55f47b0a354055ba4b0e6d4d7394cabb940694b7cd108b920eec0b80d1d5af6" gracePeriod=45
Jul 29 19:56:49.356105 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:49.356049    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8"
Jul 29 19:56:49.428387 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:49.428353    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2e72342-4853-4a90-ae14-a57bae1441fb-dshm\") pod \"d2e72342-4853-4a90-ae14-a57bae1441fb\" (UID: \"d2e72342-4853-4a90-ae14-a57bae1441fb\") "
Jul 29 19:56:49.428556 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:49.428407    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2e72342-4853-4a90-ae14-a57bae1441fb-home\") pod \"d2e72342-4853-4a90-ae14-a57bae1441fb\" (UID: \"d2e72342-4853-4a90-ae14-a57bae1441fb\") "
Jul 29 19:56:49.428556 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:49.428432    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2e72342-4853-4a90-ae14-a57bae1441fb-model-cache\") pod \"d2e72342-4853-4a90-ae14-a57bae1441fb\" (UID: \"d2e72342-4853-4a90-ae14-a57bae1441fb\") "
Jul 29 19:56:49.428556 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:49.428460    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2e72342-4853-4a90-ae14-a57bae1441fb-tmp-dir\") pod \"d2e72342-4853-4a90-ae14-a57bae1441fb\" (UID: \"d2e72342-4853-4a90-ae14-a57bae1441fb\") "
Jul 29 19:56:49.428556 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:49.428489    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svbzf\" (UniqueName: \"kubernetes.io/projected/d2e72342-4853-4a90-ae14-a57bae1441fb-kube-api-access-svbzf\") pod \"d2e72342-4853-4a90-ae14-a57bae1441fb\" (UID: \"d2e72342-4853-4a90-ae14-a57bae1441fb\") "
Jul 29 19:56:49.428556 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:49.428536    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2e72342-4853-4a90-ae14-a57bae1441fb-tls-certs\") pod \"d2e72342-4853-4a90-ae14-a57bae1441fb\" (UID: \"d2e72342-4853-4a90-ae14-a57bae1441fb\") "
Jul 29 19:56:49.428814 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:49.428700    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e72342-4853-4a90-ae14-a57bae1441fb-home" (OuterVolumeSpecName: "home") pod "d2e72342-4853-4a90-ae14-a57bae1441fb" (UID: "d2e72342-4853-4a90-ae14-a57bae1441fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:56:49.428814 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:49.428692    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e72342-4853-4a90-ae14-a57bae1441fb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d2e72342-4853-4a90-ae14-a57bae1441fb" (UID: "d2e72342-4853-4a90-ae14-a57bae1441fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:56:49.428814 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:49.428716    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e72342-4853-4a90-ae14-a57bae1441fb-model-cache" (OuterVolumeSpecName: "model-cache") pod "d2e72342-4853-4a90-ae14-a57bae1441fb" (UID: "d2e72342-4853-4a90-ae14-a57bae1441fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:56:49.430616 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:49.430587    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2e72342-4853-4a90-ae14-a57bae1441fb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d2e72342-4853-4a90-ae14-a57bae1441fb" (UID: "d2e72342-4853-4a90-ae14-a57bae1441fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:56:49.430761 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:49.430632    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e72342-4853-4a90-ae14-a57bae1441fb-dshm" (OuterVolumeSpecName: "dshm") pod "d2e72342-4853-4a90-ae14-a57bae1441fb" (UID: "d2e72342-4853-4a90-ae14-a57bae1441fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:56:49.430761 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:49.430660    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2e72342-4853-4a90-ae14-a57bae1441fb-kube-api-access-svbzf" (OuterVolumeSpecName: "kube-api-access-svbzf") pod "d2e72342-4853-4a90-ae14-a57bae1441fb" (UID: "d2e72342-4853-4a90-ae14-a57bae1441fb"). InnerVolumeSpecName "kube-api-access-svbzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:56:49.530004 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:49.529953    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2e72342-4853-4a90-ae14-a57bae1441fb-home\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:56:49.530004 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:49.530000    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2e72342-4853-4a90-ae14-a57bae1441fb-model-cache\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:56:49.530004 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:49.530009    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2e72342-4853-4a90-ae14-a57bae1441fb-tmp-dir\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:56:49.530216 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:49.530019    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svbzf\" (UniqueName: \"kubernetes.io/projected/d2e72342-4853-4a90-ae14-a57bae1441fb-kube-api-access-svbzf\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:56:49.530216 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:49.530029    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2e72342-4853-4a90-ae14-a57bae1441fb-tls-certs\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:56:49.530216 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:49.530036    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2e72342-4853-4a90-ae14-a57bae1441fb-dshm\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:56:49.604397 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:49.604366    2557 generic.go:356] "Generic (PLEG): container finished" podID="d2e72342-4853-4a90-ae14-a57bae1441fb" containerID="a55f47b0a354055ba4b0e6d4d7394cabb940694b7cd108b920eec0b80d1d5af6" exitCode=0
Jul 29 19:56:49.604543 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:49.604405    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8"
Jul 29 19:56:49.604543 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:49.604417    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8" event={"ID":"d2e72342-4853-4a90-ae14-a57bae1441fb","Type":"ContainerDied","Data":"a55f47b0a354055ba4b0e6d4d7394cabb940694b7cd108b920eec0b80d1d5af6"}
Jul 29 19:56:49.604543 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:49.604449    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8" event={"ID":"d2e72342-4853-4a90-ae14-a57bae1441fb","Type":"ContainerDied","Data":"9fd6f9ca089331f31a608f1992f49966b3ba68e7b0daa28dd3b9824c15ff2fe2"}
Jul 29 19:56:49.604543 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:49.604465    2557 scope.go:117] "RemoveContainer" containerID="a55f47b0a354055ba4b0e6d4d7394cabb940694b7cd108b920eec0b80d1d5af6"
Jul 29 19:56:49.613120 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:49.613102    2557 scope.go:117] "RemoveContainer" containerID="a55f47b0a354055ba4b0e6d4d7394cabb940694b7cd108b920eec0b80d1d5af6"
Jul 29 19:56:49.613364 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:56:49.613345    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a55f47b0a354055ba4b0e6d4d7394cabb940694b7cd108b920eec0b80d1d5af6\": container with ID starting with a55f47b0a354055ba4b0e6d4d7394cabb940694b7cd108b920eec0b80d1d5af6 not found: ID does not exist" containerID="a55f47b0a354055ba4b0e6d4d7394cabb940694b7cd108b920eec0b80d1d5af6"
Jul 29 19:56:49.613434 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:49.613378    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a55f47b0a354055ba4b0e6d4d7394cabb940694b7cd108b920eec0b80d1d5af6"} err="failed to get container status \"a55f47b0a354055ba4b0e6d4d7394cabb940694b7cd108b920eec0b80d1d5af6\": rpc error: code = NotFound desc = could not find container \"a55f47b0a354055ba4b0e6d4d7394cabb940694b7cd108b920eec0b80d1d5af6\": container with ID starting with a55f47b0a354055ba4b0e6d4d7394cabb940694b7cd108b920eec0b80d1d5af6 not found: ID does not exist"
Jul 29 19:56:49.625125 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:49.625103    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8"]
Jul 29 19:56:49.628982 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:49.628950    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-06a5f2/test-model-06a5f2-kserve-7c94d68589-tsfp8"]
Jul 29 19:56:49.678045 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:49.678022    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2e72342-4853-4a90-ae14-a57bae1441fb" path="/var/lib/kubelet/pods/d2e72342-4853-4a90-ae14-a57bae1441fb/volumes"
Jul 29 19:56:49.999092 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:49.999050    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h" podUID="5fff1b56-4ab6-4e24-bb66-17449d173461" containerName="main" containerID="cri-o://ff014c6e863d7d966dd277500f559f67206729eb025c5eb0057fc0ac97f4bda3" gracePeriod=45
Jul 29 19:56:50.253760 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:50.253701    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h"
Jul 29 19:56:50.337218 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:50.337181    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fff1b56-4ab6-4e24-bb66-17449d173461-tls-certs\") pod \"5fff1b56-4ab6-4e24-bb66-17449d173461\" (UID: \"5fff1b56-4ab6-4e24-bb66-17449d173461\") "
Jul 29 19:56:50.337218 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:50.337218    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fff1b56-4ab6-4e24-bb66-17449d173461-tmp-dir\") pod \"5fff1b56-4ab6-4e24-bb66-17449d173461\" (UID: \"5fff1b56-4ab6-4e24-bb66-17449d173461\") "
Jul 29 19:56:50.337480 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:50.337237    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fff1b56-4ab6-4e24-bb66-17449d173461-dshm\") pod \"5fff1b56-4ab6-4e24-bb66-17449d173461\" (UID: \"5fff1b56-4ab6-4e24-bb66-17449d173461\") "
Jul 29 19:56:50.337480 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:50.337260    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87t9w\" (UniqueName: \"kubernetes.io/projected/5fff1b56-4ab6-4e24-bb66-17449d173461-kube-api-access-87t9w\") pod \"5fff1b56-4ab6-4e24-bb66-17449d173461\" (UID: \"5fff1b56-4ab6-4e24-bb66-17449d173461\") "
Jul 29 19:56:50.337480 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:50.337319    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fff1b56-4ab6-4e24-bb66-17449d173461-home\") pod \"5fff1b56-4ab6-4e24-bb66-17449d173461\" (UID: \"5fff1b56-4ab6-4e24-bb66-17449d173461\") "
Jul 29 19:56:50.337480 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:50.337369    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fff1b56-4ab6-4e24-bb66-17449d173461-model-cache\") pod \"5fff1b56-4ab6-4e24-bb66-17449d173461\" (UID: \"5fff1b56-4ab6-4e24-bb66-17449d173461\") "
Jul 29 19:56:50.337702 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:50.337539    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fff1b56-4ab6-4e24-bb66-17449d173461-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5fff1b56-4ab6-4e24-bb66-17449d173461" (UID: "5fff1b56-4ab6-4e24-bb66-17449d173461"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:56:50.337702 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:50.337685    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fff1b56-4ab6-4e24-bb66-17449d173461-home" (OuterVolumeSpecName: "home") pod "5fff1b56-4ab6-4e24-bb66-17449d173461" (UID: "5fff1b56-4ab6-4e24-bb66-17449d173461"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:56:50.337807 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:50.337728    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fff1b56-4ab6-4e24-bb66-17449d173461-tmp-dir\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:56:50.337807 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:50.337777    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fff1b56-4ab6-4e24-bb66-17449d173461-model-cache" (OuterVolumeSpecName: "model-cache") pod "5fff1b56-4ab6-4e24-bb66-17449d173461" (UID: "5fff1b56-4ab6-4e24-bb66-17449d173461"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:56:50.339637 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:50.339603    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fff1b56-4ab6-4e24-bb66-17449d173461-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5fff1b56-4ab6-4e24-bb66-17449d173461" (UID: "5fff1b56-4ab6-4e24-bb66-17449d173461"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:56:50.339637 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:50.339610    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fff1b56-4ab6-4e24-bb66-17449d173461-kube-api-access-87t9w" (OuterVolumeSpecName: "kube-api-access-87t9w") pod "5fff1b56-4ab6-4e24-bb66-17449d173461" (UID: "5fff1b56-4ab6-4e24-bb66-17449d173461"). InnerVolumeSpecName "kube-api-access-87t9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:56:50.339849 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:50.339662    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fff1b56-4ab6-4e24-bb66-17449d173461-dshm" (OuterVolumeSpecName: "dshm") pod "5fff1b56-4ab6-4e24-bb66-17449d173461" (UID: "5fff1b56-4ab6-4e24-bb66-17449d173461"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:56:50.438364 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:50.438332    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fff1b56-4ab6-4e24-bb66-17449d173461-tls-certs\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:56:50.438364 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:50.438361    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fff1b56-4ab6-4e24-bb66-17449d173461-dshm\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:56:50.438545 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:50.438371    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87t9w\" (UniqueName: \"kubernetes.io/projected/5fff1b56-4ab6-4e24-bb66-17449d173461-kube-api-access-87t9w\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:56:50.438545 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:50.438381    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fff1b56-4ab6-4e24-bb66-17449d173461-home\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:56:50.438545 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:50.438390    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fff1b56-4ab6-4e24-bb66-17449d173461-model-cache\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:56:50.611526 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:50.611427    2557 generic.go:356] "Generic (PLEG): container finished" podID="5fff1b56-4ab6-4e24-bb66-17449d173461" containerID="ff014c6e863d7d966dd277500f559f67206729eb025c5eb0057fc0ac97f4bda3" exitCode=0
Jul 29 19:56:50.611526 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:50.611470    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h" event={"ID":"5fff1b56-4ab6-4e24-bb66-17449d173461","Type":"ContainerDied","Data":"ff014c6e863d7d966dd277500f559f67206729eb025c5eb0057fc0ac97f4bda3"}
Jul 29 19:56:50.611526 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:50.611474    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h"
Jul 29 19:56:50.611526 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:50.611499    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h" event={"ID":"5fff1b56-4ab6-4e24-bb66-17449d173461","Type":"ContainerDied","Data":"dc10d4e8ea089332bf15033ae493fa156341e81c19e41f3c8fa8f4d1ae18bb2c"}
Jul 29 19:56:50.611526 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:50.611520    2557 scope.go:117] "RemoveContainer" containerID="ff014c6e863d7d966dd277500f559f67206729eb025c5eb0057fc0ac97f4bda3"
Jul 29 19:56:50.621132 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:50.621111    2557 scope.go:117] "RemoveContainer" containerID="ff014c6e863d7d966dd277500f559f67206729eb025c5eb0057fc0ac97f4bda3"
Jul 29 19:56:50.621400 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:56:50.621379    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff014c6e863d7d966dd277500f559f67206729eb025c5eb0057fc0ac97f4bda3\": container with ID starting with ff014c6e863d7d966dd277500f559f67206729eb025c5eb0057fc0ac97f4bda3 not found: ID does not exist" containerID="ff014c6e863d7d966dd277500f559f67206729eb025c5eb0057fc0ac97f4bda3"
Jul 29 19:56:50.621437 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:50.621409    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff014c6e863d7d966dd277500f559f67206729eb025c5eb0057fc0ac97f4bda3"} err="failed to get container status \"ff014c6e863d7d966dd277500f559f67206729eb025c5eb0057fc0ac97f4bda3\": rpc error: code = NotFound desc = could not find container \"ff014c6e863d7d966dd277500f559f67206729eb025c5eb0057fc0ac97f4bda3\": container with ID starting with ff014c6e863d7d966dd277500f559f67206729eb025c5eb0057fc0ac97f4bda3 not found: ID does not exist"
Jul 29 19:56:50.634023 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:50.633995    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h"]
Jul 29 19:56:50.635854 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:50.635831    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-1d9a27/test-model-1d9a27-kserve-7476fccdf-ql94h"]
Jul 29 19:56:51.678149 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:51.678117    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fff1b56-4ab6-4e24-bb66-17449d173461" path="/var/lib/kubelet/pods/5fff1b56-4ab6-4e24-bb66-17449d173461/volumes"
Jul 29 19:56:52.775274 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:52.775239    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d9a27-xk2zf"]
Jul 29 19:56:52.775641 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:52.775607    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a" containerName="payload-pre-processing"
Jul 29 19:56:52.775641 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:52.775617    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a" containerName="payload-pre-processing"
Jul 29 19:56:52.775641 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:52.775634    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2e72342-4853-4a90-ae14-a57bae1441fb" containerName="main"
Jul 29 19:56:52.775641 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:52.775640    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2e72342-4853-4a90-ae14-a57bae1441fb" containerName="main"
Jul 29 19:56:52.775776 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:52.775650    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a87fc48e-7f52-48e6-9c3c-92216470881d" containerName="maas-api"
Jul 29 19:56:52.775776 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:52.775655    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a87fc48e-7f52-48e6-9c3c-92216470881d" containerName="maas-api"
Jul 29 19:56:52.775776 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:52.775664    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fff1b56-4ab6-4e24-bb66-17449d173461" containerName="main"
Jul 29 19:56:52.775776 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:52.775670    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fff1b56-4ab6-4e24-bb66-17449d173461" containerName="main"
Jul 29 19:56:52.775776 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:52.775680    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0965cf59-dee6-4391-a0f6-e84d807677b6" containerName="payload-processing"
Jul 29 19:56:52.775776 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:52.775686    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0965cf59-dee6-4391-a0f6-e84d807677b6" containerName="payload-processing"
Jul 29 19:56:52.775776 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:52.775743    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2e72342-4853-4a90-ae14-a57bae1441fb" containerName="main"
Jul 29 19:56:52.775776 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:52.775753    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0965cf59-dee6-4391-a0f6-e84d807677b6" containerName="payload-processing"
Jul 29 19:56:52.775776 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:52.775760    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a87fc48e-7f52-48e6-9c3c-92216470881d" containerName="maas-api"
Jul 29 19:56:52.775776 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:52.775770    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a244e8b8-0262-4e7e-b705-1bcfc8a7dd5a" containerName="payload-pre-processing"
Jul 29 19:56:52.775776 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:52.775778    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fff1b56-4ab6-4e24-bb66-17449d173461" containerName="main"
Jul 29 19:56:52.780244 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:52.780227    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d9a27-xk2zf"
Jul 29 19:56:52.783674 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:52.783655    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jsq2r\""
Jul 29 19:56:52.787924 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:52.787895    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d9a27-xk2zf"]
Jul 29 19:56:52.859607 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:52.859575    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70f19b6a-a3b4-4f5a-b766-7abcb39dd61f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1d9a27-xk2zf\" (UID: \"70f19b6a-a3b4-4f5a-b766-7abcb39dd61f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d9a27-xk2zf"
Jul 29 19:56:52.961033 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:52.960999    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70f19b6a-a3b4-4f5a-b766-7abcb39dd61f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1d9a27-xk2zf\" (UID: \"70f19b6a-a3b4-4f5a-b766-7abcb39dd61f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d9a27-xk2zf"
Jul 29 19:56:52.961679 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:52.961655    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70f19b6a-a3b4-4f5a-b766-7abcb39dd61f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1d9a27-xk2zf\" (UID: \"70f19b6a-a3b4-4f5a-b766-7abcb39dd61f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d9a27-xk2zf"
Jul 29 19:56:53.091366 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:53.091282    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d9a27-xk2zf"
Jul 29 19:56:53.460579 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:53.460552    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d9a27-xk2zf"]
Jul 29 19:56:53.462132 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:56:53.462104    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f19b6a_a3b4_4f5a_b766_7abcb39dd61f.slice/crio-67c763bc944bddff42287e5a00674efe12832cb066b5bed26af1c52c04971754 WatchSource:0}: Error finding container 67c763bc944bddff42287e5a00674efe12832cb066b5bed26af1c52c04971754: Status 404 returned error can't find the container with id 67c763bc944bddff42287e5a00674efe12832cb066b5bed26af1c52c04971754
Jul 29 19:56:53.635545 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:53.635510    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d9a27-xk2zf" event={"ID":"70f19b6a-a3b4-4f5a-b766-7abcb39dd61f","Type":"ContainerStarted","Data":"67c763bc944bddff42287e5a00674efe12832cb066b5bed26af1c52c04971754"}
Jul 29 19:56:54.642280 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:54.642248    2557 generic.go:356] "Generic (PLEG): container finished" podID="70f19b6a-a3b4-4f5a-b766-7abcb39dd61f" containerID="941b8f2dc6cebf27ed99e6d4e628520d7492f9bd23d639ae8509359f85013780" exitCode=0
Jul 29 19:56:54.642732 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:54.642340    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d9a27-xk2zf" event={"ID":"70f19b6a-a3b4-4f5a-b766-7abcb39dd61f","Type":"ContainerDied","Data":"941b8f2dc6cebf27ed99e6d4e628520d7492f9bd23d639ae8509359f85013780"}
Jul 29 19:56:55.780256 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:55.780228    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d9a27-xk2zf"
Jul 29 19:56:55.887827 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:55.887797    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70f19b6a-a3b4-4f5a-b766-7abcb39dd61f-maas-api-service-ca\") pod \"70f19b6a-a3b4-4f5a-b766-7abcb39dd61f\" (UID: \"70f19b6a-a3b4-4f5a-b766-7abcb39dd61f\") "
Jul 29 19:56:55.888157 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:55.888135    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70f19b6a-a3b4-4f5a-b766-7abcb39dd61f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "70f19b6a-a3b4-4f5a-b766-7abcb39dd61f" (UID: "70f19b6a-a3b4-4f5a-b766-7abcb39dd61f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:56:55.988539 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:55.988450    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70f19b6a-a3b4-4f5a-b766-7abcb39dd61f-maas-api-service-ca\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:56:56.652878 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:56.652844    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d9a27-xk2zf"
Jul 29 19:56:56.653051 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:56.652896    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d9a27-xk2zf" event={"ID":"70f19b6a-a3b4-4f5a-b766-7abcb39dd61f","Type":"ContainerDied","Data":"67c763bc944bddff42287e5a00674efe12832cb066b5bed26af1c52c04971754"}
Jul 29 19:56:56.653051 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:56:56.652945    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67c763bc944bddff42287e5a00674efe12832cb066b5bed26af1c52c04971754"
Jul 29 19:57:03.298221 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.298184    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9"]
Jul 29 19:57:03.298749 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.298398    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9" podUID="47626a9a-94cb-47bc-a34d-288f743d9b04" containerName="maas-api" containerID="cri-o://82197994cd77fc2bc8ae00e5e175eff7815a4d6bff6f97d6f101e0efb91f303b" gracePeriod=30
Jul 29 19:57:03.373286 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.373251    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1d9a27-f6579896d-mpknn"]
Jul 29 19:57:03.373567 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.373540    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-1d9a27-f6579896d-mpknn" podUID="29567083-0305-4296-8baa-30330c968658" containerName="payload-processing" containerID="cri-o://240fc6e911344750f4de1a36be0bbaa800494775a814177eb7b5e6e999dbeb54" gracePeriod=30
Jul 29 19:57:03.379797 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.379383    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1d9a27-7ff67d944d-wn4k4"]
Jul 29 19:57:03.379797 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.379627    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d9a27-7ff67d944d-wn4k4" podUID="a1d6f940-f312-422c-b951-7acb7df5b318" containerName="payload-pre-processing" containerID="cri-o://afac48069b8cecc488128f64a81d2f7b1452c99058ba65d09400024a4223ecd0" gracePeriod=30
Jul 29 19:57:03.541747 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.541703    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d9a27-xk2zf"]
Jul 29 19:57:03.553890 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.553812    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d9a27-xk2zf"]
Jul 29 19:57:03.664368 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.664341    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1d9a27-f6579896d-mpknn"
Jul 29 19:57:03.668380 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.668350    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d9a27-7ff67d944d-wn4k4"
Jul 29 19:57:03.681782 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.681755    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70f19b6a-a3b4-4f5a-b766-7abcb39dd61f" path="/var/lib/kubelet/pods/70f19b6a-a3b4-4f5a-b766-7abcb39dd61f/volumes"
Jul 29 19:57:03.695204 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.695073    2557 generic.go:356] "Generic (PLEG): container finished" podID="a1d6f940-f312-422c-b951-7acb7df5b318" containerID="afac48069b8cecc488128f64a81d2f7b1452c99058ba65d09400024a4223ecd0" exitCode=0
Jul 29 19:57:03.695204 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.695161    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d9a27-7ff67d944d-wn4k4" event={"ID":"a1d6f940-f312-422c-b951-7acb7df5b318","Type":"ContainerDied","Data":"afac48069b8cecc488128f64a81d2f7b1452c99058ba65d09400024a4223ecd0"}
Jul 29 19:57:03.695390 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.695229    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d9a27-7ff67d944d-wn4k4" event={"ID":"a1d6f940-f312-422c-b951-7acb7df5b318","Type":"ContainerDied","Data":"3ab41fc47be173bb0016f750d794c7b3d38d25b602f2817662c6c5ddd96d820e"}
Jul 29 19:57:03.695390 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.695259    2557 scope.go:117] "RemoveContainer" containerID="afac48069b8cecc488128f64a81d2f7b1452c99058ba65d09400024a4223ecd0"
Jul 29 19:57:03.695390 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.695293    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d9a27-7ff67d944d-wn4k4"
Jul 29 19:57:03.698449 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.698000    2557 generic.go:356] "Generic (PLEG): container finished" podID="29567083-0305-4296-8baa-30330c968658" containerID="240fc6e911344750f4de1a36be0bbaa800494775a814177eb7b5e6e999dbeb54" exitCode=0
Jul 29 19:57:03.698449 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.698147    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1d9a27-f6579896d-mpknn"
Jul 29 19:57:03.698449 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.698165    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1d9a27-f6579896d-mpknn" event={"ID":"29567083-0305-4296-8baa-30330c968658","Type":"ContainerDied","Data":"240fc6e911344750f4de1a36be0bbaa800494775a814177eb7b5e6e999dbeb54"}
Jul 29 19:57:03.698449 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.698235    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1d9a27-f6579896d-mpknn" event={"ID":"29567083-0305-4296-8baa-30330c968658","Type":"ContainerDied","Data":"6c03ea98944fb6db58d94a9434f7ddbafbd51fc9676491a0a928e23408c72ef8"}
Jul 29 19:57:03.700744 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.700723    2557 generic.go:356] "Generic (PLEG): container finished" podID="47626a9a-94cb-47bc-a34d-288f743d9b04" containerID="82197994cd77fc2bc8ae00e5e175eff7815a4d6bff6f97d6f101e0efb91f303b" exitCode=0
Jul 29 19:57:03.700869 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.700757    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9" event={"ID":"47626a9a-94cb-47bc-a34d-288f743d9b04","Type":"ContainerDied","Data":"82197994cd77fc2bc8ae00e5e175eff7815a4d6bff6f97d6f101e0efb91f303b"}
Jul 29 19:57:03.705108 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.705083    2557 scope.go:117] "RemoveContainer" containerID="afac48069b8cecc488128f64a81d2f7b1452c99058ba65d09400024a4223ecd0"
Jul 29 19:57:03.705420 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:57:03.705388    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afac48069b8cecc488128f64a81d2f7b1452c99058ba65d09400024a4223ecd0\": container with ID starting with afac48069b8cecc488128f64a81d2f7b1452c99058ba65d09400024a4223ecd0 not found: ID does not exist" containerID="afac48069b8cecc488128f64a81d2f7b1452c99058ba65d09400024a4223ecd0"
Jul 29 19:57:03.705494 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.705426    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afac48069b8cecc488128f64a81d2f7b1452c99058ba65d09400024a4223ecd0"} err="failed to get container status \"afac48069b8cecc488128f64a81d2f7b1452c99058ba65d09400024a4223ecd0\": rpc error: code = NotFound desc = could not find container \"afac48069b8cecc488128f64a81d2f7b1452c99058ba65d09400024a4223ecd0\": container with ID starting with afac48069b8cecc488128f64a81d2f7b1452c99058ba65d09400024a4223ecd0 not found: ID does not exist"
Jul 29 19:57:03.705494 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.705450    2557 scope.go:117] "RemoveContainer" containerID="240fc6e911344750f4de1a36be0bbaa800494775a814177eb7b5e6e999dbeb54"
Jul 29 19:57:03.713995 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.713975    2557 scope.go:117] "RemoveContainer" containerID="240fc6e911344750f4de1a36be0bbaa800494775a814177eb7b5e6e999dbeb54"
Jul 29 19:57:03.714308 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:57:03.714283    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"240fc6e911344750f4de1a36be0bbaa800494775a814177eb7b5e6e999dbeb54\": container with ID starting with 240fc6e911344750f4de1a36be0bbaa800494775a814177eb7b5e6e999dbeb54 not found: ID does not exist" containerID="240fc6e911344750f4de1a36be0bbaa800494775a814177eb7b5e6e999dbeb54"
Jul 29 19:57:03.714473 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.714313    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"240fc6e911344750f4de1a36be0bbaa800494775a814177eb7b5e6e999dbeb54"} err="failed to get container status \"240fc6e911344750f4de1a36be0bbaa800494775a814177eb7b5e6e999dbeb54\": rpc error: code = NotFound desc = could not find container \"240fc6e911344750f4de1a36be0bbaa800494775a814177eb7b5e6e999dbeb54\": container with ID starting with 240fc6e911344750f4de1a36be0bbaa800494775a814177eb7b5e6e999dbeb54 not found: ID does not exist"
Jul 29 19:57:03.743982 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.743951    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9"
Jul 29 19:57:03.761032 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.761000    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pcwb\" (UniqueName: \"kubernetes.io/projected/a1d6f940-f312-422c-b951-7acb7df5b318-kube-api-access-2pcwb\") pod \"a1d6f940-f312-422c-b951-7acb7df5b318\" (UID: \"a1d6f940-f312-422c-b951-7acb7df5b318\") "
Jul 29 19:57:03.761158 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.761044    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29567083-0305-4296-8baa-30330c968658-plugins-config-volume\") pod \"29567083-0305-4296-8baa-30330c968658\" (UID: \"29567083-0305-4296-8baa-30330c968658\") "
Jul 29 19:57:03.761158 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.761068    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfc7j\" (UniqueName: \"kubernetes.io/projected/29567083-0305-4296-8baa-30330c968658-kube-api-access-qfc7j\") pod \"29567083-0305-4296-8baa-30330c968658\" (UID: \"29567083-0305-4296-8baa-30330c968658\") "
Jul 29 19:57:03.761158 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.761118    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1d6f940-f312-422c-b951-7acb7df5b318-plugins-config-volume\") pod \"a1d6f940-f312-422c-b951-7acb7df5b318\" (UID: \"a1d6f940-f312-422c-b951-7acb7df5b318\") "
Jul 29 19:57:03.761427 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.761401    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29567083-0305-4296-8baa-30330c968658-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "29567083-0305-4296-8baa-30330c968658" (UID: "29567083-0305-4296-8baa-30330c968658"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:57:03.761526 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.761508    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1d6f940-f312-422c-b951-7acb7df5b318-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a1d6f940-f312-422c-b951-7acb7df5b318" (UID: "a1d6f940-f312-422c-b951-7acb7df5b318"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:57:03.763148 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.763126    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1d6f940-f312-422c-b951-7acb7df5b318-kube-api-access-2pcwb" (OuterVolumeSpecName: "kube-api-access-2pcwb") pod "a1d6f940-f312-422c-b951-7acb7df5b318" (UID: "a1d6f940-f312-422c-b951-7acb7df5b318"). InnerVolumeSpecName "kube-api-access-2pcwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:57:03.763255 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.763173    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29567083-0305-4296-8baa-30330c968658-kube-api-access-qfc7j" (OuterVolumeSpecName: "kube-api-access-qfc7j") pod "29567083-0305-4296-8baa-30330c968658" (UID: "29567083-0305-4296-8baa-30330c968658"). InnerVolumeSpecName "kube-api-access-qfc7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:57:03.862058 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.861981    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kt89r\" (UniqueName: \"kubernetes.io/projected/47626a9a-94cb-47bc-a34d-288f743d9b04-kube-api-access-kt89r\") pod \"47626a9a-94cb-47bc-a34d-288f743d9b04\" (UID: \"47626a9a-94cb-47bc-a34d-288f743d9b04\") "
Jul 29 19:57:03.862058 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.862047    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47626a9a-94cb-47bc-a34d-288f743d9b04-openshift-service-ca\") pod \"47626a9a-94cb-47bc-a34d-288f743d9b04\" (UID: \"47626a9a-94cb-47bc-a34d-288f743d9b04\") "
Jul 29 19:57:03.862225 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.862093    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47626a9a-94cb-47bc-a34d-288f743d9b04-maas-api-tls\") pod \"47626a9a-94cb-47bc-a34d-288f743d9b04\" (UID: \"47626a9a-94cb-47bc-a34d-288f743d9b04\") "
Jul 29 19:57:03.862362 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.862337    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29567083-0305-4296-8baa-30330c968658-plugins-config-volume\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:57:03.862362 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.862364    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfc7j\" (UniqueName: \"kubernetes.io/projected/29567083-0305-4296-8baa-30330c968658-kube-api-access-qfc7j\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:57:03.862513 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.862379    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1d6f940-f312-422c-b951-7acb7df5b318-plugins-config-volume\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:57:03.862513 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.862393    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pcwb\" (UniqueName: \"kubernetes.io/projected/a1d6f940-f312-422c-b951-7acb7df5b318-kube-api-access-2pcwb\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:57:03.862513 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.862417    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47626a9a-94cb-47bc-a34d-288f743d9b04-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "47626a9a-94cb-47bc-a34d-288f743d9b04" (UID: "47626a9a-94cb-47bc-a34d-288f743d9b04"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:57:03.864014 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.863995    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47626a9a-94cb-47bc-a34d-288f743d9b04-kube-api-access-kt89r" (OuterVolumeSpecName: "kube-api-access-kt89r") pod "47626a9a-94cb-47bc-a34d-288f743d9b04" (UID: "47626a9a-94cb-47bc-a34d-288f743d9b04"). InnerVolumeSpecName "kube-api-access-kt89r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:57:03.864082 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.864053    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47626a9a-94cb-47bc-a34d-288f743d9b04-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "47626a9a-94cb-47bc-a34d-288f743d9b04" (UID: "47626a9a-94cb-47bc-a34d-288f743d9b04"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:57:03.962871 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.962834    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt89r\" (UniqueName: \"kubernetes.io/projected/47626a9a-94cb-47bc-a34d-288f743d9b04-kube-api-access-kt89r\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:57:03.962871 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.962865    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47626a9a-94cb-47bc-a34d-288f743d9b04-openshift-service-ca\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:57:03.962871 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:03.962875    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47626a9a-94cb-47bc-a34d-288f743d9b04-maas-api-tls\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:57:04.022889 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:04.022862    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1d9a27-7ff67d944d-wn4k4"]
Jul 29 19:57:04.027977 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:04.027934    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1d9a27-7ff67d944d-wn4k4"]
Jul 29 19:57:04.041532 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:04.041507    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1d9a27-f6579896d-mpknn"]
Jul 29 19:57:04.044703 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:04.044682    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1d9a27-f6579896d-mpknn"]
Jul 29 19:57:04.706190 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:04.706095    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9"
Jul 29 19:57:04.706190 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:04.706134    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9" event={"ID":"47626a9a-94cb-47bc-a34d-288f743d9b04","Type":"ContainerDied","Data":"6afbdbacc570462e791d6bf6f0004c5551e0a083b6a824b148b265844a9ecb61"}
Jul 29 19:57:04.706190 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:04.706189    2557 scope.go:117] "RemoveContainer" containerID="82197994cd77fc2bc8ae00e5e175eff7815a4d6bff6f97d6f101e0efb91f303b"
Jul 29 19:57:04.733407 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:04.733379    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9"]
Jul 29 19:57:04.741376 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:04.741352    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1d9a27-696b8dc5bd-xlpg9"]
Jul 29 19:57:05.679093 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:05.679062    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29567083-0305-4296-8baa-30330c968658" path="/var/lib/kubelet/pods/29567083-0305-4296-8baa-30330c968658/volumes"
Jul 29 19:57:05.679463 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:05.679449    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47626a9a-94cb-47bc-a34d-288f743d9b04" path="/var/lib/kubelet/pods/47626a9a-94cb-47bc-a34d-288f743d9b04/volumes"
Jul 29 19:57:05.679765 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:05.679753    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1d6f940-f312-422c-b951-7acb7df5b318" path="/var/lib/kubelet/pods/a1d6f940-f312-422c-b951-7acb7df5b318/volumes"
Jul 29 19:57:14.889955 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:14.889913    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-466674-7f5f7bff55-c854p"]
Jul 29 19:57:14.890426 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:14.890272    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1d6f940-f312-422c-b951-7acb7df5b318" containerName="payload-pre-processing"
Jul 29 19:57:14.890426 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:14.890284    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1d6f940-f312-422c-b951-7acb7df5b318" containerName="payload-pre-processing"
Jul 29 19:57:14.890426 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:14.890298    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47626a9a-94cb-47bc-a34d-288f743d9b04" containerName="maas-api"
Jul 29 19:57:14.890426 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:14.890303    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="47626a9a-94cb-47bc-a34d-288f743d9b04" containerName="maas-api"
Jul 29 19:57:14.890426 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:14.890318    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29567083-0305-4296-8baa-30330c968658" containerName="payload-processing"
Jul 29 19:57:14.890426 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:14.890324    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="29567083-0305-4296-8baa-30330c968658" containerName="payload-processing"
Jul 29 19:57:14.890426 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:14.890329    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70f19b6a-a3b4-4f5a-b766-7abcb39dd61f" containerName="revoke-keys"
Jul 29 19:57:14.890426 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:14.890334    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="70f19b6a-a3b4-4f5a-b766-7abcb39dd61f" containerName="revoke-keys"
Jul 29 19:57:14.890426 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:14.890385    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="29567083-0305-4296-8baa-30330c968658" containerName="payload-processing"
Jul 29 19:57:14.890426 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:14.890395    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="70f19b6a-a3b4-4f5a-b766-7abcb39dd61f" containerName="revoke-keys"
Jul 29 19:57:14.890426 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:14.890401    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="47626a9a-94cb-47bc-a34d-288f743d9b04" containerName="maas-api"
Jul 29 19:57:14.890426 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:14.890408    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1d6f940-f312-422c-b951-7acb7df5b318" containerName="payload-pre-processing"
Jul 29 19:57:14.895190 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:14.895163    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466674-7f5f7bff55-c854p"
Jul 29 19:57:14.898107 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:14.898070    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-466674-serving-cert\""
Jul 29 19:57:14.898107 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:14.898083    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4dw6s\""
Jul 29 19:57:14.902769 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:14.902748    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-466674-7f5f7bff55-c854p"]
Jul 29 19:57:15.060198 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:15.060158    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd845\" (UniqueName: \"kubernetes.io/projected/2da6ccc0-0ba1-435f-964e-7e67ba0a7958-kube-api-access-rd845\") pod \"maas-api-e2e-api-a-466674-7f5f7bff55-c854p\" (UID: \"2da6ccc0-0ba1-435f-964e-7e67ba0a7958\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466674-7f5f7bff55-c854p"
Jul 29 19:57:15.060353 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:15.060211    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2da6ccc0-0ba1-435f-964e-7e67ba0a7958-maas-api-tls\") pod \"maas-api-e2e-api-a-466674-7f5f7bff55-c854p\" (UID: \"2da6ccc0-0ba1-435f-964e-7e67ba0a7958\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466674-7f5f7bff55-c854p"
Jul 29 19:57:15.060401 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:15.060361    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da6ccc0-0ba1-435f-964e-7e67ba0a7958-openshift-service-ca\") pod \"maas-api-e2e-api-a-466674-7f5f7bff55-c854p\" (UID: \"2da6ccc0-0ba1-435f-964e-7e67ba0a7958\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466674-7f5f7bff55-c854p"
Jul 29 19:57:15.161888 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:15.161791    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da6ccc0-0ba1-435f-964e-7e67ba0a7958-openshift-service-ca\") pod \"maas-api-e2e-api-a-466674-7f5f7bff55-c854p\" (UID: \"2da6ccc0-0ba1-435f-964e-7e67ba0a7958\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466674-7f5f7bff55-c854p"
Jul 29 19:57:15.161888 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:15.161834    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd845\" (UniqueName: \"kubernetes.io/projected/2da6ccc0-0ba1-435f-964e-7e67ba0a7958-kube-api-access-rd845\") pod \"maas-api-e2e-api-a-466674-7f5f7bff55-c854p\" (UID: \"2da6ccc0-0ba1-435f-964e-7e67ba0a7958\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466674-7f5f7bff55-c854p"
Jul 29 19:57:15.162123 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:15.161987    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2da6ccc0-0ba1-435f-964e-7e67ba0a7958-maas-api-tls\") pod \"maas-api-e2e-api-a-466674-7f5f7bff55-c854p\" (UID: \"2da6ccc0-0ba1-435f-964e-7e67ba0a7958\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466674-7f5f7bff55-c854p"
Jul 29 19:57:15.162600 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:15.162563    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da6ccc0-0ba1-435f-964e-7e67ba0a7958-openshift-service-ca\") pod \"maas-api-e2e-api-a-466674-7f5f7bff55-c854p\" (UID: \"2da6ccc0-0ba1-435f-964e-7e67ba0a7958\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466674-7f5f7bff55-c854p"
Jul 29 19:57:15.164446 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:15.164425    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2da6ccc0-0ba1-435f-964e-7e67ba0a7958-maas-api-tls\") pod \"maas-api-e2e-api-a-466674-7f5f7bff55-c854p\" (UID: \"2da6ccc0-0ba1-435f-964e-7e67ba0a7958\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466674-7f5f7bff55-c854p"
Jul 29 19:57:15.180179 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:15.180151    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd845\" (UniqueName: \"kubernetes.io/projected/2da6ccc0-0ba1-435f-964e-7e67ba0a7958-kube-api-access-rd845\") pod \"maas-api-e2e-api-a-466674-7f5f7bff55-c854p\" (UID: \"2da6ccc0-0ba1-435f-964e-7e67ba0a7958\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466674-7f5f7bff55-c854p"
Jul 29 19:57:15.207085 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:15.207052    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466674-7f5f7bff55-c854p"
Jul 29 19:57:15.341770 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:15.341742    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-466674-7f5f7bff55-c854p"]
Jul 29 19:57:15.343936 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:57:15.343906    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da6ccc0_0ba1_435f_964e_7e67ba0a7958.slice/crio-8794746ac59816060dafec3a2c7fda5129876387241d50820febe4a261f75f3b WatchSource:0}: Error finding container 8794746ac59816060dafec3a2c7fda5129876387241d50820febe4a261f75f3b: Status 404 returned error can't find the container with id 8794746ac59816060dafec3a2c7fda5129876387241d50820febe4a261f75f3b
Jul 29 19:57:15.345267 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:15.345247    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 19:57:15.681687 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:15.681654    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-466674-7d76c6c5f4-vnz65"]
Jul 29 19:57:15.684876 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:15.684859    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-466674-7d76c6c5f4-vnz65"
Jul 29 19:57:15.687919 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:15.687899    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-466674\""
Jul 29 19:57:15.687993 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:15.687911    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-466674-dockercfg-zhkc2\""
Jul 29 19:57:15.695177 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:15.695156    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-466674-7d76c6c5f4-vnz65"]
Jul 29 19:57:15.709815 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:15.709786    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-466674-6d875577b8-gjjk2"]
Jul 29 19:57:15.713487 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:15.713469    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466674-6d875577b8-gjjk2"
Jul 29 19:57:15.722170 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:15.722146    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-466674-6d875577b8-gjjk2"]
Jul 29 19:57:15.760605 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:15.760573    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466674-7f5f7bff55-c854p" event={"ID":"2da6ccc0-0ba1-435f-964e-7e67ba0a7958","Type":"ContainerStarted","Data":"8794746ac59816060dafec3a2c7fda5129876387241d50820febe4a261f75f3b"}
Jul 29 19:57:15.869545 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:15.869519    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7ffaaa-ff47-4c2b-871a-2c75da49dab3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-466674-7d76c6c5f4-vnz65\" (UID: \"6e7ffaaa-ff47-4c2b-871a-2c75da49dab3\") " pod="openshift-ingress/payload-processing-e2e-api-a-466674-7d76c6c5f4-vnz65"
Jul 29 19:57:15.869651 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:15.869633    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62476440-b3d0-4e31-97e9-a777dbca6972-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-466674-6d875577b8-gjjk2\" (UID: \"62476440-b3d0-4e31-97e9-a777dbca6972\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-466674-6d875577b8-gjjk2"
Jul 29 19:57:15.869715 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:15.869702    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2jzj\" (UniqueName: \"kubernetes.io/projected/6e7ffaaa-ff47-4c2b-871a-2c75da49dab3-kube-api-access-h2jzj\") pod \"payload-processing-e2e-api-a-466674-7d76c6c5f4-vnz65\" (UID: \"6e7ffaaa-ff47-4c2b-871a-2c75da49dab3\") " pod="openshift-ingress/payload-processing-e2e-api-a-466674-7d76c6c5f4-vnz65"
Jul 29 19:57:15.869778 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:15.869746    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5jhf\" (UniqueName: \"kubernetes.io/projected/62476440-b3d0-4e31-97e9-a777dbca6972-kube-api-access-x5jhf\") pod \"payload-pre-processing-e2e-api-a-466674-6d875577b8-gjjk2\" (UID: \"62476440-b3d0-4e31-97e9-a777dbca6972\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-466674-6d875577b8-gjjk2"
Jul 29 19:57:15.970799 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:15.970719    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7ffaaa-ff47-4c2b-871a-2c75da49dab3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-466674-7d76c6c5f4-vnz65\" (UID: \"6e7ffaaa-ff47-4c2b-871a-2c75da49dab3\") " pod="openshift-ingress/payload-processing-e2e-api-a-466674-7d76c6c5f4-vnz65"
Jul 29 19:57:15.970799 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:15.970770    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62476440-b3d0-4e31-97e9-a777dbca6972-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-466674-6d875577b8-gjjk2\" (UID: \"62476440-b3d0-4e31-97e9-a777dbca6972\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-466674-6d875577b8-gjjk2"
Jul 29 19:57:15.971335 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:15.970817    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2jzj\" (UniqueName: \"kubernetes.io/projected/6e7ffaaa-ff47-4c2b-871a-2c75da49dab3-kube-api-access-h2jzj\") pod \"payload-processing-e2e-api-a-466674-7d76c6c5f4-vnz65\" (UID: \"6e7ffaaa-ff47-4c2b-871a-2c75da49dab3\") " pod="openshift-ingress/payload-processing-e2e-api-a-466674-7d76c6c5f4-vnz65"
Jul 29 19:57:15.971335 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:15.970837    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5jhf\" (UniqueName: \"kubernetes.io/projected/62476440-b3d0-4e31-97e9-a777dbca6972-kube-api-access-x5jhf\") pod \"payload-pre-processing-e2e-api-a-466674-6d875577b8-gjjk2\" (UID: \"62476440-b3d0-4e31-97e9-a777dbca6972\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-466674-6d875577b8-gjjk2"
Jul 29 19:57:15.971493 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:15.971469    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7ffaaa-ff47-4c2b-871a-2c75da49dab3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-466674-7d76c6c5f4-vnz65\" (UID: \"6e7ffaaa-ff47-4c2b-871a-2c75da49dab3\") " pod="openshift-ingress/payload-processing-e2e-api-a-466674-7d76c6c5f4-vnz65"
Jul 29 19:57:15.971674 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:15.971653    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62476440-b3d0-4e31-97e9-a777dbca6972-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-466674-6d875577b8-gjjk2\" (UID: \"62476440-b3d0-4e31-97e9-a777dbca6972\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-466674-6d875577b8-gjjk2"
Jul 29 19:57:15.979097 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:15.979070    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2jzj\" (UniqueName: \"kubernetes.io/projected/6e7ffaaa-ff47-4c2b-871a-2c75da49dab3-kube-api-access-h2jzj\") pod \"payload-processing-e2e-api-a-466674-7d76c6c5f4-vnz65\" (UID: \"6e7ffaaa-ff47-4c2b-871a-2c75da49dab3\") " pod="openshift-ingress/payload-processing-e2e-api-a-466674-7d76c6c5f4-vnz65"
Jul 29 19:57:15.979206 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:15.979123    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5jhf\" (UniqueName: \"kubernetes.io/projected/62476440-b3d0-4e31-97e9-a777dbca6972-kube-api-access-x5jhf\") pod \"payload-pre-processing-e2e-api-a-466674-6d875577b8-gjjk2\" (UID: \"62476440-b3d0-4e31-97e9-a777dbca6972\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-466674-6d875577b8-gjjk2"
Jul 29 19:57:15.995824 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:15.995803    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-466674-7d76c6c5f4-vnz65"
Jul 29 19:57:16.023884 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:16.023857    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466674-6d875577b8-gjjk2"
Jul 29 19:57:16.141056 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:16.141025    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-466674-7d76c6c5f4-vnz65"]
Jul 29 19:57:16.142883 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:57:16.142851    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e7ffaaa_ff47_4c2b_871a_2c75da49dab3.slice/crio-487c9db95fa532be6cafd38c15bcd172bf8b5d480b6067d0ea9139a999c999fd WatchSource:0}: Error finding container 487c9db95fa532be6cafd38c15bcd172bf8b5d480b6067d0ea9139a999c999fd: Status 404 returned error can't find the container with id 487c9db95fa532be6cafd38c15bcd172bf8b5d480b6067d0ea9139a999c999fd
Jul 29 19:57:16.175805 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:16.175784    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-466674-6d875577b8-gjjk2"]
Jul 29 19:57:16.177833 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:57:16.177803    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62476440_b3d0_4e31_97e9_a777dbca6972.slice/crio-a978da6dd7bd17a4f8ce3a78f760fc5b06b5adc18cd8b13b83f8133bdf091e80 WatchSource:0}: Error finding container a978da6dd7bd17a4f8ce3a78f760fc5b06b5adc18cd8b13b83f8133bdf091e80: Status 404 returned error can't find the container with id a978da6dd7bd17a4f8ce3a78f760fc5b06b5adc18cd8b13b83f8133bdf091e80
Jul 29 19:57:16.767821 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:16.767780    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-466674-7d76c6c5f4-vnz65" event={"ID":"6e7ffaaa-ff47-4c2b-871a-2c75da49dab3","Type":"ContainerStarted","Data":"2066425feecb2a1dc3ddc27bb325a659f27736c0f85d24e5a445c736a207c815"}
Jul 29 19:57:16.768056 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:16.767851    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-466674-7d76c6c5f4-vnz65"
Jul 29 19:57:16.768056 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:16.767877    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-466674-7d76c6c5f4-vnz65" event={"ID":"6e7ffaaa-ff47-4c2b-871a-2c75da49dab3","Type":"ContainerStarted","Data":"487c9db95fa532be6cafd38c15bcd172bf8b5d480b6067d0ea9139a999c999fd"}
Jul 29 19:57:16.769410 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:16.769375    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466674-6d875577b8-gjjk2" event={"ID":"62476440-b3d0-4e31-97e9-a777dbca6972","Type":"ContainerStarted","Data":"15df6d91b1988100e0de7b7b5a2a97d778548fbcf3203b9c630fbe096cb2e050"}
Jul 29 19:57:16.769556 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:16.769438    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466674-6d875577b8-gjjk2"
Jul 29 19:57:16.769556 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:16.769463    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466674-6d875577b8-gjjk2" event={"ID":"62476440-b3d0-4e31-97e9-a777dbca6972","Type":"ContainerStarted","Data":"a978da6dd7bd17a4f8ce3a78f760fc5b06b5adc18cd8b13b83f8133bdf091e80"}
Jul 29 19:57:16.770866 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:16.770840    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466674-7f5f7bff55-c854p" event={"ID":"2da6ccc0-0ba1-435f-964e-7e67ba0a7958","Type":"ContainerStarted","Data":"27c496b0d9ee24e2fbfbf44df49699c8f85b2a5cde85ba9cb39be9c1c31bdea6"}
Jul 29 19:57:16.770998 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:16.770976    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466674-7f5f7bff55-c854p"
Jul 29 19:57:16.788994 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:16.788933    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-466674-7d76c6c5f4-vnz65" podStartSLOduration=1.78892005 podStartE2EDuration="1.78892005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:57:16.788466142 +0000 UTC m=+2407.615965217" watchObservedRunningTime="2026-07-29 19:57:16.78892005 +0000 UTC m=+2407.616419108"
Jul 29 19:57:16.807486 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:16.807440    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466674-6d875577b8-gjjk2" podStartSLOduration=1.807427583 podStartE2EDuration="1.807427583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:57:16.806419795 +0000 UTC m=+2407.633918852" watchObservedRunningTime="2026-07-29 19:57:16.807427583 +0000 UTC m=+2407.634926641"
Jul 29 19:57:16.824547 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:16.824497    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466674-7f5f7bff55-c854p" podStartSLOduration=2.306648049 podStartE2EDuration="2.824484936s" totalImagesPullingTime="517.836887ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:57:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:57:16.823195797 +0000 UTC m=+2407.650694837" watchObservedRunningTime="2026-07-29 19:57:16.824484936 +0000 UTC m=+2407.651984014"
Jul 29 19:57:22.780363 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:22.780333    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466674-7f5f7bff55-c854p"
Jul 29 19:57:27.776924 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:27.776891    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-466674-7d76c6c5f4-vnz65"
Jul 29 19:57:27.777332 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:27.777069    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466674-6d875577b8-gjjk2"
Jul 29 19:57:43.160875 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:43.160774    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb"]
Jul 29 19:57:43.166084 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:43.166052    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb"
Jul 29 19:57:43.169172 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:43.169146    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-ecc3b0-serving-cert\""
Jul 29 19:57:43.177919 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:43.177888    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb"]
Jul 29 19:57:43.318313 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:43.318271    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b86ead8-b3b9-4d78-9225-f30fc36b8ec6-maas-api-tls\") pod \"maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb\" (UID: \"7b86ead8-b3b9-4d78-9225-f30fc36b8ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb"
Jul 29 19:57:43.318517 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:43.318330    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b86ead8-b3b9-4d78-9225-f30fc36b8ec6-openshift-service-ca\") pod \"maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb\" (UID: \"7b86ead8-b3b9-4d78-9225-f30fc36b8ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb"
Jul 29 19:57:43.318517 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:43.318416    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz77j\" (UniqueName: \"kubernetes.io/projected/7b86ead8-b3b9-4d78-9225-f30fc36b8ec6-kube-api-access-kz77j\") pod \"maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb\" (UID: \"7b86ead8-b3b9-4d78-9225-f30fc36b8ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb"
Jul 29 19:57:43.419306 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:43.419227    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b86ead8-b3b9-4d78-9225-f30fc36b8ec6-openshift-service-ca\") pod \"maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb\" (UID: \"7b86ead8-b3b9-4d78-9225-f30fc36b8ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb"
Jul 29 19:57:43.419306 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:43.419285    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz77j\" (UniqueName: \"kubernetes.io/projected/7b86ead8-b3b9-4d78-9225-f30fc36b8ec6-kube-api-access-kz77j\") pod \"maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb\" (UID: \"7b86ead8-b3b9-4d78-9225-f30fc36b8ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb"
Jul 29 19:57:43.419487 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:43.419332    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b86ead8-b3b9-4d78-9225-f30fc36b8ec6-maas-api-tls\") pod \"maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb\" (UID: \"7b86ead8-b3b9-4d78-9225-f30fc36b8ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb"
Jul 29 19:57:43.419948 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:43.419918    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b86ead8-b3b9-4d78-9225-f30fc36b8ec6-openshift-service-ca\") pod \"maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb\" (UID: \"7b86ead8-b3b9-4d78-9225-f30fc36b8ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb"
Jul 29 19:57:43.421668 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:43.421647    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b86ead8-b3b9-4d78-9225-f30fc36b8ec6-maas-api-tls\") pod \"maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb\" (UID: \"7b86ead8-b3b9-4d78-9225-f30fc36b8ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb"
Jul 29 19:57:43.429250 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:43.429224    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz77j\" (UniqueName: \"kubernetes.io/projected/7b86ead8-b3b9-4d78-9225-f30fc36b8ec6-kube-api-access-kz77j\") pod \"maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb\" (UID: \"7b86ead8-b3b9-4d78-9225-f30fc36b8ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb"
Jul 29 19:57:43.479362 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:43.479330    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb"
Jul 29 19:57:43.612880 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:43.612856    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb"]
Jul 29 19:57:43.614863 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:57:43.614832    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b86ead8_b3b9_4d78_9225_f30fc36b8ec6.slice/crio-783e402c171e482840144a6ed2b788bdae605a1669406061c15e771473897ada WatchSource:0}: Error finding container 783e402c171e482840144a6ed2b788bdae605a1669406061c15e771473897ada: Status 404 returned error can't find the container with id 783e402c171e482840144a6ed2b788bdae605a1669406061c15e771473897ada
Jul 29 19:57:43.807627 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:43.807547    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ecc3b0-76bbcb8684-msbwt"]
Jul 29 19:57:43.812373 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:43.812346    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecc3b0-76bbcb8684-msbwt"
Jul 29 19:57:43.815131 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:43.815107    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-ecc3b0\""
Jul 29 19:57:43.821207 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:43.821183    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ecc3b0-76bbcb8684-msbwt"]
Jul 29 19:57:43.830068 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:43.830043    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ecc3b0-74486fb68b-ts2zv"]
Jul 29 19:57:43.833261 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:43.833242    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ecc3b0-74486fb68b-ts2zv"
Jul 29 19:57:43.835998 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:43.835980    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-ecc3b0-dockercfg-gzz2n\""
Jul 29 19:57:43.844807 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:43.844784    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ecc3b0-74486fb68b-ts2zv"]
Jul 29 19:57:43.901947 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:43.901915    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb" event={"ID":"7b86ead8-b3b9-4d78-9225-f30fc36b8ec6","Type":"ContainerStarted","Data":"783e402c171e482840144a6ed2b788bdae605a1669406061c15e771473897ada"}
Jul 29 19:57:43.924179 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:43.924152    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wr7z\" (UniqueName: \"kubernetes.io/projected/e74ef00d-1d0c-4f45-99a5-4a593bb566ce-kube-api-access-5wr7z\") pod \"payload-pre-processing-e2e-api-b-ecc3b0-76bbcb8684-msbwt\" (UID: \"e74ef00d-1d0c-4f45-99a5-4a593bb566ce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecc3b0-76bbcb8684-msbwt"
Jul 29 19:57:43.924338 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:43.924249    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dsdw\" (UniqueName: \"kubernetes.io/projected/db4c2b46-a925-4abf-834f-73fa213e935f-kube-api-access-8dsdw\") pod \"payload-processing-e2e-api-b-ecc3b0-74486fb68b-ts2zv\" (UID: \"db4c2b46-a925-4abf-834f-73fa213e935f\") " pod="openshift-ingress/payload-processing-e2e-api-b-ecc3b0-74486fb68b-ts2zv"
Jul 29 19:57:43.924338 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:43.924300    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e74ef00d-1d0c-4f45-99a5-4a593bb566ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ecc3b0-76bbcb8684-msbwt\" (UID: \"e74ef00d-1d0c-4f45-99a5-4a593bb566ce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecc3b0-76bbcb8684-msbwt"
Jul 29 19:57:43.924338 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:43.924333    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db4c2b46-a925-4abf-834f-73fa213e935f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ecc3b0-74486fb68b-ts2zv\" (UID: \"db4c2b46-a925-4abf-834f-73fa213e935f\") " pod="openshift-ingress/payload-processing-e2e-api-b-ecc3b0-74486fb68b-ts2zv"
Jul 29 19:57:44.025415 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:44.025361    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wr7z\" (UniqueName: \"kubernetes.io/projected/e74ef00d-1d0c-4f45-99a5-4a593bb566ce-kube-api-access-5wr7z\") pod \"payload-pre-processing-e2e-api-b-ecc3b0-76bbcb8684-msbwt\" (UID: \"e74ef00d-1d0c-4f45-99a5-4a593bb566ce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecc3b0-76bbcb8684-msbwt"
Jul 29 19:57:44.025415 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:44.025418    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dsdw\" (UniqueName: \"kubernetes.io/projected/db4c2b46-a925-4abf-834f-73fa213e935f-kube-api-access-8dsdw\") pod \"payload-processing-e2e-api-b-ecc3b0-74486fb68b-ts2zv\" (UID: \"db4c2b46-a925-4abf-834f-73fa213e935f\") " pod="openshift-ingress/payload-processing-e2e-api-b-ecc3b0-74486fb68b-ts2zv"
Jul 29 19:57:44.025665 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:44.025447    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e74ef00d-1d0c-4f45-99a5-4a593bb566ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ecc3b0-76bbcb8684-msbwt\" (UID: \"e74ef00d-1d0c-4f45-99a5-4a593bb566ce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecc3b0-76bbcb8684-msbwt"
Jul 29 19:57:44.025665 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:44.025468    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db4c2b46-a925-4abf-834f-73fa213e935f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ecc3b0-74486fb68b-ts2zv\" (UID: \"db4c2b46-a925-4abf-834f-73fa213e935f\") " pod="openshift-ingress/payload-processing-e2e-api-b-ecc3b0-74486fb68b-ts2zv"
Jul 29 19:57:44.026209 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:44.026187    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e74ef00d-1d0c-4f45-99a5-4a593bb566ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ecc3b0-76bbcb8684-msbwt\" (UID: \"e74ef00d-1d0c-4f45-99a5-4a593bb566ce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecc3b0-76bbcb8684-msbwt"
Jul 29 19:57:44.026209 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:44.026199    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db4c2b46-a925-4abf-834f-73fa213e935f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ecc3b0-74486fb68b-ts2zv\" (UID: \"db4c2b46-a925-4abf-834f-73fa213e935f\") " pod="openshift-ingress/payload-processing-e2e-api-b-ecc3b0-74486fb68b-ts2zv"
Jul 29 19:57:44.034441 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:44.034422    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wr7z\" (UniqueName: \"kubernetes.io/projected/e74ef00d-1d0c-4f45-99a5-4a593bb566ce-kube-api-access-5wr7z\") pod \"payload-pre-processing-e2e-api-b-ecc3b0-76bbcb8684-msbwt\" (UID: \"e74ef00d-1d0c-4f45-99a5-4a593bb566ce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecc3b0-76bbcb8684-msbwt"
Jul 29 19:57:44.034522 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:44.034458    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dsdw\" (UniqueName: \"kubernetes.io/projected/db4c2b46-a925-4abf-834f-73fa213e935f-kube-api-access-8dsdw\") pod \"payload-processing-e2e-api-b-ecc3b0-74486fb68b-ts2zv\" (UID: \"db4c2b46-a925-4abf-834f-73fa213e935f\") " pod="openshift-ingress/payload-processing-e2e-api-b-ecc3b0-74486fb68b-ts2zv"
Jul 29 19:57:44.124378 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:44.124301    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecc3b0-76bbcb8684-msbwt"
Jul 29 19:57:44.143424 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:44.143398    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ecc3b0-74486fb68b-ts2zv"
Jul 29 19:57:44.280496 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:44.280470    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ecc3b0-76bbcb8684-msbwt"]
Jul 29 19:57:44.282541 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:57:44.282508    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode74ef00d_1d0c_4f45_99a5_4a593bb566ce.slice/crio-2bca863cef48dcba0d0d447892801ceeb2f3ee6efa0bcb0394e2aa485d94dd5c WatchSource:0}: Error finding container 2bca863cef48dcba0d0d447892801ceeb2f3ee6efa0bcb0394e2aa485d94dd5c: Status 404 returned error can't find the container with id 2bca863cef48dcba0d0d447892801ceeb2f3ee6efa0bcb0394e2aa485d94dd5c
Jul 29 19:57:44.322069 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:44.322024    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ecc3b0-74486fb68b-ts2zv"]
Jul 29 19:57:44.324533 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:57:44.324505    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4c2b46_a925_4abf_834f_73fa213e935f.slice/crio-9929f2846fdf8787eed572ba77242b1c3058f92f46017569feb0b3c7b27574ec WatchSource:0}: Error finding container 9929f2846fdf8787eed572ba77242b1c3058f92f46017569feb0b3c7b27574ec: Status 404 returned error can't find the container with id 9929f2846fdf8787eed572ba77242b1c3058f92f46017569feb0b3c7b27574ec
Jul 29 19:57:44.908287 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:44.908253    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ecc3b0-74486fb68b-ts2zv" event={"ID":"db4c2b46-a925-4abf-834f-73fa213e935f","Type":"ContainerStarted","Data":"0cace4d1c5a5deccfb3882c5ea074bb6a418dee390ac957175c01e829ff998d7"}
Jul 29 19:57:44.908473 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:44.908305    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ecc3b0-74486fb68b-ts2zv" event={"ID":"db4c2b46-a925-4abf-834f-73fa213e935f","Type":"ContainerStarted","Data":"9929f2846fdf8787eed572ba77242b1c3058f92f46017569feb0b3c7b27574ec"}
Jul 29 19:57:44.908473 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:44.908338    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-ecc3b0-74486fb68b-ts2zv"
Jul 29 19:57:44.909821 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:44.909801    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecc3b0-76bbcb8684-msbwt" event={"ID":"e74ef00d-1d0c-4f45-99a5-4a593bb566ce","Type":"ContainerStarted","Data":"dee8d1a28aaa26850477b9621d0376531fd5cfa9aa9fd69df2273ca8c7ccef7a"}
Jul 29 19:57:44.909923 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:44.909833    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecc3b0-76bbcb8684-msbwt" event={"ID":"e74ef00d-1d0c-4f45-99a5-4a593bb566ce","Type":"ContainerStarted","Data":"2bca863cef48dcba0d0d447892801ceeb2f3ee6efa0bcb0394e2aa485d94dd5c"}
Jul 29 19:57:44.909923 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:44.909889    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecc3b0-76bbcb8684-msbwt"
Jul 29 19:57:44.911180 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:44.911160    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb" event={"ID":"7b86ead8-b3b9-4d78-9225-f30fc36b8ec6","Type":"ContainerStarted","Data":"5edc0dc622a1af6872c8dc5b26ae6609321fcf5a32f2d527b52de6ac5081ffaa"}
Jul 29 19:57:44.911314 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:44.911300    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb"
Jul 29 19:57:44.925533 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:44.925487    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-ecc3b0-74486fb68b-ts2zv" podStartSLOduration=1.925473934 podStartE2EDuration="1.925473934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:57:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:57:44.924486475 +0000 UTC m=+2435.751985537" watchObservedRunningTime="2026-07-29 19:57:44.925473934 +0000 UTC m=+2435.752973005"
Jul 29 19:57:44.942737 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:44.942687    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb" podStartSLOduration=1.425854964 podStartE2EDuration="1.942671481s" totalImagesPullingTime="516.816517ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:57:44.941058719 +0000 UTC m=+2435.768557804" watchObservedRunningTime="2026-07-29 19:57:44.942671481 +0000 UTC m=+2435.770170540"
Jul 29 19:57:44.957838 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:44.957798    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecc3b0-76bbcb8684-msbwt" podStartSLOduration=1.9577857220000001 podStartE2EDuration="1.957785722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:57:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:57:44.956321696 +0000 UTC m=+2435.783820753" watchObservedRunningTime="2026-07-29 19:57:44.957785722 +0000 UTC m=+2435.785284780"
Jul 29 19:57:50.922071 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:50.922040    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb"
Jul 29 19:57:55.918009 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:55.917955    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecc3b0-76bbcb8684-msbwt"
Jul 29 19:57:55.918422 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:57:55.918046    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-ecc3b0-74486fb68b-ts2zv"
Jul 29 19:58:09.932656 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:09.932617    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466674-mp2s7"]
Jul 29 19:58:09.937402 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:09.937380    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466674-mp2s7"
Jul 29 19:58:09.940217 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:09.940199    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jsq2r\""
Jul 29 19:58:09.944496 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:09.944471    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466674-mp2s7"]
Jul 29 19:58:10.066379 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:10.066343    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb020953-ffb0-431d-b706-d6c370bcc4f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-466674-mp2s7\" (UID: \"cb020953-ffb0-431d-b706-d6c370bcc4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466674-mp2s7"
Jul 29 19:58:10.167258 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:10.167223    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb020953-ffb0-431d-b706-d6c370bcc4f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-466674-mp2s7\" (UID: \"cb020953-ffb0-431d-b706-d6c370bcc4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466674-mp2s7"
Jul 29 19:58:10.167868 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:10.167847    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb020953-ffb0-431d-b706-d6c370bcc4f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-466674-mp2s7\" (UID: \"cb020953-ffb0-431d-b706-d6c370bcc4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466674-mp2s7"
Jul 29 19:58:10.250105 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:10.250076    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466674-mp2s7"
Jul 29 19:58:10.410133 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:10.409940    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466674-mp2s7"]
Jul 29 19:58:10.412459 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:58:10.412431    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb020953_ffb0_431d_b706_d6c370bcc4f5.slice/crio-3e2f5bbf861f6b3eed5a76ac90b1a1a048eaec5f8625c4607079c582bc3b443e WatchSource:0}: Error finding container 3e2f5bbf861f6b3eed5a76ac90b1a1a048eaec5f8625c4607079c582bc3b443e: Status 404 returned error can't find the container with id 3e2f5bbf861f6b3eed5a76ac90b1a1a048eaec5f8625c4607079c582bc3b443e
Jul 29 19:58:11.036252 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:11.036159    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466674-mp2s7" event={"ID":"cb020953-ffb0-431d-b706-d6c370bcc4f5","Type":"ContainerStarted","Data":"1c167c2e003e6435259cfc848354d4f6be97d1e824e0718d7f67a3aa23f474ad"}
Jul 29 19:58:11.036252 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:11.036214    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466674-mp2s7" event={"ID":"cb020953-ffb0-431d-b706-d6c370bcc4f5","Type":"ContainerStarted","Data":"3e2f5bbf861f6b3eed5a76ac90b1a1a048eaec5f8625c4607079c582bc3b443e"}
Jul 29 19:58:11.056025 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:11.055960    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466674-mp2s7" podStartSLOduration=2.055945365 podStartE2EDuration="2.055945365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:58:11.055752754 +0000 UTC m=+2461.883251833" watchObservedRunningTime="2026-07-29 19:58:11.055945365 +0000 UTC m=+2461.883444423"
Jul 29 19:58:12.041804 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:12.041766    2557 generic.go:356] "Generic (PLEG): container finished" podID="cb020953-ffb0-431d-b706-d6c370bcc4f5" containerID="1c167c2e003e6435259cfc848354d4f6be97d1e824e0718d7f67a3aa23f474ad" exitCode=0
Jul 29 19:58:12.042221 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:12.041855    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466674-mp2s7" event={"ID":"cb020953-ffb0-431d-b706-d6c370bcc4f5","Type":"ContainerDied","Data":"1c167c2e003e6435259cfc848354d4f6be97d1e824e0718d7f67a3aa23f474ad"}
Jul 29 19:58:12.418295 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:12.418261    2557 scope.go:117] "RemoveContainer" containerID="027c507ac2b7fc1fe05f471cb81e0dfce99fb251c3db76866dfc8c00fcb16f07"
Jul 29 19:58:13.183669 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:13.183641    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466674-mp2s7"
Jul 29 19:58:13.292875 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:13.292840    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb020953-ffb0-431d-b706-d6c370bcc4f5-maas-api-service-ca\") pod \"cb020953-ffb0-431d-b706-d6c370bcc4f5\" (UID: \"cb020953-ffb0-431d-b706-d6c370bcc4f5\") "
Jul 29 19:58:13.293239 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:13.293214    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb020953-ffb0-431d-b706-d6c370bcc4f5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cb020953-ffb0-431d-b706-d6c370bcc4f5" (UID: "cb020953-ffb0-431d-b706-d6c370bcc4f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:58:13.394409 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:13.394377    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb020953-ffb0-431d-b706-d6c370bcc4f5-maas-api-service-ca\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:58:14.052763 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:14.052683    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466674-mp2s7"
Jul 29 19:58:14.052921 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:14.052684    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466674-mp2s7" event={"ID":"cb020953-ffb0-431d-b706-d6c370bcc4f5","Type":"ContainerDied","Data":"3e2f5bbf861f6b3eed5a76ac90b1a1a048eaec5f8625c4607079c582bc3b443e"}
Jul 29 19:58:14.052921 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:14.052818    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e2f5bbf861f6b3eed5a76ac90b1a1a048eaec5f8625c4607079c582bc3b443e"
Jul 29 19:58:25.680438 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:25.680401    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-466674-7f5f7bff55-c854p"]
Jul 29 19:58:25.680991 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:25.680644    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466674-7f5f7bff55-c854p" podUID="2da6ccc0-0ba1-435f-964e-7e67ba0a7958" containerName="maas-api" containerID="cri-o://27c496b0d9ee24e2fbfbf44df49699c8f85b2a5cde85ba9cb39be9c1c31bdea6" gracePeriod=30
Jul 29 19:58:25.732116 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:25.732085    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-466674-7d76c6c5f4-vnz65"]
Jul 29 19:58:25.732334 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:25.732309    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-466674-7d76c6c5f4-vnz65" podUID="6e7ffaaa-ff47-4c2b-871a-2c75da49dab3" containerName="payload-processing" containerID="cri-o://2066425feecb2a1dc3ddc27bb325a659f27736c0f85d24e5a445c736a207c815" gracePeriod=30
Jul 29 19:58:25.740817 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:25.739699    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-466674-6d875577b8-gjjk2"]
Jul 29 19:58:25.740817 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:25.739917    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466674-6d875577b8-gjjk2" podUID="62476440-b3d0-4e31-97e9-a777dbca6972" containerName="payload-pre-processing" containerID="cri-o://15df6d91b1988100e0de7b7b5a2a97d778548fbcf3203b9c630fbe096cb2e050" gracePeriod=30
Jul 29 19:58:25.885603 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:25.885566    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466674-mp2s7"]
Jul 29 19:58:25.889705 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:25.889670    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466674-mp2s7"]
Jul 29 19:58:26.009108 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.009077    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466674-7f5f7bff55-c854p"
Jul 29 19:58:26.011259 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.011240    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-466674-7d76c6c5f4-vnz65"
Jul 29 19:58:26.032762 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.032734    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466674-6d875577b8-gjjk2"
Jul 29 19:58:26.107578 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.107551    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2jzj\" (UniqueName: \"kubernetes.io/projected/6e7ffaaa-ff47-4c2b-871a-2c75da49dab3-kube-api-access-h2jzj\") pod \"6e7ffaaa-ff47-4c2b-871a-2c75da49dab3\" (UID: \"6e7ffaaa-ff47-4c2b-871a-2c75da49dab3\") "
Jul 29 19:58:26.107758 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.107616    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5jhf\" (UniqueName: \"kubernetes.io/projected/62476440-b3d0-4e31-97e9-a777dbca6972-kube-api-access-x5jhf\") pod \"62476440-b3d0-4e31-97e9-a777dbca6972\" (UID: \"62476440-b3d0-4e31-97e9-a777dbca6972\") "
Jul 29 19:58:26.107758 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.107669    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7ffaaa-ff47-4c2b-871a-2c75da49dab3-plugins-config-volume\") pod \"6e7ffaaa-ff47-4c2b-871a-2c75da49dab3\" (UID: \"6e7ffaaa-ff47-4c2b-871a-2c75da49dab3\") "
Jul 29 19:58:26.107758 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.107701    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rd845\" (UniqueName: \"kubernetes.io/projected/2da6ccc0-0ba1-435f-964e-7e67ba0a7958-kube-api-access-rd845\") pod \"2da6ccc0-0ba1-435f-964e-7e67ba0a7958\" (UID: \"2da6ccc0-0ba1-435f-964e-7e67ba0a7958\") "
Jul 29 19:58:26.107758 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.107726    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da6ccc0-0ba1-435f-964e-7e67ba0a7958-openshift-service-ca\") pod \"2da6ccc0-0ba1-435f-964e-7e67ba0a7958\" (UID: \"2da6ccc0-0ba1-435f-964e-7e67ba0a7958\") "
Jul 29 19:58:26.107758 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.107754    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62476440-b3d0-4e31-97e9-a777dbca6972-plugins-config-volume\") pod \"62476440-b3d0-4e31-97e9-a777dbca6972\" (UID: \"62476440-b3d0-4e31-97e9-a777dbca6972\") "
Jul 29 19:58:26.108087 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.107782    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2da6ccc0-0ba1-435f-964e-7e67ba0a7958-maas-api-tls\") pod \"2da6ccc0-0ba1-435f-964e-7e67ba0a7958\" (UID: \"2da6ccc0-0ba1-435f-964e-7e67ba0a7958\") "
Jul 29 19:58:26.108174 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.108144    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2da6ccc0-0ba1-435f-964e-7e67ba0a7958-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2da6ccc0-0ba1-435f-964e-7e67ba0a7958" (UID: "2da6ccc0-0ba1-435f-964e-7e67ba0a7958"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:58:26.108262 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.108206    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62476440-b3d0-4e31-97e9-a777dbca6972-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "62476440-b3d0-4e31-97e9-a777dbca6972" (UID: "62476440-b3d0-4e31-97e9-a777dbca6972"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:58:26.109025 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.108564    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e7ffaaa-ff47-4c2b-871a-2c75da49dab3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6e7ffaaa-ff47-4c2b-871a-2c75da49dab3" (UID: "6e7ffaaa-ff47-4c2b-871a-2c75da49dab3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:58:26.109025 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.108673    2557 generic.go:356] "Generic (PLEG): container finished" podID="2da6ccc0-0ba1-435f-964e-7e67ba0a7958" containerID="27c496b0d9ee24e2fbfbf44df49699c8f85b2a5cde85ba9cb39be9c1c31bdea6" exitCode=0
Jul 29 19:58:26.109025 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.108745    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466674-7f5f7bff55-c854p"
Jul 29 19:58:26.109025 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.108773    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466674-7f5f7bff55-c854p" event={"ID":"2da6ccc0-0ba1-435f-964e-7e67ba0a7958","Type":"ContainerDied","Data":"27c496b0d9ee24e2fbfbf44df49699c8f85b2a5cde85ba9cb39be9c1c31bdea6"}
Jul 29 19:58:26.109025 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.108847    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466674-7f5f7bff55-c854p" event={"ID":"2da6ccc0-0ba1-435f-964e-7e67ba0a7958","Type":"ContainerDied","Data":"8794746ac59816060dafec3a2c7fda5129876387241d50820febe4a261f75f3b"}
Jul 29 19:58:26.109025 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.108875    2557 scope.go:117] "RemoveContainer" containerID="27c496b0d9ee24e2fbfbf44df49699c8f85b2a5cde85ba9cb39be9c1c31bdea6"
Jul 29 19:58:26.110195 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.110120    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da6ccc0-0ba1-435f-964e-7e67ba0a7958-kube-api-access-rd845" (OuterVolumeSpecName: "kube-api-access-rd845") pod "2da6ccc0-0ba1-435f-964e-7e67ba0a7958" (UID: "2da6ccc0-0ba1-435f-964e-7e67ba0a7958"). InnerVolumeSpecName "kube-api-access-rd845". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:58:26.110325 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.110219    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e7ffaaa-ff47-4c2b-871a-2c75da49dab3-kube-api-access-h2jzj" (OuterVolumeSpecName: "kube-api-access-h2jzj") pod "6e7ffaaa-ff47-4c2b-871a-2c75da49dab3" (UID: "6e7ffaaa-ff47-4c2b-871a-2c75da49dab3"). InnerVolumeSpecName "kube-api-access-h2jzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:58:26.110476 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.110327    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2da6ccc0-0ba1-435f-964e-7e67ba0a7958-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2da6ccc0-0ba1-435f-964e-7e67ba0a7958" (UID: "2da6ccc0-0ba1-435f-964e-7e67ba0a7958"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:58:26.110752 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.110725    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62476440-b3d0-4e31-97e9-a777dbca6972-kube-api-access-x5jhf" (OuterVolumeSpecName: "kube-api-access-x5jhf") pod "62476440-b3d0-4e31-97e9-a777dbca6972" (UID: "62476440-b3d0-4e31-97e9-a777dbca6972"). InnerVolumeSpecName "kube-api-access-x5jhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:58:26.110845 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.110809    2557 generic.go:356] "Generic (PLEG): container finished" podID="6e7ffaaa-ff47-4c2b-871a-2c75da49dab3" containerID="2066425feecb2a1dc3ddc27bb325a659f27736c0f85d24e5a445c736a207c815" exitCode=0
Jul 29 19:58:26.110905 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.110860    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-466674-7d76c6c5f4-vnz65"
Jul 29 19:58:26.110905 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.110889    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-466674-7d76c6c5f4-vnz65" event={"ID":"6e7ffaaa-ff47-4c2b-871a-2c75da49dab3","Type":"ContainerDied","Data":"2066425feecb2a1dc3ddc27bb325a659f27736c0f85d24e5a445c736a207c815"}
Jul 29 19:58:26.111039 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.110938    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-466674-7d76c6c5f4-vnz65" event={"ID":"6e7ffaaa-ff47-4c2b-871a-2c75da49dab3","Type":"ContainerDied","Data":"487c9db95fa532be6cafd38c15bcd172bf8b5d480b6067d0ea9139a999c999fd"}
Jul 29 19:58:26.112306 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.112271    2557 generic.go:356] "Generic (PLEG): container finished" podID="62476440-b3d0-4e31-97e9-a777dbca6972" containerID="15df6d91b1988100e0de7b7b5a2a97d778548fbcf3203b9c630fbe096cb2e050" exitCode=0
Jul 29 19:58:26.112646 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.112332    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466674-6d875577b8-gjjk2"
Jul 29 19:58:26.112646 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.112334    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466674-6d875577b8-gjjk2" event={"ID":"62476440-b3d0-4e31-97e9-a777dbca6972","Type":"ContainerDied","Data":"15df6d91b1988100e0de7b7b5a2a97d778548fbcf3203b9c630fbe096cb2e050"}
Jul 29 19:58:26.112646 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.112628    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466674-6d875577b8-gjjk2" event={"ID":"62476440-b3d0-4e31-97e9-a777dbca6972","Type":"ContainerDied","Data":"a978da6dd7bd17a4f8ce3a78f760fc5b06b5adc18cd8b13b83f8133bdf091e80"}
Jul 29 19:58:26.120432 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.120405    2557 scope.go:117] "RemoveContainer" containerID="27c496b0d9ee24e2fbfbf44df49699c8f85b2a5cde85ba9cb39be9c1c31bdea6"
Jul 29 19:58:26.120818 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:58:26.120787    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c496b0d9ee24e2fbfbf44df49699c8f85b2a5cde85ba9cb39be9c1c31bdea6\": container with ID starting with 27c496b0d9ee24e2fbfbf44df49699c8f85b2a5cde85ba9cb39be9c1c31bdea6 not found: ID does not exist" containerID="27c496b0d9ee24e2fbfbf44df49699c8f85b2a5cde85ba9cb39be9c1c31bdea6"
Jul 29 19:58:26.120924 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.120828    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c496b0d9ee24e2fbfbf44df49699c8f85b2a5cde85ba9cb39be9c1c31bdea6"} err="failed to get container status \"27c496b0d9ee24e2fbfbf44df49699c8f85b2a5cde85ba9cb39be9c1c31bdea6\": rpc error: code = NotFound desc = could not find container \"27c496b0d9ee24e2fbfbf44df49699c8f85b2a5cde85ba9cb39be9c1c31bdea6\": container with ID starting with 27c496b0d9ee24e2fbfbf44df49699c8f85b2a5cde85ba9cb39be9c1c31bdea6 not found: ID does not exist"
Jul 29 19:58:26.120924 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.120851    2557 scope.go:117] "RemoveContainer" containerID="2066425feecb2a1dc3ddc27bb325a659f27736c0f85d24e5a445c736a207c815"
Jul 29 19:58:26.131259 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.131167    2557 scope.go:117] "RemoveContainer" containerID="2066425feecb2a1dc3ddc27bb325a659f27736c0f85d24e5a445c736a207c815"
Jul 29 19:58:26.131886 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:58:26.131863    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2066425feecb2a1dc3ddc27bb325a659f27736c0f85d24e5a445c736a207c815\": container with ID starting with 2066425feecb2a1dc3ddc27bb325a659f27736c0f85d24e5a445c736a207c815 not found: ID does not exist" containerID="2066425feecb2a1dc3ddc27bb325a659f27736c0f85d24e5a445c736a207c815"
Jul 29 19:58:26.131954 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.131893    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2066425feecb2a1dc3ddc27bb325a659f27736c0f85d24e5a445c736a207c815"} err="failed to get container status \"2066425feecb2a1dc3ddc27bb325a659f27736c0f85d24e5a445c736a207c815\": rpc error: code = NotFound desc = could not find container \"2066425feecb2a1dc3ddc27bb325a659f27736c0f85d24e5a445c736a207c815\": container with ID starting with 2066425feecb2a1dc3ddc27bb325a659f27736c0f85d24e5a445c736a207c815 not found: ID does not exist"
Jul 29 19:58:26.131954 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.131911    2557 scope.go:117] "RemoveContainer" containerID="15df6d91b1988100e0de7b7b5a2a97d778548fbcf3203b9c630fbe096cb2e050"
Jul 29 19:58:26.140290 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.140257    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-466674-7d76c6c5f4-vnz65"]
Jul 29 19:58:26.140933 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.140914    2557 scope.go:117] "RemoveContainer" containerID="15df6d91b1988100e0de7b7b5a2a97d778548fbcf3203b9c630fbe096cb2e050"
Jul 29 19:58:26.141240 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:58:26.141218    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15df6d91b1988100e0de7b7b5a2a97d778548fbcf3203b9c630fbe096cb2e050\": container with ID starting with 15df6d91b1988100e0de7b7b5a2a97d778548fbcf3203b9c630fbe096cb2e050 not found: ID does not exist" containerID="15df6d91b1988100e0de7b7b5a2a97d778548fbcf3203b9c630fbe096cb2e050"
Jul 29 19:58:26.141319 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.141246    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15df6d91b1988100e0de7b7b5a2a97d778548fbcf3203b9c630fbe096cb2e050"} err="failed to get container status \"15df6d91b1988100e0de7b7b5a2a97d778548fbcf3203b9c630fbe096cb2e050\": rpc error: code = NotFound desc = could not find container \"15df6d91b1988100e0de7b7b5a2a97d778548fbcf3203b9c630fbe096cb2e050\": container with ID starting with 15df6d91b1988100e0de7b7b5a2a97d778548fbcf3203b9c630fbe096cb2e050 not found: ID does not exist"
Jul 29 19:58:26.142585 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.142567    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-466674-7d76c6c5f4-vnz65"]
Jul 29 19:58:26.153474 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.153448    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-466674-6d875577b8-gjjk2"]
Jul 29 19:58:26.158708 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.158682    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-466674-6d875577b8-gjjk2"]
Jul 29 19:58:26.208769 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.208697    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7ffaaa-ff47-4c2b-871a-2c75da49dab3-plugins-config-volume\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:58:26.208769 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.208724    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd845\" (UniqueName: \"kubernetes.io/projected/2da6ccc0-0ba1-435f-964e-7e67ba0a7958-kube-api-access-rd845\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:58:26.208769 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.208736    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da6ccc0-0ba1-435f-964e-7e67ba0a7958-openshift-service-ca\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:58:26.208769 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.208747    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62476440-b3d0-4e31-97e9-a777dbca6972-plugins-config-volume\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:58:26.208769 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.208758    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2da6ccc0-0ba1-435f-964e-7e67ba0a7958-maas-api-tls\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:58:26.208769 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.208766    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2jzj\" (UniqueName: \"kubernetes.io/projected/6e7ffaaa-ff47-4c2b-871a-2c75da49dab3-kube-api-access-h2jzj\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:58:26.208769 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.208775    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5jhf\" (UniqueName: \"kubernetes.io/projected/62476440-b3d0-4e31-97e9-a777dbca6972-kube-api-access-x5jhf\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:58:26.430870 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.430840    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-466674-7f5f7bff55-c854p"]
Jul 29 19:58:26.434424 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:26.434400    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-466674-7f5f7bff55-c854p"]
Jul 29 19:58:27.678790 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:27.678753    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2da6ccc0-0ba1-435f-964e-7e67ba0a7958" path="/var/lib/kubelet/pods/2da6ccc0-0ba1-435f-964e-7e67ba0a7958/volumes"
Jul 29 19:58:27.679182 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:27.679114    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62476440-b3d0-4e31-97e9-a777dbca6972" path="/var/lib/kubelet/pods/62476440-b3d0-4e31-97e9-a777dbca6972/volumes"
Jul 29 19:58:27.679435 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:27.679421    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e7ffaaa-ff47-4c2b-871a-2c75da49dab3" path="/var/lib/kubelet/pods/6e7ffaaa-ff47-4c2b-871a-2c75da49dab3/volumes"
Jul 29 19:58:27.679705 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:27.679692    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb020953-ffb0-431d-b706-d6c370bcc4f5" path="/var/lib/kubelet/pods/cb020953-ffb0-431d-b706-d6c370bcc4f5/volumes"
Jul 29 19:58:32.550249 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:32.550214    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ecc3b0-m5h4s"]
Jul 29 19:58:32.550622 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:32.550568    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e7ffaaa-ff47-4c2b-871a-2c75da49dab3" containerName="payload-processing"
Jul 29 19:58:32.550622 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:32.550579    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e7ffaaa-ff47-4c2b-871a-2c75da49dab3" containerName="payload-processing"
Jul 29 19:58:32.550622 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:32.550592    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2da6ccc0-0ba1-435f-964e-7e67ba0a7958" containerName="maas-api"
Jul 29 19:58:32.550622 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:32.550598    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2da6ccc0-0ba1-435f-964e-7e67ba0a7958" containerName="maas-api"
Jul 29 19:58:32.550622 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:32.550611    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62476440-b3d0-4e31-97e9-a777dbca6972" containerName="payload-pre-processing"
Jul 29 19:58:32.550622 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:32.550617    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="62476440-b3d0-4e31-97e9-a777dbca6972" containerName="payload-pre-processing"
Jul 29 19:58:32.550622 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:32.550625    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb020953-ffb0-431d-b706-d6c370bcc4f5" containerName="revoke-keys"
Jul 29 19:58:32.550836 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:32.550632    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb020953-ffb0-431d-b706-d6c370bcc4f5" containerName="revoke-keys"
Jul 29 19:58:32.550836 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:32.550684    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e7ffaaa-ff47-4c2b-871a-2c75da49dab3" containerName="payload-processing"
Jul 29 19:58:32.550836 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:32.550693    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb020953-ffb0-431d-b706-d6c370bcc4f5" containerName="revoke-keys"
Jul 29 19:58:32.550836 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:32.550702    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="62476440-b3d0-4e31-97e9-a777dbca6972" containerName="payload-pre-processing"
Jul 29 19:58:32.550836 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:32.550708    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2da6ccc0-0ba1-435f-964e-7e67ba0a7958" containerName="maas-api"
Jul 29 19:58:32.555343 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:32.555324    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ecc3b0-m5h4s"
Jul 29 19:58:32.558714 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:32.558693    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jsq2r\""
Jul 29 19:58:32.562659 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:32.562530    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ecc3b0-m5h4s"]
Jul 29 19:58:32.667691 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:32.667652    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04cdc66d-ac7b-41e7-866a-a2fd47ab8613-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ecc3b0-m5h4s\" (UID: \"04cdc66d-ac7b-41e7-866a-a2fd47ab8613\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ecc3b0-m5h4s"
Jul 29 19:58:32.769012 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:32.768950    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04cdc66d-ac7b-41e7-866a-a2fd47ab8613-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ecc3b0-m5h4s\" (UID: \"04cdc66d-ac7b-41e7-866a-a2fd47ab8613\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ecc3b0-m5h4s"
Jul 29 19:58:32.769601 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:32.769582    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04cdc66d-ac7b-41e7-866a-a2fd47ab8613-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ecc3b0-m5h4s\" (UID: \"04cdc66d-ac7b-41e7-866a-a2fd47ab8613\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ecc3b0-m5h4s"
Jul 29 19:58:32.867777 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:32.867695    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ecc3b0-m5h4s"
Jul 29 19:58:33.007091 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:33.007067    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ecc3b0-m5h4s"]
Jul 29 19:58:33.009329 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:58:33.009295    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04cdc66d_ac7b_41e7_866a_a2fd47ab8613.slice/crio-9e5d64d8d22fdd403603651a8c7759308fd6c3f4158a8d28ff2036d2ab874cf3 WatchSource:0}: Error finding container 9e5d64d8d22fdd403603651a8c7759308fd6c3f4158a8d28ff2036d2ab874cf3: Status 404 returned error can't find the container with id 9e5d64d8d22fdd403603651a8c7759308fd6c3f4158a8d28ff2036d2ab874cf3
Jul 29 19:58:33.151066 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:33.151032    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ecc3b0-m5h4s" event={"ID":"04cdc66d-ac7b-41e7-866a-a2fd47ab8613","Type":"ContainerStarted","Data":"9e5d64d8d22fdd403603651a8c7759308fd6c3f4158a8d28ff2036d2ab874cf3"}
Jul 29 19:58:34.157803 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:34.157769    2557 generic.go:356] "Generic (PLEG): container finished" podID="04cdc66d-ac7b-41e7-866a-a2fd47ab8613" containerID="15d95e67848762e0a512f305b5614a595cfc6c25a3313c05262ff1feda1ebd56" exitCode=0
Jul 29 19:58:34.158192 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:34.157846    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ecc3b0-m5h4s" event={"ID":"04cdc66d-ac7b-41e7-866a-a2fd47ab8613","Type":"ContainerDied","Data":"15d95e67848762e0a512f305b5614a595cfc6c25a3313c05262ff1feda1ebd56"}
Jul 29 19:58:35.287351 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:35.287324    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ecc3b0-m5h4s"
Jul 29 19:58:35.389739 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:35.389688    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04cdc66d-ac7b-41e7-866a-a2fd47ab8613-maas-api-service-ca\") pod \"04cdc66d-ac7b-41e7-866a-a2fd47ab8613\" (UID: \"04cdc66d-ac7b-41e7-866a-a2fd47ab8613\") "
Jul 29 19:58:35.390072 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:35.390050    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04cdc66d-ac7b-41e7-866a-a2fd47ab8613-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "04cdc66d-ac7b-41e7-866a-a2fd47ab8613" (UID: "04cdc66d-ac7b-41e7-866a-a2fd47ab8613"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:58:35.490876 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:35.490786    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04cdc66d-ac7b-41e7-866a-a2fd47ab8613-maas-api-service-ca\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:58:36.167389 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:36.167349    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ecc3b0-m5h4s" event={"ID":"04cdc66d-ac7b-41e7-866a-a2fd47ab8613","Type":"ContainerDied","Data":"9e5d64d8d22fdd403603651a8c7759308fd6c3f4158a8d28ff2036d2ab874cf3"}
Jul 29 19:58:36.167566 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:36.167408    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e5d64d8d22fdd403603651a8c7759308fd6c3f4158a8d28ff2036d2ab874cf3"
Jul 29 19:58:36.167566 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:36.167370    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ecc3b0-m5h4s"
Jul 29 19:58:48.068364 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:48.068329    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb"]
Jul 29 19:58:48.068771 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:48.068602    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb" podUID="7b86ead8-b3b9-4d78-9225-f30fc36b8ec6" containerName="maas-api" containerID="cri-o://5edc0dc622a1af6872c8dc5b26ae6609321fcf5a32f2d527b52de6ac5081ffaa" gracePeriod=30
Jul 29 19:58:48.129021 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:48.128987    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ecc3b0-74486fb68b-ts2zv"]
Jul 29 19:58:48.129320 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:48.129260    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-ecc3b0-74486fb68b-ts2zv" podUID="db4c2b46-a925-4abf-834f-73fa213e935f" containerName="payload-processing" containerID="cri-o://0cace4d1c5a5deccfb3882c5ea074bb6a418dee390ac957175c01e829ff998d7" gracePeriod=30
Jul 29 19:58:48.141155 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:48.141128    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ecc3b0-76bbcb8684-msbwt"]
Jul 29 19:58:48.141353 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:48.141332    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecc3b0-76bbcb8684-msbwt" podUID="e74ef00d-1d0c-4f45-99a5-4a593bb566ce" containerName="payload-pre-processing" containerID="cri-o://dee8d1a28aaa26850477b9621d0376531fd5cfa9aa9fd69df2273ca8c7ccef7a" gracePeriod=30
Jul 29 19:58:48.222222 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:48.222194    2557 generic.go:356] "Generic (PLEG): container finished" podID="7b86ead8-b3b9-4d78-9225-f30fc36b8ec6" containerID="5edc0dc622a1af6872c8dc5b26ae6609321fcf5a32f2d527b52de6ac5081ffaa" exitCode=0
Jul 29 19:58:48.222314 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:48.222267    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb" event={"ID":"7b86ead8-b3b9-4d78-9225-f30fc36b8ec6","Type":"ContainerDied","Data":"5edc0dc622a1af6872c8dc5b26ae6609321fcf5a32f2d527b52de6ac5081ffaa"}
Jul 29 19:58:48.302468 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:48.302438    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ecc3b0-m5h4s"]
Jul 29 19:58:48.304754 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:48.304722    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ecc3b0-m5h4s"]
Jul 29 19:58:48.400480 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:48.400454    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb"
Jul 29 19:58:48.412850 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:48.412828    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ecc3b0-74486fb68b-ts2zv"
Jul 29 19:58:48.423032 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:48.423008    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecc3b0-76bbcb8684-msbwt"
Jul 29 19:58:48.504609 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:48.504575    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b86ead8-b3b9-4d78-9225-f30fc36b8ec6-openshift-service-ca\") pod \"7b86ead8-b3b9-4d78-9225-f30fc36b8ec6\" (UID: \"7b86ead8-b3b9-4d78-9225-f30fc36b8ec6\") "
Jul 29 19:58:48.504609 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:48.504613    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db4c2b46-a925-4abf-834f-73fa213e935f-plugins-config-volume\") pod \"db4c2b46-a925-4abf-834f-73fa213e935f\" (UID: \"db4c2b46-a925-4abf-834f-73fa213e935f\") "
Jul 29 19:58:48.504857 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:48.504649    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b86ead8-b3b9-4d78-9225-f30fc36b8ec6-maas-api-tls\") pod \"7b86ead8-b3b9-4d78-9225-f30fc36b8ec6\" (UID: \"7b86ead8-b3b9-4d78-9225-f30fc36b8ec6\") "
Jul 29 19:58:48.504857 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:48.504666    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e74ef00d-1d0c-4f45-99a5-4a593bb566ce-plugins-config-volume\") pod \"e74ef00d-1d0c-4f45-99a5-4a593bb566ce\" (UID: \"e74ef00d-1d0c-4f45-99a5-4a593bb566ce\") "
Jul 29 19:58:48.504857 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:48.504693    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wr7z\" (UniqueName: \"kubernetes.io/projected/e74ef00d-1d0c-4f45-99a5-4a593bb566ce-kube-api-access-5wr7z\") pod \"e74ef00d-1d0c-4f45-99a5-4a593bb566ce\" (UID: \"e74ef00d-1d0c-4f45-99a5-4a593bb566ce\") "
Jul 29 19:58:48.504857 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:48.504723    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dsdw\" (UniqueName: \"kubernetes.io/projected/db4c2b46-a925-4abf-834f-73fa213e935f-kube-api-access-8dsdw\") pod \"db4c2b46-a925-4abf-834f-73fa213e935f\" (UID: \"db4c2b46-a925-4abf-834f-73fa213e935f\") "
Jul 29 19:58:48.504857 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:48.504748    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kz77j\" (UniqueName: \"kubernetes.io/projected/7b86ead8-b3b9-4d78-9225-f30fc36b8ec6-kube-api-access-kz77j\") pod \"7b86ead8-b3b9-4d78-9225-f30fc36b8ec6\" (UID: \"7b86ead8-b3b9-4d78-9225-f30fc36b8ec6\") "
Jul 29 19:58:48.505144 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:48.505051    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b86ead8-b3b9-4d78-9225-f30fc36b8ec6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7b86ead8-b3b9-4d78-9225-f30fc36b8ec6" (UID: "7b86ead8-b3b9-4d78-9225-f30fc36b8ec6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:58:48.505144 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:48.505070    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db4c2b46-a925-4abf-834f-73fa213e935f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "db4c2b46-a925-4abf-834f-73fa213e935f" (UID: "db4c2b46-a925-4abf-834f-73fa213e935f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:58:48.505144 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:48.505074    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e74ef00d-1d0c-4f45-99a5-4a593bb566ce-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e74ef00d-1d0c-4f45-99a5-4a593bb566ce" (UID: "e74ef00d-1d0c-4f45-99a5-4a593bb566ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:58:48.506879 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:48.506848    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e74ef00d-1d0c-4f45-99a5-4a593bb566ce-kube-api-access-5wr7z" (OuterVolumeSpecName: "kube-api-access-5wr7z") pod "e74ef00d-1d0c-4f45-99a5-4a593bb566ce" (UID: "e74ef00d-1d0c-4f45-99a5-4a593bb566ce"). InnerVolumeSpecName "kube-api-access-5wr7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:58:48.507077 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:48.506880    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b86ead8-b3b9-4d78-9225-f30fc36b8ec6-kube-api-access-kz77j" (OuterVolumeSpecName: "kube-api-access-kz77j") pod "7b86ead8-b3b9-4d78-9225-f30fc36b8ec6" (UID: "7b86ead8-b3b9-4d78-9225-f30fc36b8ec6"). InnerVolumeSpecName "kube-api-access-kz77j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:58:48.507077 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:48.506885    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b86ead8-b3b9-4d78-9225-f30fc36b8ec6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7b86ead8-b3b9-4d78-9225-f30fc36b8ec6" (UID: "7b86ead8-b3b9-4d78-9225-f30fc36b8ec6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:58:48.507077 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:48.506895    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db4c2b46-a925-4abf-834f-73fa213e935f-kube-api-access-8dsdw" (OuterVolumeSpecName: "kube-api-access-8dsdw") pod "db4c2b46-a925-4abf-834f-73fa213e935f" (UID: "db4c2b46-a925-4abf-834f-73fa213e935f"). InnerVolumeSpecName "kube-api-access-8dsdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:58:48.605589 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:48.605491    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz77j\" (UniqueName: \"kubernetes.io/projected/7b86ead8-b3b9-4d78-9225-f30fc36b8ec6-kube-api-access-kz77j\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:58:48.605589 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:48.605532    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b86ead8-b3b9-4d78-9225-f30fc36b8ec6-openshift-service-ca\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:58:48.605589 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:48.605542    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db4c2b46-a925-4abf-834f-73fa213e935f-plugins-config-volume\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:58:48.605589 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:48.605552    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b86ead8-b3b9-4d78-9225-f30fc36b8ec6-maas-api-tls\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:58:48.605589 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:48.605561    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e74ef00d-1d0c-4f45-99a5-4a593bb566ce-plugins-config-volume\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:58:48.605589 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:48.605570    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wr7z\" (UniqueName: \"kubernetes.io/projected/e74ef00d-1d0c-4f45-99a5-4a593bb566ce-kube-api-access-5wr7z\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:58:48.605589 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:48.605578    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dsdw\" (UniqueName: \"kubernetes.io/projected/db4c2b46-a925-4abf-834f-73fa213e935f-kube-api-access-8dsdw\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 19:58:49.228279 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:49.228242    2557 generic.go:356] "Generic (PLEG): container finished" podID="db4c2b46-a925-4abf-834f-73fa213e935f" containerID="0cace4d1c5a5deccfb3882c5ea074bb6a418dee390ac957175c01e829ff998d7" exitCode=0
Jul 29 19:58:49.228750 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:49.228286    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ecc3b0-74486fb68b-ts2zv"
Jul 29 19:58:49.228750 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:49.228324    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ecc3b0-74486fb68b-ts2zv" event={"ID":"db4c2b46-a925-4abf-834f-73fa213e935f","Type":"ContainerDied","Data":"0cace4d1c5a5deccfb3882c5ea074bb6a418dee390ac957175c01e829ff998d7"}
Jul 29 19:58:49.228750 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:49.228375    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ecc3b0-74486fb68b-ts2zv" event={"ID":"db4c2b46-a925-4abf-834f-73fa213e935f","Type":"ContainerDied","Data":"9929f2846fdf8787eed572ba77242b1c3058f92f46017569feb0b3c7b27574ec"}
Jul 29 19:58:49.228750 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:49.228397    2557 scope.go:117] "RemoveContainer" containerID="0cace4d1c5a5deccfb3882c5ea074bb6a418dee390ac957175c01e829ff998d7"
Jul 29 19:58:49.229790 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:49.229761    2557 generic.go:356] "Generic (PLEG): container finished" podID="e74ef00d-1d0c-4f45-99a5-4a593bb566ce" containerID="dee8d1a28aaa26850477b9621d0376531fd5cfa9aa9fd69df2273ca8c7ccef7a" exitCode=0
Jul 29 19:58:49.229890 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:49.229808    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecc3b0-76bbcb8684-msbwt"
Jul 29 19:58:49.229890 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:49.229863    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecc3b0-76bbcb8684-msbwt" event={"ID":"e74ef00d-1d0c-4f45-99a5-4a593bb566ce","Type":"ContainerDied","Data":"dee8d1a28aaa26850477b9621d0376531fd5cfa9aa9fd69df2273ca8c7ccef7a"}
Jul 29 19:58:49.230118 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:49.229905    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ecc3b0-76bbcb8684-msbwt" event={"ID":"e74ef00d-1d0c-4f45-99a5-4a593bb566ce","Type":"ContainerDied","Data":"2bca863cef48dcba0d0d447892801ceeb2f3ee6efa0bcb0394e2aa485d94dd5c"}
Jul 29 19:58:49.231352 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:49.231328    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb" event={"ID":"7b86ead8-b3b9-4d78-9225-f30fc36b8ec6","Type":"ContainerDied","Data":"783e402c171e482840144a6ed2b788bdae605a1669406061c15e771473897ada"}
Jul 29 19:58:49.231352 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:49.231342    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb"
Jul 29 19:58:49.237274 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:49.237251    2557 scope.go:117] "RemoveContainer" containerID="0cace4d1c5a5deccfb3882c5ea074bb6a418dee390ac957175c01e829ff998d7"
Jul 29 19:58:49.237521 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:58:49.237502    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cace4d1c5a5deccfb3882c5ea074bb6a418dee390ac957175c01e829ff998d7\": container with ID starting with 0cace4d1c5a5deccfb3882c5ea074bb6a418dee390ac957175c01e829ff998d7 not found: ID does not exist" containerID="0cace4d1c5a5deccfb3882c5ea074bb6a418dee390ac957175c01e829ff998d7"
Jul 29 19:58:49.237589 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:49.237527    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cace4d1c5a5deccfb3882c5ea074bb6a418dee390ac957175c01e829ff998d7"} err="failed to get container status \"0cace4d1c5a5deccfb3882c5ea074bb6a418dee390ac957175c01e829ff998d7\": rpc error: code = NotFound desc = could not find container \"0cace4d1c5a5deccfb3882c5ea074bb6a418dee390ac957175c01e829ff998d7\": container with ID starting with 0cace4d1c5a5deccfb3882c5ea074bb6a418dee390ac957175c01e829ff998d7 not found: ID does not exist"
Jul 29 19:58:49.237589 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:49.237545    2557 scope.go:117] "RemoveContainer" containerID="dee8d1a28aaa26850477b9621d0376531fd5cfa9aa9fd69df2273ca8c7ccef7a"
Jul 29 19:58:49.246882 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:49.246867    2557 scope.go:117] "RemoveContainer" containerID="dee8d1a28aaa26850477b9621d0376531fd5cfa9aa9fd69df2273ca8c7ccef7a"
Jul 29 19:58:49.247142 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:58:49.247123    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dee8d1a28aaa26850477b9621d0376531fd5cfa9aa9fd69df2273ca8c7ccef7a\": container with ID starting with dee8d1a28aaa26850477b9621d0376531fd5cfa9aa9fd69df2273ca8c7ccef7a not found: ID does not exist" containerID="dee8d1a28aaa26850477b9621d0376531fd5cfa9aa9fd69df2273ca8c7ccef7a"
Jul 29 19:58:49.247218 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:49.247146    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dee8d1a28aaa26850477b9621d0376531fd5cfa9aa9fd69df2273ca8c7ccef7a"} err="failed to get container status \"dee8d1a28aaa26850477b9621d0376531fd5cfa9aa9fd69df2273ca8c7ccef7a\": rpc error: code = NotFound desc = could not find container \"dee8d1a28aaa26850477b9621d0376531fd5cfa9aa9fd69df2273ca8c7ccef7a\": container with ID starting with dee8d1a28aaa26850477b9621d0376531fd5cfa9aa9fd69df2273ca8c7ccef7a not found: ID does not exist"
Jul 29 19:58:49.247218 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:49.247161    2557 scope.go:117] "RemoveContainer" containerID="5edc0dc622a1af6872c8dc5b26ae6609321fcf5a32f2d527b52de6ac5081ffaa"
Jul 29 19:58:49.256441 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:49.256420    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb"]
Jul 29 19:58:49.259541 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:49.259520    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ecc3b0-7f48747949-c5kjb"]
Jul 29 19:58:49.270076 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:49.270053    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ecc3b0-74486fb68b-ts2zv"]
Jul 29 19:58:49.273950 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:49.273929    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ecc3b0-74486fb68b-ts2zv"]
Jul 29 19:58:49.284309 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:49.284289    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ecc3b0-76bbcb8684-msbwt"]
Jul 29 19:58:49.287676 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:49.287657    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ecc3b0-76bbcb8684-msbwt"]
Jul 29 19:58:49.678638 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:49.678603    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04cdc66d-ac7b-41e7-866a-a2fd47ab8613" path="/var/lib/kubelet/pods/04cdc66d-ac7b-41e7-866a-a2fd47ab8613/volumes"
Jul 29 19:58:49.679024 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:49.679010    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b86ead8-b3b9-4d78-9225-f30fc36b8ec6" path="/var/lib/kubelet/pods/7b86ead8-b3b9-4d78-9225-f30fc36b8ec6/volumes"
Jul 29 19:58:49.679349 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:49.679337    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db4c2b46-a925-4abf-834f-73fa213e935f" path="/var/lib/kubelet/pods/db4c2b46-a925-4abf-834f-73fa213e935f/volumes"
Jul 29 19:58:49.679661 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:49.679650    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e74ef00d-1d0c-4f45-99a5-4a593bb566ce" path="/var/lib/kubelet/pods/e74ef00d-1d0c-4f45-99a5-4a593bb566ce/volumes"
Jul 29 19:58:59.621422 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:59.621385    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2"]
Jul 29 19:58:59.621855 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:59.621735    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db4c2b46-a925-4abf-834f-73fa213e935f" containerName="payload-processing"
Jul 29 19:58:59.621855 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:59.621750    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="db4c2b46-a925-4abf-834f-73fa213e935f" containerName="payload-processing"
Jul 29 19:58:59.621855 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:59.621761    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e74ef00d-1d0c-4f45-99a5-4a593bb566ce" containerName="payload-pre-processing"
Jul 29 19:58:59.621855 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:59.621766    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e74ef00d-1d0c-4f45-99a5-4a593bb566ce" containerName="payload-pre-processing"
Jul 29 19:58:59.621855 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:59.621775    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b86ead8-b3b9-4d78-9225-f30fc36b8ec6" containerName="maas-api"
Jul 29 19:58:59.621855 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:59.621780    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b86ead8-b3b9-4d78-9225-f30fc36b8ec6" containerName="maas-api"
Jul 29 19:58:59.621855 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:59.621787    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04cdc66d-ac7b-41e7-866a-a2fd47ab8613" containerName="revoke-keys"
Jul 29 19:58:59.621855 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:59.621792    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="04cdc66d-ac7b-41e7-866a-a2fd47ab8613" containerName="revoke-keys"
Jul 29 19:58:59.621855 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:59.621849    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b86ead8-b3b9-4d78-9225-f30fc36b8ec6" containerName="maas-api"
Jul 29 19:58:59.621855 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:59.621856    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="db4c2b46-a925-4abf-834f-73fa213e935f" containerName="payload-processing"
Jul 29 19:58:59.622234 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:59.621864    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e74ef00d-1d0c-4f45-99a5-4a593bb566ce" containerName="payload-pre-processing"
Jul 29 19:58:59.622234 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:59.621872    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="04cdc66d-ac7b-41e7-866a-a2fd47ab8613" containerName="revoke-keys"
Jul 29 19:58:59.626739 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:59.626714    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2"
Jul 29 19:58:59.629282 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:59.629256    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-641e39-serving-cert\""
Jul 29 19:58:59.629415 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:59.629394    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4dw6s\""
Jul 29 19:58:59.636661 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:59.636636    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2"]
Jul 29 19:58:59.707663 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:59.707630    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4769186-0b54-46b4-bb07-ecc3a5b92e94-openshift-service-ca\") pod \"maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2\" (UID: \"e4769186-0b54-46b4-bb07-ecc3a5b92e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2"
Jul 29 19:58:59.707663 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:59.707665    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4769186-0b54-46b4-bb07-ecc3a5b92e94-maas-api-tls\") pod \"maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2\" (UID: \"e4769186-0b54-46b4-bb07-ecc3a5b92e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2"
Jul 29 19:58:59.707862 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:59.707708    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2299\" (UniqueName: \"kubernetes.io/projected/e4769186-0b54-46b4-bb07-ecc3a5b92e94-kube-api-access-v2299\") pod \"maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2\" (UID: \"e4769186-0b54-46b4-bb07-ecc3a5b92e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2"
Jul 29 19:58:59.808584 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:59.808550    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2299\" (UniqueName: \"kubernetes.io/projected/e4769186-0b54-46b4-bb07-ecc3a5b92e94-kube-api-access-v2299\") pod \"maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2\" (UID: \"e4769186-0b54-46b4-bb07-ecc3a5b92e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2"
Jul 29 19:58:59.808759 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:59.808638    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4769186-0b54-46b4-bb07-ecc3a5b92e94-openshift-service-ca\") pod \"maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2\" (UID: \"e4769186-0b54-46b4-bb07-ecc3a5b92e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2"
Jul 29 19:58:59.808759 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:59.808689    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4769186-0b54-46b4-bb07-ecc3a5b92e94-maas-api-tls\") pod \"maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2\" (UID: \"e4769186-0b54-46b4-bb07-ecc3a5b92e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2"
Jul 29 19:58:59.808903 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:58:59.808806    2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-641e39-serving-cert: secret "maas-api-e2e-shared-a-641e39-serving-cert" not found
Jul 29 19:58:59.808903 ip-10-0-128-173 kubenswrapper[2557]: E0729 19:58:59.808879    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4769186-0b54-46b4-bb07-ecc3a5b92e94-maas-api-tls podName:e4769186-0b54-46b4-bb07-ecc3a5b92e94 nodeName:}" failed. No retries permitted until 2026-07-29 19:59:00.308858166 +0000 UTC m=+2511.136357222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e4769186-0b54-46b4-bb07-ecc3a5b92e94-maas-api-tls") pod "maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2" (UID: "e4769186-0b54-46b4-bb07-ecc3a5b92e94") : secret "maas-api-e2e-shared-a-641e39-serving-cert" not found
Jul 29 19:58:59.809415 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:59.809394    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4769186-0b54-46b4-bb07-ecc3a5b92e94-openshift-service-ca\") pod \"maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2\" (UID: \"e4769186-0b54-46b4-bb07-ecc3a5b92e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2"
Jul 29 19:58:59.817811 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:58:59.817786    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2299\" (UniqueName: \"kubernetes.io/projected/e4769186-0b54-46b4-bb07-ecc3a5b92e94-kube-api-access-v2299\") pod \"maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2\" (UID: \"e4769186-0b54-46b4-bb07-ecc3a5b92e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2"
Jul 29 19:59:00.313048 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:00.313015    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4769186-0b54-46b4-bb07-ecc3a5b92e94-maas-api-tls\") pod \"maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2\" (UID: \"e4769186-0b54-46b4-bb07-ecc3a5b92e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2"
Jul 29 19:59:00.315354 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:00.315322    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4769186-0b54-46b4-bb07-ecc3a5b92e94-maas-api-tls\") pod \"maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2\" (UID: \"e4769186-0b54-46b4-bb07-ecc3a5b92e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2"
Jul 29 19:59:00.332908 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:00.332879    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-641e39-787b6966b8-rwjd7"]
Jul 29 19:59:00.336530 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:00.336514    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-641e39-787b6966b8-rwjd7"
Jul 29 19:59:00.339127 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:00.339104    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-641e39\""
Jul 29 19:59:00.339262 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:00.339127    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-641e39-dockercfg-5hm6w\""
Jul 29 19:59:00.347429 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:00.347405    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-641e39-787b6966b8-rwjd7"]
Jul 29 19:59:00.357767 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:00.357740    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-641e39-5cb6cb867d-c4fdv"]
Jul 29 19:59:00.370184 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:00.370165    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-641e39-5cb6cb867d-c4fdv"
Jul 29 19:59:00.370660 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:00.370628    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-641e39-5cb6cb867d-c4fdv"]
Jul 29 19:59:00.414219 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:00.414187    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e3a032-66b8-4274-9aa5-3da2c46f6835-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-641e39-787b6966b8-rwjd7\" (UID: \"e2e3a032-66b8-4274-9aa5-3da2c46f6835\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-641e39-787b6966b8-rwjd7"
Jul 29 19:59:00.414342 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:00.414221    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgc6z\" (UniqueName: \"kubernetes.io/projected/a31200ff-316c-439b-821d-0a6f939ac40e-kube-api-access-vgc6z\") pod \"payload-processing-e2e-shared-a-641e39-5cb6cb867d-c4fdv\" (UID: \"a31200ff-316c-439b-821d-0a6f939ac40e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-641e39-5cb6cb867d-c4fdv"
Jul 29 19:59:00.414385 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:00.414353    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a31200ff-316c-439b-821d-0a6f939ac40e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-641e39-5cb6cb867d-c4fdv\" (UID: \"a31200ff-316c-439b-821d-0a6f939ac40e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-641e39-5cb6cb867d-c4fdv"
Jul 29 19:59:00.414423 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:00.414383    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnkqp\" (UniqueName: \"kubernetes.io/projected/e2e3a032-66b8-4274-9aa5-3da2c46f6835-kube-api-access-hnkqp\") pod \"payload-pre-processing-e2e-shared-a-641e39-787b6966b8-rwjd7\" (UID: \"e2e3a032-66b8-4274-9aa5-3da2c46f6835\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-641e39-787b6966b8-rwjd7"
Jul 29 19:59:00.515530 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:00.515490    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e3a032-66b8-4274-9aa5-3da2c46f6835-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-641e39-787b6966b8-rwjd7\" (UID: \"e2e3a032-66b8-4274-9aa5-3da2c46f6835\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-641e39-787b6966b8-rwjd7"
Jul 29 19:59:00.515530 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:00.515531    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgc6z\" (UniqueName: \"kubernetes.io/projected/a31200ff-316c-439b-821d-0a6f939ac40e-kube-api-access-vgc6z\") pod \"payload-processing-e2e-shared-a-641e39-5cb6cb867d-c4fdv\" (UID: \"a31200ff-316c-439b-821d-0a6f939ac40e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-641e39-5cb6cb867d-c4fdv"
Jul 29 19:59:00.515762 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:00.515648    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a31200ff-316c-439b-821d-0a6f939ac40e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-641e39-5cb6cb867d-c4fdv\" (UID: \"a31200ff-316c-439b-821d-0a6f939ac40e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-641e39-5cb6cb867d-c4fdv"
Jul 29 19:59:00.515762 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:00.515681    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnkqp\" (UniqueName: \"kubernetes.io/projected/e2e3a032-66b8-4274-9aa5-3da2c46f6835-kube-api-access-hnkqp\") pod \"payload-pre-processing-e2e-shared-a-641e39-787b6966b8-rwjd7\" (UID: \"e2e3a032-66b8-4274-9aa5-3da2c46f6835\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-641e39-787b6966b8-rwjd7"
Jul 29 19:59:00.516314 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:00.516289    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e3a032-66b8-4274-9aa5-3da2c46f6835-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-641e39-787b6966b8-rwjd7\" (UID: \"e2e3a032-66b8-4274-9aa5-3da2c46f6835\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-641e39-787b6966b8-rwjd7"
Jul 29 19:59:00.516428 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:00.516405    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a31200ff-316c-439b-821d-0a6f939ac40e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-641e39-5cb6cb867d-c4fdv\" (UID: \"a31200ff-316c-439b-821d-0a6f939ac40e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-641e39-5cb6cb867d-c4fdv"
Jul 29 19:59:00.526774 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:00.526746    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgc6z\" (UniqueName: \"kubernetes.io/projected/a31200ff-316c-439b-821d-0a6f939ac40e-kube-api-access-vgc6z\") pod \"payload-processing-e2e-shared-a-641e39-5cb6cb867d-c4fdv\" (UID: \"a31200ff-316c-439b-821d-0a6f939ac40e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-641e39-5cb6cb867d-c4fdv"
Jul 29 19:59:00.526884 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:00.526862    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnkqp\" (UniqueName: \"kubernetes.io/projected/e2e3a032-66b8-4274-9aa5-3da2c46f6835-kube-api-access-hnkqp\") pod \"payload-pre-processing-e2e-shared-a-641e39-787b6966b8-rwjd7\" (UID: \"e2e3a032-66b8-4274-9aa5-3da2c46f6835\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-641e39-787b6966b8-rwjd7"
Jul 29 19:59:00.539236 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:00.539216    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2"
Jul 29 19:59:00.647921 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:00.647892    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-641e39-787b6966b8-rwjd7"
Jul 29 19:59:00.671503 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:00.671480    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2"]
Jul 29 19:59:00.674318 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:59:00.674288    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4769186_0b54_46b4_bb07_ecc3a5b92e94.slice/crio-09d4817fe849c03a0cc53df18d136dcc9e998ad737f3249bd2a361580ea479a5 WatchSource:0}: Error finding container 09d4817fe849c03a0cc53df18d136dcc9e998ad737f3249bd2a361580ea479a5: Status 404 returned error can't find the container with id 09d4817fe849c03a0cc53df18d136dcc9e998ad737f3249bd2a361580ea479a5
Jul 29 19:59:00.681784 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:00.681766    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-641e39-5cb6cb867d-c4fdv"
Jul 29 19:59:00.805985 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:00.805509    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-641e39-787b6966b8-rwjd7"]
Jul 29 19:59:00.813502 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:59:00.813461    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e3a032_66b8_4274_9aa5_3da2c46f6835.slice/crio-9526c32df6e9c8d0fef2c09f2fccae067496df5cf1737160993507d60e652420 WatchSource:0}: Error finding container 9526c32df6e9c8d0fef2c09f2fccae067496df5cf1737160993507d60e652420: Status 404 returned error can't find the container with id 9526c32df6e9c8d0fef2c09f2fccae067496df5cf1737160993507d60e652420
Jul 29 19:59:00.858951 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:00.858929    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-641e39-5cb6cb867d-c4fdv"]
Jul 29 19:59:00.860930 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:59:00.860903    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda31200ff_316c_439b_821d_0a6f939ac40e.slice/crio-184dc1924a0e461dd2f3425d2db63bb98f8b127c0010e5c59a21ac64e6d56196 WatchSource:0}: Error finding container 184dc1924a0e461dd2f3425d2db63bb98f8b127c0010e5c59a21ac64e6d56196: Status 404 returned error can't find the container with id 184dc1924a0e461dd2f3425d2db63bb98f8b127c0010e5c59a21ac64e6d56196
Jul 29 19:59:01.291589 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:01.291498    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-641e39-5cb6cb867d-c4fdv" event={"ID":"a31200ff-316c-439b-821d-0a6f939ac40e","Type":"ContainerStarted","Data":"01b8ec7cf67373ed1e3c8ec39b473623ebe5118f1737bcbf74f78285eb6ed0f1"}
Jul 29 19:59:01.291589 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:01.291559    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-641e39-5cb6cb867d-c4fdv" event={"ID":"a31200ff-316c-439b-821d-0a6f939ac40e","Type":"ContainerStarted","Data":"184dc1924a0e461dd2f3425d2db63bb98f8b127c0010e5c59a21ac64e6d56196"}
Jul 29 19:59:01.291810 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:01.291674    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-641e39-5cb6cb867d-c4fdv"
Jul 29 19:59:01.293083 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:01.293061    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2" event={"ID":"e4769186-0b54-46b4-bb07-ecc3a5b92e94","Type":"ContainerStarted","Data":"03beaba7553bd230ae735346dcc46f4b84d083f7640929e9ff19b0ed34b04e52"}
Jul 29 19:59:01.293203 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:01.293096    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2" event={"ID":"e4769186-0b54-46b4-bb07-ecc3a5b92e94","Type":"ContainerStarted","Data":"09d4817fe849c03a0cc53df18d136dcc9e998ad737f3249bd2a361580ea479a5"}
Jul 29 19:59:01.293203 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:01.293145    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2"
Jul 29 19:59:01.294586 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:01.294561    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-641e39-787b6966b8-rwjd7" event={"ID":"e2e3a032-66b8-4274-9aa5-3da2c46f6835","Type":"ContainerStarted","Data":"b2f1a5cc278921296e20d794e813e70f4fab6f855e8b991a633ded26d1a209d5"}
Jul 29 19:59:01.294689 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:01.294596    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-641e39-787b6966b8-rwjd7" event={"ID":"e2e3a032-66b8-4274-9aa5-3da2c46f6835","Type":"ContainerStarted","Data":"9526c32df6e9c8d0fef2c09f2fccae067496df5cf1737160993507d60e652420"}
Jul 29 19:59:01.294689 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:01.294628    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-641e39-787b6966b8-rwjd7"
Jul 29 19:59:01.310475 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:01.310434    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-641e39-5cb6cb867d-c4fdv" podStartSLOduration=1.310420729 podStartE2EDuration="1.310420729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:59:01.309062153 +0000 UTC m=+2512.136561211" watchObservedRunningTime="2026-07-29 19:59:01.310420729 +0000 UTC m=+2512.137919764"
Jul 29 19:59:01.325334 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:01.325290    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-641e39-787b6966b8-rwjd7" podStartSLOduration=1.325280961 podStartE2EDuration="1.325280961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:59:01.324652024 +0000 UTC m=+2512.152151082" watchObservedRunningTime="2026-07-29 19:59:01.325280961 +0000 UTC m=+2512.152780019"
Jul 29 19:59:01.341357 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:01.341305    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2" podStartSLOduration=1.945590538 podStartE2EDuration="2.341287108s" totalImagesPullingTime="395.69657ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:58:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:59:01.340304355 +0000 UTC m=+2512.167803415" watchObservedRunningTime="2026-07-29 19:59:01.341287108 +0000 UTC m=+2512.168786171"
Jul 29 19:59:07.304323 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:07.304285    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2"
Jul 29 19:59:12.301070 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:12.301031    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-641e39-5cb6cb867d-c4fdv"
Jul 29 19:59:12.301496 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:12.301113    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-641e39-787b6966b8-rwjd7"
Jul 29 19:59:27.596171 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:27.596137    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l"]
Jul 29 19:59:27.600245 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:27.600227    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l"
Jul 29 19:59:27.602848 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:27.602822    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-4c787c-serving-cert\""
Jul 29 19:59:27.613421 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:27.613396    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l"]
Jul 29 19:59:27.655333 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:27.655299    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8-maas-api-tls\") pod \"maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l\" (UID: \"5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l"
Jul 29 19:59:27.655495 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:27.655432    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qphr2\" (UniqueName: \"kubernetes.io/projected/5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8-kube-api-access-qphr2\") pod \"maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l\" (UID: \"5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l"
Jul 29 19:59:27.655495 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:27.655484    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l\" (UID: \"5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l"
Jul 29 19:59:27.756082 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:27.756049    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8-maas-api-tls\") pod \"maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l\" (UID: \"5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l"
Jul 29 19:59:27.756218 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:27.756131    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qphr2\" (UniqueName: \"kubernetes.io/projected/5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8-kube-api-access-qphr2\") pod \"maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l\" (UID: \"5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l"
Jul 29 19:59:27.756218 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:27.756167    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l\" (UID: \"5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l"
Jul 29 19:59:27.756766 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:27.756737    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l\" (UID: \"5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l"
Jul 29 19:59:27.758454 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:27.758425    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8-maas-api-tls\") pod \"maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l\" (UID: \"5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l"
Jul 29 19:59:27.771502 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:27.771474    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qphr2\" (UniqueName: \"kubernetes.io/projected/5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8-kube-api-access-qphr2\") pod \"maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l\" (UID: \"5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l"
Jul 29 19:59:27.914315 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:27.914285    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l"
Jul 29 19:59:28.049133 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:28.049103    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l"]
Jul 29 19:59:28.051675 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:59:28.051651    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f3fe360_665e_4e28_a8f0_a29b7d0cc1f8.slice/crio-73b25d8299a07e84f64d6c6112c835adf2fe3de6c9ed7b22543b6cc07f5a5269 WatchSource:0}: Error finding container 73b25d8299a07e84f64d6c6112c835adf2fe3de6c9ed7b22543b6cc07f5a5269: Status 404 returned error can't find the container with id 73b25d8299a07e84f64d6c6112c835adf2fe3de6c9ed7b22543b6cc07f5a5269
Jul 29 19:59:28.295636 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:28.295560    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-4c787c-8f587d5f6-n64xz"]
Jul 29 19:59:28.301188 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:28.301157    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4c787c-8f587d5f6-n64xz"
Jul 29 19:59:28.305142 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:28.305114    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-4c787c\""
Jul 29 19:59:28.313710 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:28.313683    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-4c787c-8f587d5f6-n64xz"]
Jul 29 19:59:28.322343 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:28.322309    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-4c787c-5489f58b58-hddhc"]
Jul 29 19:59:28.325783 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:28.325764    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-4c787c-5489f58b58-hddhc"
Jul 29 19:59:28.331293 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:28.331273    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-4c787c-dockercfg-6ld5j\""
Jul 29 19:59:28.339988 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:28.339950    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-4c787c-5489f58b58-hddhc"]
Jul 29 19:59:28.359886 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:28.359850    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74179647-d44f-48c3-b934-ba7a2226bc0a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-4c787c-5489f58b58-hddhc\" (UID: \"74179647-d44f-48c3-b934-ba7a2226bc0a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-4c787c-5489f58b58-hddhc"
Jul 29 19:59:28.359886 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:28.359880    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rsjp\" (UniqueName: \"kubernetes.io/projected/15466f4c-344d-48b3-9487-dfd21dab978e-kube-api-access-5rsjp\") pod \"payload-pre-processing-e2e-shared-b-4c787c-8f587d5f6-n64xz\" (UID: \"15466f4c-344d-48b3-9487-dfd21dab978e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4c787c-8f587d5f6-n64xz"
Jul 29 19:59:28.360109 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:28.359912    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kscw6\" (UniqueName: \"kubernetes.io/projected/74179647-d44f-48c3-b934-ba7a2226bc0a-kube-api-access-kscw6\") pod \"payload-processing-e2e-shared-b-4c787c-5489f58b58-hddhc\" (UID: \"74179647-d44f-48c3-b934-ba7a2226bc0a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-4c787c-5489f58b58-hddhc"
Jul 29 19:59:28.360109 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:28.360028    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15466f4c-344d-48b3-9487-dfd21dab978e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-4c787c-8f587d5f6-n64xz\" (UID: \"15466f4c-344d-48b3-9487-dfd21dab978e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4c787c-8f587d5f6-n64xz"
Jul 29 19:59:28.425192 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:28.425162    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l" event={"ID":"5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8","Type":"ContainerStarted","Data":"73b25d8299a07e84f64d6c6112c835adf2fe3de6c9ed7b22543b6cc07f5a5269"}
Jul 29 19:59:28.460635 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:28.460607    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15466f4c-344d-48b3-9487-dfd21dab978e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-4c787c-8f587d5f6-n64xz\" (UID: \"15466f4c-344d-48b3-9487-dfd21dab978e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4c787c-8f587d5f6-n64xz"
Jul 29 19:59:28.460771 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:28.460670    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74179647-d44f-48c3-b934-ba7a2226bc0a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-4c787c-5489f58b58-hddhc\" (UID: \"74179647-d44f-48c3-b934-ba7a2226bc0a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-4c787c-5489f58b58-hddhc"
Jul 29 19:59:28.460771 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:28.460702    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rsjp\" (UniqueName: \"kubernetes.io/projected/15466f4c-344d-48b3-9487-dfd21dab978e-kube-api-access-5rsjp\") pod \"payload-pre-processing-e2e-shared-b-4c787c-8f587d5f6-n64xz\" (UID: \"15466f4c-344d-48b3-9487-dfd21dab978e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4c787c-8f587d5f6-n64xz"
Jul 29 19:59:28.460771 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:28.460756    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kscw6\" (UniqueName: \"kubernetes.io/projected/74179647-d44f-48c3-b934-ba7a2226bc0a-kube-api-access-kscw6\") pod \"payload-processing-e2e-shared-b-4c787c-5489f58b58-hddhc\" (UID: \"74179647-d44f-48c3-b934-ba7a2226bc0a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-4c787c-5489f58b58-hddhc"
Jul 29 19:59:28.461405 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:28.461384    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15466f4c-344d-48b3-9487-dfd21dab978e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-4c787c-8f587d5f6-n64xz\" (UID: \"15466f4c-344d-48b3-9487-dfd21dab978e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4c787c-8f587d5f6-n64xz"
Jul 29 19:59:28.461465 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:28.461415    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74179647-d44f-48c3-b934-ba7a2226bc0a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-4c787c-5489f58b58-hddhc\" (UID: \"74179647-d44f-48c3-b934-ba7a2226bc0a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-4c787c-5489f58b58-hddhc"
Jul 29 19:59:28.469363 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:28.469344    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rsjp\" (UniqueName: \"kubernetes.io/projected/15466f4c-344d-48b3-9487-dfd21dab978e-kube-api-access-5rsjp\") pod \"payload-pre-processing-e2e-shared-b-4c787c-8f587d5f6-n64xz\" (UID: \"15466f4c-344d-48b3-9487-dfd21dab978e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4c787c-8f587d5f6-n64xz"
Jul 29 19:59:28.469464 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:28.469368    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kscw6\" (UniqueName: \"kubernetes.io/projected/74179647-d44f-48c3-b934-ba7a2226bc0a-kube-api-access-kscw6\") pod \"payload-processing-e2e-shared-b-4c787c-5489f58b58-hddhc\" (UID: \"74179647-d44f-48c3-b934-ba7a2226bc0a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-4c787c-5489f58b58-hddhc"
Jul 29 19:59:28.614621 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:28.614530    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4c787c-8f587d5f6-n64xz"
Jul 29 19:59:28.635601 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:28.635564    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-4c787c-5489f58b58-hddhc"
Jul 29 19:59:28.776529 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:28.776427    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-4c787c-8f587d5f6-n64xz"]
Jul 29 19:59:28.808080 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:28.808051    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-4c787c-5489f58b58-hddhc"]
Jul 29 19:59:28.819354 ip-10-0-128-173 kubenswrapper[2557]: W0729 19:59:28.819323    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74179647_d44f_48c3_b934_ba7a2226bc0a.slice/crio-ffd7243bb21e5f9a734d119c744e7c16d49db5217a8df7aab870e35d40b048b6 WatchSource:0}: Error finding container ffd7243bb21e5f9a734d119c744e7c16d49db5217a8df7aab870e35d40b048b6: Status 404 returned error can't find the container with id ffd7243bb21e5f9a734d119c744e7c16d49db5217a8df7aab870e35d40b048b6
Jul 29 19:59:29.431277 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:29.431232    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4c787c-8f587d5f6-n64xz" event={"ID":"15466f4c-344d-48b3-9487-dfd21dab978e","Type":"ContainerStarted","Data":"a576e044a71312db3356a661cd3818e3058aa1e3677c49455202702437b9b1c5"}
Jul 29 19:59:29.431463 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:29.431307    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4c787c-8f587d5f6-n64xz" event={"ID":"15466f4c-344d-48b3-9487-dfd21dab978e","Type":"ContainerStarted","Data":"6a460c67b31432e4b1c30859a2ddc0cf9073880cd610fee8cf11412adac1dd12"}
Jul 29 19:59:29.431463 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:29.431337    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4c787c-8f587d5f6-n64xz"
Jul 29 19:59:29.432944 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:29.432917    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-4c787c-5489f58b58-hddhc" event={"ID":"74179647-d44f-48c3-b934-ba7a2226bc0a","Type":"ContainerStarted","Data":"8a6edd4524fa0c1bb01597ee541e1092ab9f6c9c396b43819d9912e9ffc21b21"}
Jul 29 19:59:29.433119 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:29.432978    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-4c787c-5489f58b58-hddhc" event={"ID":"74179647-d44f-48c3-b934-ba7a2226bc0a","Type":"ContainerStarted","Data":"ffd7243bb21e5f9a734d119c744e7c16d49db5217a8df7aab870e35d40b048b6"}
Jul 29 19:59:29.433237 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:29.433213    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-4c787c-5489f58b58-hddhc"
Jul 29 19:59:29.434601 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:29.434577    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l" event={"ID":"5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8","Type":"ContainerStarted","Data":"d7175b952a2dff5e7c3b77201f79f34f17ba6974925fc0042850934ba2551b2f"}
Jul 29 19:59:29.434704 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:29.434632    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l"
Jul 29 19:59:29.454546 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:29.454502    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4c787c-8f587d5f6-n64xz" podStartSLOduration=1.454490005 podStartE2EDuration="1.454490005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:59:29.45361939 +0000 UTC m=+2540.281118447" watchObservedRunningTime="2026-07-29 19:59:29.454490005 +0000 UTC m=+2540.281989041"
Jul 29 19:59:29.481803 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:29.481757    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-4c787c-5489f58b58-hddhc" podStartSLOduration=1.481743117 podStartE2EDuration="1.481743117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:59:29.48034858 +0000 UTC m=+2540.307847638" watchObservedRunningTime="2026-07-29 19:59:29.481743117 +0000 UTC m=+2540.309242174"
Jul 29 19:59:29.503517 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:29.503465    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l" podStartSLOduration=2.068751545 podStartE2EDuration="2.503450662s" totalImagesPullingTime="434.699117ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:59:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:59:29.50227555 +0000 UTC m=+2540.329774612" watchObservedRunningTime="2026-07-29 19:59:29.503450662 +0000 UTC m=+2540.330949723"
Jul 29 19:59:35.448183 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:35.448153    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l"
Jul 29 19:59:40.442815 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:40.442786    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4c787c-8f587d5f6-n64xz"
Jul 29 19:59:41.447145 ip-10-0-128-173 kubenswrapper[2557]: I0729 19:59:41.447114    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-4c787c-5489f58b58-hddhc"
Jul 29 20:00:00.038595 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:00.038562    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs"]
Jul 29 20:00:00.042322 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:00.042303    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs"
Jul 29 20:00:00.046800 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:00.046771    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-641e39\"/\"auth-test-model-641e39-kserve-self-signed-certs\""
Jul 29 20:00:00.046917 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:00.046806    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-641e39\"/\"openshift-service-ca.crt\""
Jul 29 20:00:00.046917 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:00.046815    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-641e39\"/\"default-dockercfg-mgmzf\""
Jul 29 20:00:00.046917 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:00.046777    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-641e39\"/\"kube-root-ca.crt\""
Jul 29 20:00:00.050819 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:00.050797    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs"]
Jul 29 20:00:00.147391 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:00.147351    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de2179da-9af4-4c9f-869a-bb2b13f180ac-tls-certs\") pod \"auth-test-model-641e39-kserve-6f76bc44b7-dbpfs\" (UID: \"de2179da-9af4-4c9f-869a-bb2b13f180ac\") " pod="ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs"
Jul 29 20:00:00.147575 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:00.147434    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlf7f\" (UniqueName: \"kubernetes.io/projected/de2179da-9af4-4c9f-869a-bb2b13f180ac-kube-api-access-mlf7f\") pod \"auth-test-model-641e39-kserve-6f76bc44b7-dbpfs\" (UID: \"de2179da-9af4-4c9f-869a-bb2b13f180ac\") " pod="ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs"
Jul 29 20:00:00.147575 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:00.147457    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de2179da-9af4-4c9f-869a-bb2b13f180ac-dshm\") pod \"auth-test-model-641e39-kserve-6f76bc44b7-dbpfs\" (UID: \"de2179da-9af4-4c9f-869a-bb2b13f180ac\") " pod="ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs"
Jul 29 20:00:00.147575 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:00.147482    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de2179da-9af4-4c9f-869a-bb2b13f180ac-model-cache\") pod \"auth-test-model-641e39-kserve-6f76bc44b7-dbpfs\" (UID: \"de2179da-9af4-4c9f-869a-bb2b13f180ac\") " pod="ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs"
Jul 29 20:00:00.147575 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:00.147503    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de2179da-9af4-4c9f-869a-bb2b13f180ac-home\") pod \"auth-test-model-641e39-kserve-6f76bc44b7-dbpfs\" (UID: \"de2179da-9af4-4c9f-869a-bb2b13f180ac\") " pod="ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs"
Jul 29 20:00:00.147575 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:00.147519    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de2179da-9af4-4c9f-869a-bb2b13f180ac-tmp-dir\") pod \"auth-test-model-641e39-kserve-6f76bc44b7-dbpfs\" (UID: \"de2179da-9af4-4c9f-869a-bb2b13f180ac\") " pod="ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs"
Jul 29 20:00:00.248032 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:00.247995    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de2179da-9af4-4c9f-869a-bb2b13f180ac-tls-certs\") pod \"auth-test-model-641e39-kserve-6f76bc44b7-dbpfs\" (UID: \"de2179da-9af4-4c9f-869a-bb2b13f180ac\") " pod="ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs"
Jul 29 20:00:00.248214 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:00.248095    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlf7f\" (UniqueName: \"kubernetes.io/projected/de2179da-9af4-4c9f-869a-bb2b13f180ac-kube-api-access-mlf7f\") pod \"auth-test-model-641e39-kserve-6f76bc44b7-dbpfs\" (UID: \"de2179da-9af4-4c9f-869a-bb2b13f180ac\") " pod="ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs"
Jul 29 20:00:00.248214 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:00.248124    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de2179da-9af4-4c9f-869a-bb2b13f180ac-dshm\") pod \"auth-test-model-641e39-kserve-6f76bc44b7-dbpfs\" (UID: \"de2179da-9af4-4c9f-869a-bb2b13f180ac\") " pod="ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs"
Jul 29 20:00:00.248214 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:00.248170    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de2179da-9af4-4c9f-869a-bb2b13f180ac-model-cache\") pod \"auth-test-model-641e39-kserve-6f76bc44b7-dbpfs\" (UID: \"de2179da-9af4-4c9f-869a-bb2b13f180ac\") " pod="ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs"
Jul 29 20:00:00.248214 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:00.248199    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de2179da-9af4-4c9f-869a-bb2b13f180ac-home\") pod \"auth-test-model-641e39-kserve-6f76bc44b7-dbpfs\" (UID: \"de2179da-9af4-4c9f-869a-bb2b13f180ac\") " pod="ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs"
Jul 29 20:00:00.248445 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:00.248215    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de2179da-9af4-4c9f-869a-bb2b13f180ac-tmp-dir\") pod \"auth-test-model-641e39-kserve-6f76bc44b7-dbpfs\" (UID: \"de2179da-9af4-4c9f-869a-bb2b13f180ac\") " pod="ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs"
Jul 29 20:00:00.248681 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:00.248630    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de2179da-9af4-4c9f-869a-bb2b13f180ac-model-cache\") pod \"auth-test-model-641e39-kserve-6f76bc44b7-dbpfs\" (UID: \"de2179da-9af4-4c9f-869a-bb2b13f180ac\") " pod="ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs"
Jul 29 20:00:00.248681 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:00.248644    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de2179da-9af4-4c9f-869a-bb2b13f180ac-tmp-dir\") pod \"auth-test-model-641e39-kserve-6f76bc44b7-dbpfs\" (UID: \"de2179da-9af4-4c9f-869a-bb2b13f180ac\") " pod="ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs"
Jul 29 20:00:00.248876 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:00.248709    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de2179da-9af4-4c9f-869a-bb2b13f180ac-home\") pod \"auth-test-model-641e39-kserve-6f76bc44b7-dbpfs\" (UID: \"de2179da-9af4-4c9f-869a-bb2b13f180ac\") " pod="ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs"
Jul 29 20:00:00.250639 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:00.250615    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de2179da-9af4-4c9f-869a-bb2b13f180ac-dshm\") pod \"auth-test-model-641e39-kserve-6f76bc44b7-dbpfs\" (UID: \"de2179da-9af4-4c9f-869a-bb2b13f180ac\") " pod="ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs"
Jul 29 20:00:00.250795 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:00.250778    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de2179da-9af4-4c9f-869a-bb2b13f180ac-tls-certs\") pod \"auth-test-model-641e39-kserve-6f76bc44b7-dbpfs\" (UID: \"de2179da-9af4-4c9f-869a-bb2b13f180ac\") " pod="ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs"
Jul 29 20:00:00.257124 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:00.257101    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlf7f\" (UniqueName: \"kubernetes.io/projected/de2179da-9af4-4c9f-869a-bb2b13f180ac-kube-api-access-mlf7f\") pod \"auth-test-model-641e39-kserve-6f76bc44b7-dbpfs\" (UID: \"de2179da-9af4-4c9f-869a-bb2b13f180ac\") " pod="ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs"
Jul 29 20:00:00.353045 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:00.352951    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs"
Jul 29 20:00:00.503307 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:00.503198    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs"]
Jul 29 20:00:00.505610 ip-10-0-128-173 kubenswrapper[2557]: W0729 20:00:00.505581    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde2179da_9af4_4c9f_869a_bb2b13f180ac.slice/crio-f62ca8023b482882cebde0e597d8fb9ed4f01c5316a24a4e14435fd54136c3de WatchSource:0}: Error finding container f62ca8023b482882cebde0e597d8fb9ed4f01c5316a24a4e14435fd54136c3de: Status 404 returned error can't find the container with id f62ca8023b482882cebde0e597d8fb9ed4f01c5316a24a4e14435fd54136c3de
Jul 29 20:00:00.587257 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:00.587218    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs" event={"ID":"de2179da-9af4-4c9f-869a-bb2b13f180ac","Type":"ContainerStarted","Data":"bd7df4d5eef8a3e2c237e081fb51fe0bd001809e67d98d4900f102f5c3fd3aa8"}
Jul 29 20:00:00.587257 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:00.587260    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs" event={"ID":"de2179da-9af4-4c9f-869a-bb2b13f180ac","Type":"ContainerStarted","Data":"f62ca8023b482882cebde0e597d8fb9ed4f01c5316a24a4e14435fd54136c3de"}
Jul 29 20:00:10.353694 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:10.353657    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs"
Jul 29 20:00:10.354250 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:10.353730    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs"
Jul 29 20:00:10.367176 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:10.367145    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs"
Jul 29 20:00:10.386649 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:10.386599    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs" podStartSLOduration=10.386584006 podStartE2EDuration="10.386584006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:00:00.630628805 +0000 UTC m=+2571.458127862" watchObservedRunningTime="2026-07-29 20:00:10.386584006 +0000 UTC m=+2581.214083063"
Jul 29 20:00:10.654284 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:10.654253    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs"
Jul 29 20:00:30.643165 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:30.643132    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8"]
Jul 29 20:00:30.647017 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:30.646996    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8"
Jul 29 20:00:30.649938 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:30.649916    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4c787c\"/\"default-dockercfg-lfr2l\""
Jul 29 20:00:30.651089 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:30.651067    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4c787c\"/\"openshift-service-ca.crt\""
Jul 29 20:00:30.651375 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:30.651356    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4c787c\"/\"kube-root-ca.crt\""
Jul 29 20:00:30.651487 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:30.651459    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4c787c\"/\"auth-test-model-4c787c-kserve-self-signed-certs\""
Jul 29 20:00:30.657359 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:30.657334    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8"]
Jul 29 20:00:30.724821 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:30.724791    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb07aea4-c399-4689-9136-50c0e80397f5-dshm\") pod \"auth-test-model-4c787c-kserve-5fbf766b99-sbvt8\" (UID: \"eb07aea4-c399-4689-9136-50c0e80397f5\") " pod="ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8"
Jul 29 20:00:30.724821 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:30.724822    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb07aea4-c399-4689-9136-50c0e80397f5-tmp-dir\") pod \"auth-test-model-4c787c-kserve-5fbf766b99-sbvt8\" (UID: \"eb07aea4-c399-4689-9136-50c0e80397f5\") " pod="ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8"
Jul 29 20:00:30.725137 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:30.724839    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb07aea4-c399-4689-9136-50c0e80397f5-home\") pod \"auth-test-model-4c787c-kserve-5fbf766b99-sbvt8\" (UID: \"eb07aea4-c399-4689-9136-50c0e80397f5\") " pod="ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8"
Jul 29 20:00:30.725137 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:30.724856    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb07aea4-c399-4689-9136-50c0e80397f5-model-cache\") pod \"auth-test-model-4c787c-kserve-5fbf766b99-sbvt8\" (UID: \"eb07aea4-c399-4689-9136-50c0e80397f5\") " pod="ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8"
Jul 29 20:00:30.725137 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:30.725052    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jdj4\" (UniqueName: \"kubernetes.io/projected/eb07aea4-c399-4689-9136-50c0e80397f5-kube-api-access-4jdj4\") pod \"auth-test-model-4c787c-kserve-5fbf766b99-sbvt8\" (UID: \"eb07aea4-c399-4689-9136-50c0e80397f5\") " pod="ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8"
Jul 29 20:00:30.725137 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:30.725107    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb07aea4-c399-4689-9136-50c0e80397f5-tls-certs\") pod \"auth-test-model-4c787c-kserve-5fbf766b99-sbvt8\" (UID: \"eb07aea4-c399-4689-9136-50c0e80397f5\") " pod="ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8"
Jul 29 20:00:30.826137 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:30.826096    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb07aea4-c399-4689-9136-50c0e80397f5-dshm\") pod \"auth-test-model-4c787c-kserve-5fbf766b99-sbvt8\" (UID: \"eb07aea4-c399-4689-9136-50c0e80397f5\") " pod="ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8"
Jul 29 20:00:30.826137 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:30.826136    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb07aea4-c399-4689-9136-50c0e80397f5-tmp-dir\") pod \"auth-test-model-4c787c-kserve-5fbf766b99-sbvt8\" (UID: \"eb07aea4-c399-4689-9136-50c0e80397f5\") " pod="ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8"
Jul 29 20:00:30.826363 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:30.826165    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb07aea4-c399-4689-9136-50c0e80397f5-home\") pod \"auth-test-model-4c787c-kserve-5fbf766b99-sbvt8\" (UID: \"eb07aea4-c399-4689-9136-50c0e80397f5\") " pod="ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8"
Jul 29 20:00:30.826363 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:30.826187    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb07aea4-c399-4689-9136-50c0e80397f5-model-cache\") pod \"auth-test-model-4c787c-kserve-5fbf766b99-sbvt8\" (UID: \"eb07aea4-c399-4689-9136-50c0e80397f5\") " pod="ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8"
Jul 29 20:00:30.826363 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:30.826256    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jdj4\" (UniqueName: \"kubernetes.io/projected/eb07aea4-c399-4689-9136-50c0e80397f5-kube-api-access-4jdj4\") pod \"auth-test-model-4c787c-kserve-5fbf766b99-sbvt8\" (UID: \"eb07aea4-c399-4689-9136-50c0e80397f5\") " pod="ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8"
Jul 29 20:00:30.826363 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:30.826292    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb07aea4-c399-4689-9136-50c0e80397f5-tls-certs\") pod \"auth-test-model-4c787c-kserve-5fbf766b99-sbvt8\" (UID: \"eb07aea4-c399-4689-9136-50c0e80397f5\") " pod="ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8"
Jul 29 20:00:30.826592 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:30.826565    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb07aea4-c399-4689-9136-50c0e80397f5-tmp-dir\") pod \"auth-test-model-4c787c-kserve-5fbf766b99-sbvt8\" (UID: \"eb07aea4-c399-4689-9136-50c0e80397f5\") " pod="ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8"
Jul 29 20:00:30.826670 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:30.826608    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb07aea4-c399-4689-9136-50c0e80397f5-home\") pod \"auth-test-model-4c787c-kserve-5fbf766b99-sbvt8\" (UID: \"eb07aea4-c399-4689-9136-50c0e80397f5\") " pod="ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8"
Jul 29 20:00:30.826670 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:30.826619    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb07aea4-c399-4689-9136-50c0e80397f5-model-cache\") pod \"auth-test-model-4c787c-kserve-5fbf766b99-sbvt8\" (UID: \"eb07aea4-c399-4689-9136-50c0e80397f5\") " pod="ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8"
Jul 29 20:00:30.828596 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:30.828570    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb07aea4-c399-4689-9136-50c0e80397f5-dshm\") pod \"auth-test-model-4c787c-kserve-5fbf766b99-sbvt8\" (UID: \"eb07aea4-c399-4689-9136-50c0e80397f5\") " pod="ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8"
Jul 29 20:00:30.828765 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:30.828749    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb07aea4-c399-4689-9136-50c0e80397f5-tls-certs\") pod \"auth-test-model-4c787c-kserve-5fbf766b99-sbvt8\" (UID: \"eb07aea4-c399-4689-9136-50c0e80397f5\") " pod="ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8"
Jul 29 20:00:30.834646 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:30.834624    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jdj4\" (UniqueName: \"kubernetes.io/projected/eb07aea4-c399-4689-9136-50c0e80397f5-kube-api-access-4jdj4\") pod \"auth-test-model-4c787c-kserve-5fbf766b99-sbvt8\" (UID: \"eb07aea4-c399-4689-9136-50c0e80397f5\") " pod="ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8"
Jul 29 20:00:30.957833 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:30.957745    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8"
Jul 29 20:00:31.090212 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:31.090186    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8"]
Jul 29 20:00:31.092152 ip-10-0-128-173 kubenswrapper[2557]: W0729 20:00:31.092115    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb07aea4_c399_4689_9136_50c0e80397f5.slice/crio-5b6faecb10e85c7de10400f6422000b7c81985ca5fd561575ea386672b70aa77 WatchSource:0}: Error finding container 5b6faecb10e85c7de10400f6422000b7c81985ca5fd561575ea386672b70aa77: Status 404 returned error can't find the container with id 5b6faecb10e85c7de10400f6422000b7c81985ca5fd561575ea386672b70aa77
Jul 29 20:00:31.741219 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:31.741181    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8" event={"ID":"eb07aea4-c399-4689-9136-50c0e80397f5","Type":"ContainerStarted","Data":"abe21a45f90c21aece2d6c59d8c4f010eb5fef1183086377d23b7fb48b47c4cf"}
Jul 29 20:00:31.741219 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:31.741225    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8" event={"ID":"eb07aea4-c399-4689-9136-50c0e80397f5","Type":"ContainerStarted","Data":"5b6faecb10e85c7de10400f6422000b7c81985ca5fd561575ea386672b70aa77"}
Jul 29 20:00:31.762991 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:31.762931    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8" podStartSLOduration=1.7629175799999999 podStartE2EDuration="1.76291758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:00:31.761097061 +0000 UTC m=+2602.588596153" watchObservedRunningTime="2026-07-29 20:00:31.76291758 +0000 UTC m=+2602.590416638"
Jul 29 20:00:40.958062 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:40.958023    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8"
Jul 29 20:00:40.958467 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:40.958118    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8"
Jul 29 20:00:40.970937 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:40.970913    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8"
Jul 29 20:00:41.801489 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:00:41.801462    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8"
Jul 29 20:03:12.492486 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:03:12.492450    2557 scope.go:117] "RemoveContainer" containerID="941b8f2dc6cebf27ed99e6d4e628520d7492f9bd23d639ae8509359f85013780"
Jul 29 20:04:12.513530 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:04:12.513495    2557 scope.go:117] "RemoveContainer" containerID="1c167c2e003e6435259cfc848354d4f6be97d1e824e0718d7f67a3aa23f474ad"
Jul 29 20:04:47.420979 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:04:47.420928    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs"]
Jul 29 20:04:48.216024 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:04:48.215988    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8"]
Jul 29 20:05:02.439841 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:02.439728    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs" podUID="de2179da-9af4-4c9f-869a-bb2b13f180ac" containerName="main" containerID="cri-o://bd7df4d5eef8a3e2c237e081fb51fe0bd001809e67d98d4900f102f5c3fd3aa8" gracePeriod=45
Jul 29 20:05:02.687480 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:02.687454    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs"
Jul 29 20:05:02.852939 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:02.852845    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlf7f\" (UniqueName: \"kubernetes.io/projected/de2179da-9af4-4c9f-869a-bb2b13f180ac-kube-api-access-mlf7f\") pod \"de2179da-9af4-4c9f-869a-bb2b13f180ac\" (UID: \"de2179da-9af4-4c9f-869a-bb2b13f180ac\") "
Jul 29 20:05:02.852939 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:02.852892    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de2179da-9af4-4c9f-869a-bb2b13f180ac-tmp-dir\") pod \"de2179da-9af4-4c9f-869a-bb2b13f180ac\" (UID: \"de2179da-9af4-4c9f-869a-bb2b13f180ac\") "
Jul 29 20:05:02.853204 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:02.852988    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de2179da-9af4-4c9f-869a-bb2b13f180ac-home\") pod \"de2179da-9af4-4c9f-869a-bb2b13f180ac\" (UID: \"de2179da-9af4-4c9f-869a-bb2b13f180ac\") "
Jul 29 20:05:02.853204 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:02.853020    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de2179da-9af4-4c9f-869a-bb2b13f180ac-model-cache\") pod \"de2179da-9af4-4c9f-869a-bb2b13f180ac\" (UID: \"de2179da-9af4-4c9f-869a-bb2b13f180ac\") "
Jul 29 20:05:02.853204 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:02.853067    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de2179da-9af4-4c9f-869a-bb2b13f180ac-dshm\") pod \"de2179da-9af4-4c9f-869a-bb2b13f180ac\" (UID: \"de2179da-9af4-4c9f-869a-bb2b13f180ac\") "
Jul 29 20:05:02.853204 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:02.853084    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de2179da-9af4-4c9f-869a-bb2b13f180ac-tls-certs\") pod \"de2179da-9af4-4c9f-869a-bb2b13f180ac\" (UID: \"de2179da-9af4-4c9f-869a-bb2b13f180ac\") "
Jul 29 20:05:02.853204 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:02.853193    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de2179da-9af4-4c9f-869a-bb2b13f180ac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "de2179da-9af4-4c9f-869a-bb2b13f180ac" (UID: "de2179da-9af4-4c9f-869a-bb2b13f180ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:05:02.853457 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:02.853229    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de2179da-9af4-4c9f-869a-bb2b13f180ac-home" (OuterVolumeSpecName: "home") pod "de2179da-9af4-4c9f-869a-bb2b13f180ac" (UID: "de2179da-9af4-4c9f-869a-bb2b13f180ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:05:02.853457 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:02.853290    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de2179da-9af4-4c9f-869a-bb2b13f180ac-model-cache" (OuterVolumeSpecName: "model-cache") pod "de2179da-9af4-4c9f-869a-bb2b13f180ac" (UID: "de2179da-9af4-4c9f-869a-bb2b13f180ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:05:02.853566 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:02.853503    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de2179da-9af4-4c9f-869a-bb2b13f180ac-tmp-dir\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:05:02.853566 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:02.853525    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de2179da-9af4-4c9f-869a-bb2b13f180ac-home\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:05:02.853566 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:02.853533    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de2179da-9af4-4c9f-869a-bb2b13f180ac-model-cache\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:05:02.855213 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:02.855193    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de2179da-9af4-4c9f-869a-bb2b13f180ac-dshm" (OuterVolumeSpecName: "dshm") pod "de2179da-9af4-4c9f-869a-bb2b13f180ac" (UID: "de2179da-9af4-4c9f-869a-bb2b13f180ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:05:02.855285 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:02.855257    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de2179da-9af4-4c9f-869a-bb2b13f180ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "de2179da-9af4-4c9f-869a-bb2b13f180ac" (UID: "de2179da-9af4-4c9f-869a-bb2b13f180ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:05:02.855334 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:02.855318    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de2179da-9af4-4c9f-869a-bb2b13f180ac-kube-api-access-mlf7f" (OuterVolumeSpecName: "kube-api-access-mlf7f") pod "de2179da-9af4-4c9f-869a-bb2b13f180ac" (UID: "de2179da-9af4-4c9f-869a-bb2b13f180ac"). InnerVolumeSpecName "kube-api-access-mlf7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:05:02.954972 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:02.954939    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de2179da-9af4-4c9f-869a-bb2b13f180ac-dshm\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:05:02.955141 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:02.954988    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de2179da-9af4-4c9f-869a-bb2b13f180ac-tls-certs\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:05:02.955141 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:02.954999    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlf7f\" (UniqueName: \"kubernetes.io/projected/de2179da-9af4-4c9f-869a-bb2b13f180ac-kube-api-access-mlf7f\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:05:03.078190 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:03.078153    2557 generic.go:356] "Generic (PLEG): container finished" podID="de2179da-9af4-4c9f-869a-bb2b13f180ac" containerID="bd7df4d5eef8a3e2c237e081fb51fe0bd001809e67d98d4900f102f5c3fd3aa8" exitCode=0
Jul 29 20:05:03.078190 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:03.078193    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs"
Jul 29 20:05:03.078498 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:03.078232    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs" event={"ID":"de2179da-9af4-4c9f-869a-bb2b13f180ac","Type":"ContainerDied","Data":"bd7df4d5eef8a3e2c237e081fb51fe0bd001809e67d98d4900f102f5c3fd3aa8"}
Jul 29 20:05:03.078498 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:03.078275    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs" event={"ID":"de2179da-9af4-4c9f-869a-bb2b13f180ac","Type":"ContainerDied","Data":"f62ca8023b482882cebde0e597d8fb9ed4f01c5316a24a4e14435fd54136c3de"}
Jul 29 20:05:03.078498 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:03.078289    2557 scope.go:117] "RemoveContainer" containerID="bd7df4d5eef8a3e2c237e081fb51fe0bd001809e67d98d4900f102f5c3fd3aa8"
Jul 29 20:05:03.087448 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:03.087434    2557 scope.go:117] "RemoveContainer" containerID="bd7df4d5eef8a3e2c237e081fb51fe0bd001809e67d98d4900f102f5c3fd3aa8"
Jul 29 20:05:03.087683 ip-10-0-128-173 kubenswrapper[2557]: E0729 20:05:03.087666    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd7df4d5eef8a3e2c237e081fb51fe0bd001809e67d98d4900f102f5c3fd3aa8\": container with ID starting with bd7df4d5eef8a3e2c237e081fb51fe0bd001809e67d98d4900f102f5c3fd3aa8 not found: ID does not exist" containerID="bd7df4d5eef8a3e2c237e081fb51fe0bd001809e67d98d4900f102f5c3fd3aa8"
Jul 29 20:05:03.087728 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:03.087690    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd7df4d5eef8a3e2c237e081fb51fe0bd001809e67d98d4900f102f5c3fd3aa8"} err="failed to get container status \"bd7df4d5eef8a3e2c237e081fb51fe0bd001809e67d98d4900f102f5c3fd3aa8\": rpc error: code = NotFound desc = could not find container \"bd7df4d5eef8a3e2c237e081fb51fe0bd001809e67d98d4900f102f5c3fd3aa8\": container with ID starting with bd7df4d5eef8a3e2c237e081fb51fe0bd001809e67d98d4900f102f5c3fd3aa8 not found: ID does not exist"
Jul 29 20:05:03.099860 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:03.099841    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs"]
Jul 29 20:05:03.103395 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:03.103340    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-641e39/auth-test-model-641e39-kserve-6f76bc44b7-dbpfs"]
Jul 29 20:05:03.234902 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:03.234862    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8" podUID="eb07aea4-c399-4689-9136-50c0e80397f5" containerName="main" containerID="cri-o://abe21a45f90c21aece2d6c59d8c4f010eb5fef1183086377d23b7fb48b47c4cf" gracePeriod=45
Jul 29 20:05:03.490235 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:03.490209    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8"
Jul 29 20:05:03.660913 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:03.660824    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb07aea4-c399-4689-9136-50c0e80397f5-home\") pod \"eb07aea4-c399-4689-9136-50c0e80397f5\" (UID: \"eb07aea4-c399-4689-9136-50c0e80397f5\") "
Jul 29 20:05:03.660913 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:03.660903    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb07aea4-c399-4689-9136-50c0e80397f5-model-cache\") pod \"eb07aea4-c399-4689-9136-50c0e80397f5\" (UID: \"eb07aea4-c399-4689-9136-50c0e80397f5\") "
Jul 29 20:05:03.661126 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:03.660924    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb07aea4-c399-4689-9136-50c0e80397f5-tls-certs\") pod \"eb07aea4-c399-4689-9136-50c0e80397f5\" (UID: \"eb07aea4-c399-4689-9136-50c0e80397f5\") "
Jul 29 20:05:03.661126 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:03.660938    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb07aea4-c399-4689-9136-50c0e80397f5-dshm\") pod \"eb07aea4-c399-4689-9136-50c0e80397f5\" (UID: \"eb07aea4-c399-4689-9136-50c0e80397f5\") "
Jul 29 20:05:03.661126 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:03.661028    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb07aea4-c399-4689-9136-50c0e80397f5-tmp-dir\") pod \"eb07aea4-c399-4689-9136-50c0e80397f5\" (UID: \"eb07aea4-c399-4689-9136-50c0e80397f5\") "
Jul 29 20:05:03.661126 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:03.661056    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jdj4\" (UniqueName: \"kubernetes.io/projected/eb07aea4-c399-4689-9136-50c0e80397f5-kube-api-access-4jdj4\") pod \"eb07aea4-c399-4689-9136-50c0e80397f5\" (UID: \"eb07aea4-c399-4689-9136-50c0e80397f5\") "
Jul 29 20:05:03.661292 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:03.661137    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb07aea4-c399-4689-9136-50c0e80397f5-home" (OuterVolumeSpecName: "home") pod "eb07aea4-c399-4689-9136-50c0e80397f5" (UID: "eb07aea4-c399-4689-9136-50c0e80397f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:05:03.661292 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:03.661234    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb07aea4-c399-4689-9136-50c0e80397f5-model-cache" (OuterVolumeSpecName: "model-cache") pod "eb07aea4-c399-4689-9136-50c0e80397f5" (UID: "eb07aea4-c399-4689-9136-50c0e80397f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:05:03.661366 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:03.661307    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb07aea4-c399-4689-9136-50c0e80397f5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eb07aea4-c399-4689-9136-50c0e80397f5" (UID: "eb07aea4-c399-4689-9136-50c0e80397f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:05:03.661436 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:03.661420    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb07aea4-c399-4689-9136-50c0e80397f5-tmp-dir\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:05:03.661491 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:03.661434    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb07aea4-c399-4689-9136-50c0e80397f5-home\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:05:03.661491 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:03.661444    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb07aea4-c399-4689-9136-50c0e80397f5-model-cache\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:05:03.663195 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:03.663168    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb07aea4-c399-4689-9136-50c0e80397f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eb07aea4-c399-4689-9136-50c0e80397f5" (UID: "eb07aea4-c399-4689-9136-50c0e80397f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:05:03.663270 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:03.663192    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb07aea4-c399-4689-9136-50c0e80397f5-dshm" (OuterVolumeSpecName: "dshm") pod "eb07aea4-c399-4689-9136-50c0e80397f5" (UID: "eb07aea4-c399-4689-9136-50c0e80397f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:05:03.663305 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:03.663286    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb07aea4-c399-4689-9136-50c0e80397f5-kube-api-access-4jdj4" (OuterVolumeSpecName: "kube-api-access-4jdj4") pod "eb07aea4-c399-4689-9136-50c0e80397f5" (UID: "eb07aea4-c399-4689-9136-50c0e80397f5"). InnerVolumeSpecName "kube-api-access-4jdj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:05:03.678675 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:03.678644    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de2179da-9af4-4c9f-869a-bb2b13f180ac" path="/var/lib/kubelet/pods/de2179da-9af4-4c9f-869a-bb2b13f180ac/volumes"
Jul 29 20:05:03.762765 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:03.762731    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jdj4\" (UniqueName: \"kubernetes.io/projected/eb07aea4-c399-4689-9136-50c0e80397f5-kube-api-access-4jdj4\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:05:03.762765 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:03.762759    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb07aea4-c399-4689-9136-50c0e80397f5-tls-certs\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:05:03.762765 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:03.762769    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb07aea4-c399-4689-9136-50c0e80397f5-dshm\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:05:04.085347 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:04.085260    2557 generic.go:356] "Generic (PLEG): container finished" podID="eb07aea4-c399-4689-9136-50c0e80397f5" containerID="abe21a45f90c21aece2d6c59d8c4f010eb5fef1183086377d23b7fb48b47c4cf" exitCode=0
Jul 29 20:05:04.085347 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:04.085301    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8" event={"ID":"eb07aea4-c399-4689-9136-50c0e80397f5","Type":"ContainerDied","Data":"abe21a45f90c21aece2d6c59d8c4f010eb5fef1183086377d23b7fb48b47c4cf"}
Jul 29 20:05:04.085347 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:04.085324    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8"
Jul 29 20:05:04.085631 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:04.085351    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8" event={"ID":"eb07aea4-c399-4689-9136-50c0e80397f5","Type":"ContainerDied","Data":"5b6faecb10e85c7de10400f6422000b7c81985ca5fd561575ea386672b70aa77"}
Jul 29 20:05:04.085631 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:04.085376    2557 scope.go:117] "RemoveContainer" containerID="abe21a45f90c21aece2d6c59d8c4f010eb5fef1183086377d23b7fb48b47c4cf"
Jul 29 20:05:04.094134 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:04.094113    2557 scope.go:117] "RemoveContainer" containerID="abe21a45f90c21aece2d6c59d8c4f010eb5fef1183086377d23b7fb48b47c4cf"
Jul 29 20:05:04.094406 ip-10-0-128-173 kubenswrapper[2557]: E0729 20:05:04.094387    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abe21a45f90c21aece2d6c59d8c4f010eb5fef1183086377d23b7fb48b47c4cf\": container with ID starting with abe21a45f90c21aece2d6c59d8c4f010eb5fef1183086377d23b7fb48b47c4cf not found: ID does not exist" containerID="abe21a45f90c21aece2d6c59d8c4f010eb5fef1183086377d23b7fb48b47c4cf"
Jul 29 20:05:04.094455 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:04.094418    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe21a45f90c21aece2d6c59d8c4f010eb5fef1183086377d23b7fb48b47c4cf"} err="failed to get container status \"abe21a45f90c21aece2d6c59d8c4f010eb5fef1183086377d23b7fb48b47c4cf\": rpc error: code = NotFound desc = could not find container \"abe21a45f90c21aece2d6c59d8c4f010eb5fef1183086377d23b7fb48b47c4cf\": container with ID starting with abe21a45f90c21aece2d6c59d8c4f010eb5fef1183086377d23b7fb48b47c4cf not found: ID does not exist"
Jul 29 20:05:04.101776 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:04.101751    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8"]
Jul 29 20:05:04.107921 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:04.107897    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-4c787c/auth-test-model-4c787c-kserve-5fbf766b99-sbvt8"]
Jul 29 20:05:05.684990 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:05.681411    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb07aea4-c399-4689-9136-50c0e80397f5" path="/var/lib/kubelet/pods/eb07aea4-c399-4689-9136-50c0e80397f5/volumes"
Jul 29 20:05:12.531795 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:12.531756    2557 scope.go:117] "RemoveContainer" containerID="15d95e67848762e0a512f305b5614a595cfc6c25a3313c05262ff1feda1ebd56"
Jul 29 20:05:19.741431 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:19.741390    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs"]
Jul 29 20:05:19.742034 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:19.741987    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb07aea4-c399-4689-9136-50c0e80397f5" containerName="main"
Jul 29 20:05:19.742034 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:19.742009    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb07aea4-c399-4689-9136-50c0e80397f5" containerName="main"
Jul 29 20:05:19.742034 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:19.742035    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de2179da-9af4-4c9f-869a-bb2b13f180ac" containerName="main"
Jul 29 20:05:19.742216 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:19.742043    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="de2179da-9af4-4c9f-869a-bb2b13f180ac" containerName="main"
Jul 29 20:05:19.742216 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:19.742151    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb07aea4-c399-4689-9136-50c0e80397f5" containerName="main"
Jul 29 20:05:19.742216 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:19.742165    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="de2179da-9af4-4c9f-869a-bb2b13f180ac" containerName="main"
Jul 29 20:05:19.745270 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:19.745250    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs"
Jul 29 20:05:19.748103 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:19.748082    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4c787c\"/\"kube-root-ca.crt\""
Jul 29 20:05:19.749163 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:19.749142    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4c787c\"/\"default-dockercfg-lfr2l\""
Jul 29 20:05:19.749263 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:19.749142    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4c787c\"/\"openshift-service-ca.crt\""
Jul 29 20:05:19.749263 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:19.749251    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4c787c\"/\"sub-test-model-4c787c-kserve-self-signed-certs\""
Jul 29 20:05:19.754825 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:19.754800    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs"]
Jul 29 20:05:19.805137 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:19.805105    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dad012b-cad5-427e-a69c-871d2aa81c7e-dshm\") pod \"sub-test-model-4c787c-kserve-85f7764bc9-7d9rs\" (UID: \"9dad012b-cad5-427e-a69c-871d2aa81c7e\") " pod="ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs"
Jul 29 20:05:19.805275 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:19.805164    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k27v\" (UniqueName: \"kubernetes.io/projected/9dad012b-cad5-427e-a69c-871d2aa81c7e-kube-api-access-2k27v\") pod \"sub-test-model-4c787c-kserve-85f7764bc9-7d9rs\" (UID: \"9dad012b-cad5-427e-a69c-871d2aa81c7e\") " pod="ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs"
Jul 29 20:05:19.805275 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:19.805201    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dad012b-cad5-427e-a69c-871d2aa81c7e-tls-certs\") pod \"sub-test-model-4c787c-kserve-85f7764bc9-7d9rs\" (UID: \"9dad012b-cad5-427e-a69c-871d2aa81c7e\") " pod="ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs"
Jul 29 20:05:19.805275 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:19.805226    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dad012b-cad5-427e-a69c-871d2aa81c7e-model-cache\") pod \"sub-test-model-4c787c-kserve-85f7764bc9-7d9rs\" (UID: \"9dad012b-cad5-427e-a69c-871d2aa81c7e\") " pod="ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs"
Jul 29 20:05:19.805275 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:19.805265    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dad012b-cad5-427e-a69c-871d2aa81c7e-home\") pod \"sub-test-model-4c787c-kserve-85f7764bc9-7d9rs\" (UID: \"9dad012b-cad5-427e-a69c-871d2aa81c7e\") " pod="ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs"
Jul 29 20:05:19.805467 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:19.805308    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dad012b-cad5-427e-a69c-871d2aa81c7e-tmp-dir\") pod \"sub-test-model-4c787c-kserve-85f7764bc9-7d9rs\" (UID: \"9dad012b-cad5-427e-a69c-871d2aa81c7e\") " pod="ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs"
Jul 29 20:05:19.906338 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:19.906305    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dad012b-cad5-427e-a69c-871d2aa81c7e-dshm\") pod \"sub-test-model-4c787c-kserve-85f7764bc9-7d9rs\" (UID: \"9dad012b-cad5-427e-a69c-871d2aa81c7e\") " pod="ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs"
Jul 29 20:05:19.906469 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:19.906380    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k27v\" (UniqueName: \"kubernetes.io/projected/9dad012b-cad5-427e-a69c-871d2aa81c7e-kube-api-access-2k27v\") pod \"sub-test-model-4c787c-kserve-85f7764bc9-7d9rs\" (UID: \"9dad012b-cad5-427e-a69c-871d2aa81c7e\") " pod="ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs"
Jul 29 20:05:19.906469 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:19.906408    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dad012b-cad5-427e-a69c-871d2aa81c7e-tls-certs\") pod \"sub-test-model-4c787c-kserve-85f7764bc9-7d9rs\" (UID: \"9dad012b-cad5-427e-a69c-871d2aa81c7e\") " pod="ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs"
Jul 29 20:05:19.906469 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:19.906424    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dad012b-cad5-427e-a69c-871d2aa81c7e-model-cache\") pod \"sub-test-model-4c787c-kserve-85f7764bc9-7d9rs\" (UID: \"9dad012b-cad5-427e-a69c-871d2aa81c7e\") " pod="ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs"
Jul 29 20:05:19.906469 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:19.906454    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dad012b-cad5-427e-a69c-871d2aa81c7e-home\") pod \"sub-test-model-4c787c-kserve-85f7764bc9-7d9rs\" (UID: \"9dad012b-cad5-427e-a69c-871d2aa81c7e\") " pod="ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs"
Jul 29 20:05:19.906606 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:19.906470    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dad012b-cad5-427e-a69c-871d2aa81c7e-tmp-dir\") pod \"sub-test-model-4c787c-kserve-85f7764bc9-7d9rs\" (UID: \"9dad012b-cad5-427e-a69c-871d2aa81c7e\") " pod="ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs"
Jul 29 20:05:19.906874 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:19.906848    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dad012b-cad5-427e-a69c-871d2aa81c7e-tmp-dir\") pod \"sub-test-model-4c787c-kserve-85f7764bc9-7d9rs\" (UID: \"9dad012b-cad5-427e-a69c-871d2aa81c7e\") " pod="ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs"
Jul 29 20:05:19.907069 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:19.907040    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dad012b-cad5-427e-a69c-871d2aa81c7e-model-cache\") pod \"sub-test-model-4c787c-kserve-85f7764bc9-7d9rs\" (UID: \"9dad012b-cad5-427e-a69c-871d2aa81c7e\") " pod="ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs"
Jul 29 20:05:19.907220 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:19.907202    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dad012b-cad5-427e-a69c-871d2aa81c7e-home\") pod \"sub-test-model-4c787c-kserve-85f7764bc9-7d9rs\" (UID: \"9dad012b-cad5-427e-a69c-871d2aa81c7e\") " pod="ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs"
Jul 29 20:05:19.908721 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:19.908695    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dad012b-cad5-427e-a69c-871d2aa81c7e-dshm\") pod \"sub-test-model-4c787c-kserve-85f7764bc9-7d9rs\" (UID: \"9dad012b-cad5-427e-a69c-871d2aa81c7e\") " pod="ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs"
Jul 29 20:05:19.909261 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:19.909238    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dad012b-cad5-427e-a69c-871d2aa81c7e-tls-certs\") pod \"sub-test-model-4c787c-kserve-85f7764bc9-7d9rs\" (UID: \"9dad012b-cad5-427e-a69c-871d2aa81c7e\") " pod="ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs"
Jul 29 20:05:19.921783 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:19.921757    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k27v\" (UniqueName: \"kubernetes.io/projected/9dad012b-cad5-427e-a69c-871d2aa81c7e-kube-api-access-2k27v\") pod \"sub-test-model-4c787c-kserve-85f7764bc9-7d9rs\" (UID: \"9dad012b-cad5-427e-a69c-871d2aa81c7e\") " pod="ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs"
Jul 29 20:05:20.055352 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:20.055277    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs"
Jul 29 20:05:20.191318 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:20.191294    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs"]
Jul 29 20:05:20.193532 ip-10-0-128-173 kubenswrapper[2557]: W0729 20:05:20.193506    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9dad012b_cad5_427e_a69c_871d2aa81c7e.slice/crio-bc1234fda8ae4881342c6edabbfd36f130a663dac8d5c9c659ea8d85f8a00f34 WatchSource:0}: Error finding container bc1234fda8ae4881342c6edabbfd36f130a663dac8d5c9c659ea8d85f8a00f34: Status 404 returned error can't find the container with id bc1234fda8ae4881342c6edabbfd36f130a663dac8d5c9c659ea8d85f8a00f34
Jul 29 20:05:21.171620 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:21.171581    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs" event={"ID":"9dad012b-cad5-427e-a69c-871d2aa81c7e","Type":"ContainerStarted","Data":"e604b03e1fa38f026c6faabbe3fa08d642c8261659cf4398a45764f270f7d572"}
Jul 29 20:05:21.172017 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:21.171633    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs" event={"ID":"9dad012b-cad5-427e-a69c-871d2aa81c7e","Type":"ContainerStarted","Data":"bc1234fda8ae4881342c6edabbfd36f130a663dac8d5c9c659ea8d85f8a00f34"}
Jul 29 20:05:21.192371 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:21.192321    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs" podStartSLOduration=2.192307787 podStartE2EDuration="2.192307787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:05:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:05:21.189662801 +0000 UTC m=+2892.017161858" watchObservedRunningTime="2026-07-29 20:05:21.192307787 +0000 UTC m=+2892.019806844"
Jul 29 20:05:30.056170 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:30.056132    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs"
Jul 29 20:05:30.056647 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:30.056185    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs"
Jul 29 20:05:30.068652 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:30.068615    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs"
Jul 29 20:05:30.230198 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:05:30.230167    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs"
Jul 29 20:06:17.849979 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:17.849941    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs"]
Jul 29 20:06:24.754289 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:24.754242    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds"]
Jul 29 20:06:24.758249 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:24.758224    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds"
Jul 29 20:06:24.762242 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:24.762217    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-641e39\"/\"openshift-service-ca.crt\""
Jul 29 20:06:24.762387 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:24.762254    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-641e39\"/\"rate-test-model-641e39-kserve-self-signed-certs\""
Jul 29 20:06:24.762387 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:24.762319    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-641e39\"/\"default-dockercfg-mgmzf\""
Jul 29 20:06:24.762518 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:24.762396    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-641e39\"/\"kube-root-ca.crt\""
Jul 29 20:06:24.767458 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:24.767434    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds"]
Jul 29 20:06:24.887689 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:24.887651    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18434bd7-41d8-4fd6-b897-9d51433dc9fc-tls-certs\") pod \"rate-test-model-641e39-kserve-5c74cb6d5f-bcpds\" (UID: \"18434bd7-41d8-4fd6-b897-9d51433dc9fc\") " pod="ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds"
Jul 29 20:06:24.887876 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:24.887758    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18434bd7-41d8-4fd6-b897-9d51433dc9fc-dshm\") pod \"rate-test-model-641e39-kserve-5c74cb6d5f-bcpds\" (UID: \"18434bd7-41d8-4fd6-b897-9d51433dc9fc\") " pod="ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds"
Jul 29 20:06:24.887876 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:24.887819    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18434bd7-41d8-4fd6-b897-9d51433dc9fc-home\") pod \"rate-test-model-641e39-kserve-5c74cb6d5f-bcpds\" (UID: \"18434bd7-41d8-4fd6-b897-9d51433dc9fc\") " pod="ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds"
Jul 29 20:06:24.887876 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:24.887846    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18434bd7-41d8-4fd6-b897-9d51433dc9fc-model-cache\") pod \"rate-test-model-641e39-kserve-5c74cb6d5f-bcpds\" (UID: \"18434bd7-41d8-4fd6-b897-9d51433dc9fc\") " pod="ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds"
Jul 29 20:06:24.888097 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:24.887949    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18434bd7-41d8-4fd6-b897-9d51433dc9fc-tmp-dir\") pod \"rate-test-model-641e39-kserve-5c74cb6d5f-bcpds\" (UID: \"18434bd7-41d8-4fd6-b897-9d51433dc9fc\") " pod="ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds"
Jul 29 20:06:24.888097 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:24.888060    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll9lx\" (UniqueName: \"kubernetes.io/projected/18434bd7-41d8-4fd6-b897-9d51433dc9fc-kube-api-access-ll9lx\") pod \"rate-test-model-641e39-kserve-5c74cb6d5f-bcpds\" (UID: \"18434bd7-41d8-4fd6-b897-9d51433dc9fc\") " pod="ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds"
Jul 29 20:06:24.989453 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:24.989417    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18434bd7-41d8-4fd6-b897-9d51433dc9fc-model-cache\") pod \"rate-test-model-641e39-kserve-5c74cb6d5f-bcpds\" (UID: \"18434bd7-41d8-4fd6-b897-9d51433dc9fc\") " pod="ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds"
Jul 29 20:06:24.989714 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:24.989470    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18434bd7-41d8-4fd6-b897-9d51433dc9fc-tmp-dir\") pod \"rate-test-model-641e39-kserve-5c74cb6d5f-bcpds\" (UID: \"18434bd7-41d8-4fd6-b897-9d51433dc9fc\") " pod="ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds"
Jul 29 20:06:24.989714 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:24.989502    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll9lx\" (UniqueName: \"kubernetes.io/projected/18434bd7-41d8-4fd6-b897-9d51433dc9fc-kube-api-access-ll9lx\") pod \"rate-test-model-641e39-kserve-5c74cb6d5f-bcpds\" (UID: \"18434bd7-41d8-4fd6-b897-9d51433dc9fc\") " pod="ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds"
Jul 29 20:06:24.989714 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:24.989529    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18434bd7-41d8-4fd6-b897-9d51433dc9fc-tls-certs\") pod \"rate-test-model-641e39-kserve-5c74cb6d5f-bcpds\" (UID: \"18434bd7-41d8-4fd6-b897-9d51433dc9fc\") " pod="ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds"
Jul 29 20:06:24.989714 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:24.989567    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18434bd7-41d8-4fd6-b897-9d51433dc9fc-dshm\") pod \"rate-test-model-641e39-kserve-5c74cb6d5f-bcpds\" (UID: \"18434bd7-41d8-4fd6-b897-9d51433dc9fc\") " pod="ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds"
Jul 29 20:06:24.989714 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:24.989612    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18434bd7-41d8-4fd6-b897-9d51433dc9fc-home\") pod \"rate-test-model-641e39-kserve-5c74cb6d5f-bcpds\" (UID: \"18434bd7-41d8-4fd6-b897-9d51433dc9fc\") " pod="ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds"
Jul 29 20:06:24.990026 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:24.989910    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18434bd7-41d8-4fd6-b897-9d51433dc9fc-model-cache\") pod \"rate-test-model-641e39-kserve-5c74cb6d5f-bcpds\" (UID: \"18434bd7-41d8-4fd6-b897-9d51433dc9fc\") " pod="ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds"
Jul 29 20:06:24.990026 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:24.989930    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18434bd7-41d8-4fd6-b897-9d51433dc9fc-tmp-dir\") pod \"rate-test-model-641e39-kserve-5c74cb6d5f-bcpds\" (UID: \"18434bd7-41d8-4fd6-b897-9d51433dc9fc\") " pod="ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds"
Jul 29 20:06:24.990026 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:24.989984    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18434bd7-41d8-4fd6-b897-9d51433dc9fc-home\") pod \"rate-test-model-641e39-kserve-5c74cb6d5f-bcpds\" (UID: \"18434bd7-41d8-4fd6-b897-9d51433dc9fc\") " pod="ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds"
Jul 29 20:06:24.991880 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:24.991861    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18434bd7-41d8-4fd6-b897-9d51433dc9fc-dshm\") pod \"rate-test-model-641e39-kserve-5c74cb6d5f-bcpds\" (UID: \"18434bd7-41d8-4fd6-b897-9d51433dc9fc\") " pod="ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds"
Jul 29 20:06:24.992285 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:24.992263    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18434bd7-41d8-4fd6-b897-9d51433dc9fc-tls-certs\") pod \"rate-test-model-641e39-kserve-5c74cb6d5f-bcpds\" (UID: \"18434bd7-41d8-4fd6-b897-9d51433dc9fc\") " pod="ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds"
Jul 29 20:06:24.998525 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:24.998502    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll9lx\" (UniqueName: \"kubernetes.io/projected/18434bd7-41d8-4fd6-b897-9d51433dc9fc-kube-api-access-ll9lx\") pod \"rate-test-model-641e39-kserve-5c74cb6d5f-bcpds\" (UID: \"18434bd7-41d8-4fd6-b897-9d51433dc9fc\") " pod="ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds"
Jul 29 20:06:25.069303 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:25.069200    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds"
Jul 29 20:06:25.202784 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:25.202758    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds"]
Jul 29 20:06:25.205150 ip-10-0-128-173 kubenswrapper[2557]: W0729 20:06:25.205118    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18434bd7_41d8_4fd6_b897_9d51433dc9fc.slice/crio-48e1ac410e5c118bab0a041b24441d4bdc3169f94fe95793adc88396c2770d00 WatchSource:0}: Error finding container 48e1ac410e5c118bab0a041b24441d4bdc3169f94fe95793adc88396c2770d00: Status 404 returned error can't find the container with id 48e1ac410e5c118bab0a041b24441d4bdc3169f94fe95793adc88396c2770d00
Jul 29 20:06:25.486154 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:25.486115    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds" event={"ID":"18434bd7-41d8-4fd6-b897-9d51433dc9fc","Type":"ContainerStarted","Data":"2495fd4e5cad9b7ec702619274184a1776c6dca96043163791eb1af9db7e9845"}
Jul 29 20:06:25.486364 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:25.486165    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds" event={"ID":"18434bd7-41d8-4fd6-b897-9d51433dc9fc","Type":"ContainerStarted","Data":"48e1ac410e5c118bab0a041b24441d4bdc3169f94fe95793adc88396c2770d00"}
Jul 29 20:06:25.503881 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:25.503831    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds" podStartSLOduration=1.503818055 podStartE2EDuration="1.503818055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:06:25.502309945 +0000 UTC m=+2956.329809003" watchObservedRunningTime="2026-07-29 20:06:25.503818055 +0000 UTC m=+2956.331317115"
Jul 29 20:06:32.869127 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:32.869034    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs" podUID="9dad012b-cad5-427e-a69c-871d2aa81c7e" containerName="main" containerID="cri-o://e604b03e1fa38f026c6faabbe3fa08d642c8261659cf4398a45764f270f7d572" gracePeriod=45
Jul 29 20:06:33.132178 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:33.132118    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs"
Jul 29 20:06:33.266779 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:33.266744    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dad012b-cad5-427e-a69c-871d2aa81c7e-tls-certs\") pod \"9dad012b-cad5-427e-a69c-871d2aa81c7e\" (UID: \"9dad012b-cad5-427e-a69c-871d2aa81c7e\") "
Jul 29 20:06:33.266999 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:33.266797    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2k27v\" (UniqueName: \"kubernetes.io/projected/9dad012b-cad5-427e-a69c-871d2aa81c7e-kube-api-access-2k27v\") pod \"9dad012b-cad5-427e-a69c-871d2aa81c7e\" (UID: \"9dad012b-cad5-427e-a69c-871d2aa81c7e\") "
Jul 29 20:06:33.266999 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:33.266822    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dad012b-cad5-427e-a69c-871d2aa81c7e-tmp-dir\") pod \"9dad012b-cad5-427e-a69c-871d2aa81c7e\" (UID: \"9dad012b-cad5-427e-a69c-871d2aa81c7e\") "
Jul 29 20:06:33.266999 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:33.266908    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dad012b-cad5-427e-a69c-871d2aa81c7e-dshm\") pod \"9dad012b-cad5-427e-a69c-871d2aa81c7e\" (UID: \"9dad012b-cad5-427e-a69c-871d2aa81c7e\") "
Jul 29 20:06:33.266999 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:33.266945    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dad012b-cad5-427e-a69c-871d2aa81c7e-home\") pod \"9dad012b-cad5-427e-a69c-871d2aa81c7e\" (UID: \"9dad012b-cad5-427e-a69c-871d2aa81c7e\") "
Jul 29 20:06:33.267240 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:33.267006    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dad012b-cad5-427e-a69c-871d2aa81c7e-model-cache\") pod \"9dad012b-cad5-427e-a69c-871d2aa81c7e\" (UID: \"9dad012b-cad5-427e-a69c-871d2aa81c7e\") "
Jul 29 20:06:33.267300 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:33.267229    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dad012b-cad5-427e-a69c-871d2aa81c7e-home" (OuterVolumeSpecName: "home") pod "9dad012b-cad5-427e-a69c-871d2aa81c7e" (UID: "9dad012b-cad5-427e-a69c-871d2aa81c7e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:06:33.267300 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:33.267238    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dad012b-cad5-427e-a69c-871d2aa81c7e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9dad012b-cad5-427e-a69c-871d2aa81c7e" (UID: "9dad012b-cad5-427e-a69c-871d2aa81c7e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:06:33.267416 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:33.267328    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dad012b-cad5-427e-a69c-871d2aa81c7e-model-cache" (OuterVolumeSpecName: "model-cache") pod "9dad012b-cad5-427e-a69c-871d2aa81c7e" (UID: "9dad012b-cad5-427e-a69c-871d2aa81c7e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:06:33.267473 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:33.267421    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dad012b-cad5-427e-a69c-871d2aa81c7e-home\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:06:33.267473 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:33.267439    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dad012b-cad5-427e-a69c-871d2aa81c7e-model-cache\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:06:33.267473 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:33.267455    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dad012b-cad5-427e-a69c-871d2aa81c7e-tmp-dir\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:06:33.269116 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:33.269090    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dad012b-cad5-427e-a69c-871d2aa81c7e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9dad012b-cad5-427e-a69c-871d2aa81c7e" (UID: "9dad012b-cad5-427e-a69c-871d2aa81c7e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:06:33.269116 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:33.269090    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dad012b-cad5-427e-a69c-871d2aa81c7e-kube-api-access-2k27v" (OuterVolumeSpecName: "kube-api-access-2k27v") pod "9dad012b-cad5-427e-a69c-871d2aa81c7e" (UID: "9dad012b-cad5-427e-a69c-871d2aa81c7e"). InnerVolumeSpecName "kube-api-access-2k27v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:06:33.269269 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:33.269184    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dad012b-cad5-427e-a69c-871d2aa81c7e-dshm" (OuterVolumeSpecName: "dshm") pod "9dad012b-cad5-427e-a69c-871d2aa81c7e" (UID: "9dad012b-cad5-427e-a69c-871d2aa81c7e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:06:33.368785 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:33.368750    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dad012b-cad5-427e-a69c-871d2aa81c7e-tls-certs\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:06:33.368785 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:33.368781    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k27v\" (UniqueName: \"kubernetes.io/projected/9dad012b-cad5-427e-a69c-871d2aa81c7e-kube-api-access-2k27v\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:06:33.368785 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:33.368790    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dad012b-cad5-427e-a69c-871d2aa81c7e-dshm\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:06:33.525206 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:33.525175    2557 generic.go:356] "Generic (PLEG): container finished" podID="9dad012b-cad5-427e-a69c-871d2aa81c7e" containerID="e604b03e1fa38f026c6faabbe3fa08d642c8261659cf4398a45764f270f7d572" exitCode=0
Jul 29 20:06:33.525407 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:33.525219    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs"
Jul 29 20:06:33.525407 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:33.525268    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs" event={"ID":"9dad012b-cad5-427e-a69c-871d2aa81c7e","Type":"ContainerDied","Data":"e604b03e1fa38f026c6faabbe3fa08d642c8261659cf4398a45764f270f7d572"}
Jul 29 20:06:33.525407 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:33.525306    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs" event={"ID":"9dad012b-cad5-427e-a69c-871d2aa81c7e","Type":"ContainerDied","Data":"bc1234fda8ae4881342c6edabbfd36f130a663dac8d5c9c659ea8d85f8a00f34"}
Jul 29 20:06:33.525407 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:33.525324    2557 scope.go:117] "RemoveContainer" containerID="e604b03e1fa38f026c6faabbe3fa08d642c8261659cf4398a45764f270f7d572"
Jul 29 20:06:33.541379 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:33.541360    2557 scope.go:117] "RemoveContainer" containerID="e604b03e1fa38f026c6faabbe3fa08d642c8261659cf4398a45764f270f7d572"
Jul 29 20:06:33.541655 ip-10-0-128-173 kubenswrapper[2557]: E0729 20:06:33.541632    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e604b03e1fa38f026c6faabbe3fa08d642c8261659cf4398a45764f270f7d572\": container with ID starting with e604b03e1fa38f026c6faabbe3fa08d642c8261659cf4398a45764f270f7d572 not found: ID does not exist" containerID="e604b03e1fa38f026c6faabbe3fa08d642c8261659cf4398a45764f270f7d572"
Jul 29 20:06:33.541721 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:33.541659    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e604b03e1fa38f026c6faabbe3fa08d642c8261659cf4398a45764f270f7d572"} err="failed to get container status \"e604b03e1fa38f026c6faabbe3fa08d642c8261659cf4398a45764f270f7d572\": rpc error: code = NotFound desc = could not find container \"e604b03e1fa38f026c6faabbe3fa08d642c8261659cf4398a45764f270f7d572\": container with ID starting with e604b03e1fa38f026c6faabbe3fa08d642c8261659cf4398a45764f270f7d572 not found: ID does not exist"
Jul 29 20:06:33.552650 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:33.552628    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs"]
Jul 29 20:06:33.556288 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:33.556264    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-4c787c/sub-test-model-4c787c-kserve-85f7764bc9-7d9rs"]
Jul 29 20:06:33.678754 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:33.678722    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dad012b-cad5-427e-a69c-871d2aa81c7e" path="/var/lib/kubelet/pods/9dad012b-cad5-427e-a69c-871d2aa81c7e/volumes"
Jul 29 20:06:35.070414 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:35.070382    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds"
Jul 29 20:06:35.070802 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:35.070433    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds"
Jul 29 20:06:35.083004 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:35.082977    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds"
Jul 29 20:06:35.547518 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:35.547486    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds"
Jul 29 20:06:50.662774 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:50.662737    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg"]
Jul 29 20:06:50.663289 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:50.663164    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dad012b-cad5-427e-a69c-871d2aa81c7e" containerName="main"
Jul 29 20:06:50.663289 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:50.663178    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dad012b-cad5-427e-a69c-871d2aa81c7e" containerName="main"
Jul 29 20:06:50.663289 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:50.663261    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dad012b-cad5-427e-a69c-871d2aa81c7e" containerName="main"
Jul 29 20:06:50.666162 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:50.666145    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg"
Jul 29 20:06:50.670300 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:50.670279    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4c787c\"/\"kube-root-ca.crt\""
Jul 29 20:06:50.670300 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:50.670291    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4c787c\"/\"openshift-service-ca.crt\""
Jul 29 20:06:50.670487 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:50.670278    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4c787c\"/\"default-dockercfg-lfr2l\""
Jul 29 20:06:50.670574 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:50.670553    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4c787c\"/\"rate-test-model-4c787c-kserve-self-signed-certs\""
Jul 29 20:06:50.676710 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:50.676688    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg"]
Jul 29 20:06:50.718583 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:50.718553    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57f82bf0-495d-4338-806d-446f25c183e2-tls-certs\") pod \"rate-test-model-4c787c-kserve-58775bcf67-zndsg\" (UID: \"57f82bf0-495d-4338-806d-446f25c183e2\") " pod="ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg"
Jul 29 20:06:50.718711 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:50.718649    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v44w\" (UniqueName: \"kubernetes.io/projected/57f82bf0-495d-4338-806d-446f25c183e2-kube-api-access-6v44w\") pod \"rate-test-model-4c787c-kserve-58775bcf67-zndsg\" (UID: \"57f82bf0-495d-4338-806d-446f25c183e2\") " pod="ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg"
Jul 29 20:06:50.718711 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:50.718691    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f82bf0-495d-4338-806d-446f25c183e2-dshm\") pod \"rate-test-model-4c787c-kserve-58775bcf67-zndsg\" (UID: \"57f82bf0-495d-4338-806d-446f25c183e2\") " pod="ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg"
Jul 29 20:06:50.718791 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:50.718756    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f82bf0-495d-4338-806d-446f25c183e2-model-cache\") pod \"rate-test-model-4c787c-kserve-58775bcf67-zndsg\" (UID: \"57f82bf0-495d-4338-806d-446f25c183e2\") " pod="ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg"
Jul 29 20:06:50.718791 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:50.718787    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57f82bf0-495d-4338-806d-446f25c183e2-home\") pod \"rate-test-model-4c787c-kserve-58775bcf67-zndsg\" (UID: \"57f82bf0-495d-4338-806d-446f25c183e2\") " pod="ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg"
Jul 29 20:06:50.718870 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:50.718811    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f82bf0-495d-4338-806d-446f25c183e2-tmp-dir\") pod \"rate-test-model-4c787c-kserve-58775bcf67-zndsg\" (UID: \"57f82bf0-495d-4338-806d-446f25c183e2\") " pod="ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg"
Jul 29 20:06:50.819348 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:50.819315    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v44w\" (UniqueName: \"kubernetes.io/projected/57f82bf0-495d-4338-806d-446f25c183e2-kube-api-access-6v44w\") pod \"rate-test-model-4c787c-kserve-58775bcf67-zndsg\" (UID: \"57f82bf0-495d-4338-806d-446f25c183e2\") " pod="ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg"
Jul 29 20:06:50.819507 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:50.819363    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f82bf0-495d-4338-806d-446f25c183e2-dshm\") pod \"rate-test-model-4c787c-kserve-58775bcf67-zndsg\" (UID: \"57f82bf0-495d-4338-806d-446f25c183e2\") " pod="ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg"
Jul 29 20:06:50.819507 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:50.819480    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f82bf0-495d-4338-806d-446f25c183e2-model-cache\") pod \"rate-test-model-4c787c-kserve-58775bcf67-zndsg\" (UID: \"57f82bf0-495d-4338-806d-446f25c183e2\") " pod="ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg"
Jul 29 20:06:50.819610 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:50.819528    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57f82bf0-495d-4338-806d-446f25c183e2-home\") pod \"rate-test-model-4c787c-kserve-58775bcf67-zndsg\" (UID: \"57f82bf0-495d-4338-806d-446f25c183e2\") " pod="ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg"
Jul 29 20:06:50.819610 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:50.819561    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f82bf0-495d-4338-806d-446f25c183e2-tmp-dir\") pod \"rate-test-model-4c787c-kserve-58775bcf67-zndsg\" (UID: \"57f82bf0-495d-4338-806d-446f25c183e2\") " pod="ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg"
Jul 29 20:06:50.819610 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:50.819594    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57f82bf0-495d-4338-806d-446f25c183e2-tls-certs\") pod \"rate-test-model-4c787c-kserve-58775bcf67-zndsg\" (UID: \"57f82bf0-495d-4338-806d-446f25c183e2\") " pod="ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg"
Jul 29 20:06:50.819899 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:50.819875    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f82bf0-495d-4338-806d-446f25c183e2-model-cache\") pod \"rate-test-model-4c787c-kserve-58775bcf67-zndsg\" (UID: \"57f82bf0-495d-4338-806d-446f25c183e2\") " pod="ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg"
Jul 29 20:06:50.819988 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:50.819882    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57f82bf0-495d-4338-806d-446f25c183e2-home\") pod \"rate-test-model-4c787c-kserve-58775bcf67-zndsg\" (UID: \"57f82bf0-495d-4338-806d-446f25c183e2\") " pod="ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg"
Jul 29 20:06:50.820065 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:50.819993    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f82bf0-495d-4338-806d-446f25c183e2-tmp-dir\") pod \"rate-test-model-4c787c-kserve-58775bcf67-zndsg\" (UID: \"57f82bf0-495d-4338-806d-446f25c183e2\") " pod="ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg"
Jul 29 20:06:50.821612 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:50.821591    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f82bf0-495d-4338-806d-446f25c183e2-dshm\") pod \"rate-test-model-4c787c-kserve-58775bcf67-zndsg\" (UID: \"57f82bf0-495d-4338-806d-446f25c183e2\") " pod="ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg"
Jul 29 20:06:50.822108 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:50.822085    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57f82bf0-495d-4338-806d-446f25c183e2-tls-certs\") pod \"rate-test-model-4c787c-kserve-58775bcf67-zndsg\" (UID: \"57f82bf0-495d-4338-806d-446f25c183e2\") " pod="ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg"
Jul 29 20:06:50.827981 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:50.827950    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v44w\" (UniqueName: \"kubernetes.io/projected/57f82bf0-495d-4338-806d-446f25c183e2-kube-api-access-6v44w\") pod \"rate-test-model-4c787c-kserve-58775bcf67-zndsg\" (UID: \"57f82bf0-495d-4338-806d-446f25c183e2\") " pod="ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg"
Jul 29 20:06:50.976584 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:50.976501    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg"
Jul 29 20:06:51.109641 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:51.109613    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg"]
Jul 29 20:06:51.111400 ip-10-0-128-173 kubenswrapper[2557]: W0729 20:06:51.111371    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57f82bf0_495d_4338_806d_446f25c183e2.slice/crio-7d60b752fcfac88e366051369f06bf82c6183c87e2265c3c2ffb457376198985 WatchSource:0}: Error finding container 7d60b752fcfac88e366051369f06bf82c6183c87e2265c3c2ffb457376198985: Status 404 returned error can't find the container with id 7d60b752fcfac88e366051369f06bf82c6183c87e2265c3c2ffb457376198985
Jul 29 20:06:51.613254 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:51.613214    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg" event={"ID":"57f82bf0-495d-4338-806d-446f25c183e2","Type":"ContainerStarted","Data":"6ed28f7c7f8a9d9232d7f64a8f31de40502acf5aa9e883ee66f9490d18e955a8"}
Jul 29 20:06:51.613424 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:51.613263    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg" event={"ID":"57f82bf0-495d-4338-806d-446f25c183e2","Type":"ContainerStarted","Data":"7d60b752fcfac88e366051369f06bf82c6183c87e2265c3c2ffb457376198985"}
Jul 29 20:06:51.631816 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:06:51.631764    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg" podStartSLOduration=1.631749023 podStartE2EDuration="1.631749023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:06:51.629907804 +0000 UTC m=+2982.457406865" watchObservedRunningTime="2026-07-29 20:06:51.631749023 +0000 UTC m=+2982.459248080"
Jul 29 20:07:00.977513 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:07:00.977470    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg"
Jul 29 20:07:00.977946 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:07:00.977538    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg"
Jul 29 20:07:00.989864 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:07:00.989837    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg"
Jul 29 20:07:01.679125 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:07:01.679096    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg"
Jul 29 20:08:04.867208 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:04.867125    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds"]
Jul 29 20:08:05.892256 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:05.892215    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg"]
Jul 29 20:08:19.885871 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:19.885824    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds" podUID="18434bd7-41d8-4fd6-b897-9d51433dc9fc" containerName="main" containerID="cri-o://2495fd4e5cad9b7ec702619274184a1776c6dca96043163791eb1af9db7e9845" gracePeriod=45
Jul 29 20:08:20.037015 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:20.036981    2557 generic.go:356] "Generic (PLEG): container finished" podID="18434bd7-41d8-4fd6-b897-9d51433dc9fc" containerID="2495fd4e5cad9b7ec702619274184a1776c6dca96043163791eb1af9db7e9845" exitCode=0
Jul 29 20:08:20.037180 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:20.037059    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds" event={"ID":"18434bd7-41d8-4fd6-b897-9d51433dc9fc","Type":"ContainerDied","Data":"2495fd4e5cad9b7ec702619274184a1776c6dca96043163791eb1af9db7e9845"}
Jul 29 20:08:20.132319 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:20.132287    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds"
Jul 29 20:08:20.250467 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:20.250384    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18434bd7-41d8-4fd6-b897-9d51433dc9fc-home\") pod \"18434bd7-41d8-4fd6-b897-9d51433dc9fc\" (UID: \"18434bd7-41d8-4fd6-b897-9d51433dc9fc\") "
Jul 29 20:08:20.250467 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:20.250435    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18434bd7-41d8-4fd6-b897-9d51433dc9fc-model-cache\") pod \"18434bd7-41d8-4fd6-b897-9d51433dc9fc\" (UID: \"18434bd7-41d8-4fd6-b897-9d51433dc9fc\") "
Jul 29 20:08:20.250467 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:20.250455    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18434bd7-41d8-4fd6-b897-9d51433dc9fc-tmp-dir\") pod \"18434bd7-41d8-4fd6-b897-9d51433dc9fc\" (UID: \"18434bd7-41d8-4fd6-b897-9d51433dc9fc\") "
Jul 29 20:08:20.250740 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:20.250508    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ll9lx\" (UniqueName: \"kubernetes.io/projected/18434bd7-41d8-4fd6-b897-9d51433dc9fc-kube-api-access-ll9lx\") pod \"18434bd7-41d8-4fd6-b897-9d51433dc9fc\" (UID: \"18434bd7-41d8-4fd6-b897-9d51433dc9fc\") "
Jul 29 20:08:20.250740 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:20.250573    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18434bd7-41d8-4fd6-b897-9d51433dc9fc-tls-certs\") pod \"18434bd7-41d8-4fd6-b897-9d51433dc9fc\" (UID: \"18434bd7-41d8-4fd6-b897-9d51433dc9fc\") "
Jul 29 20:08:20.250740 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:20.250609    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18434bd7-41d8-4fd6-b897-9d51433dc9fc-dshm\") pod \"18434bd7-41d8-4fd6-b897-9d51433dc9fc\" (UID: \"18434bd7-41d8-4fd6-b897-9d51433dc9fc\") "
Jul 29 20:08:20.250740 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:20.250687    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18434bd7-41d8-4fd6-b897-9d51433dc9fc-home" (OuterVolumeSpecName: "home") pod "18434bd7-41d8-4fd6-b897-9d51433dc9fc" (UID: "18434bd7-41d8-4fd6-b897-9d51433dc9fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:08:20.250740 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:20.250715    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18434bd7-41d8-4fd6-b897-9d51433dc9fc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "18434bd7-41d8-4fd6-b897-9d51433dc9fc" (UID: "18434bd7-41d8-4fd6-b897-9d51433dc9fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:08:20.250740 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:20.250693    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18434bd7-41d8-4fd6-b897-9d51433dc9fc-model-cache" (OuterVolumeSpecName: "model-cache") pod "18434bd7-41d8-4fd6-b897-9d51433dc9fc" (UID: "18434bd7-41d8-4fd6-b897-9d51433dc9fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:08:20.251054 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:20.250904    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18434bd7-41d8-4fd6-b897-9d51433dc9fc-home\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:08:20.251054 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:20.250926    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18434bd7-41d8-4fd6-b897-9d51433dc9fc-model-cache\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:08:20.251054 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:20.250939    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18434bd7-41d8-4fd6-b897-9d51433dc9fc-tmp-dir\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:08:20.252775 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:20.252747    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18434bd7-41d8-4fd6-b897-9d51433dc9fc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "18434bd7-41d8-4fd6-b897-9d51433dc9fc" (UID: "18434bd7-41d8-4fd6-b897-9d51433dc9fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:08:20.252775 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:20.252760    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18434bd7-41d8-4fd6-b897-9d51433dc9fc-kube-api-access-ll9lx" (OuterVolumeSpecName: "kube-api-access-ll9lx") pod "18434bd7-41d8-4fd6-b897-9d51433dc9fc" (UID: "18434bd7-41d8-4fd6-b897-9d51433dc9fc"). InnerVolumeSpecName "kube-api-access-ll9lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:08:20.252775 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:20.252753    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18434bd7-41d8-4fd6-b897-9d51433dc9fc-dshm" (OuterVolumeSpecName: "dshm") pod "18434bd7-41d8-4fd6-b897-9d51433dc9fc" (UID: "18434bd7-41d8-4fd6-b897-9d51433dc9fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:08:20.352382 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:20.352352    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18434bd7-41d8-4fd6-b897-9d51433dc9fc-tls-certs\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:08:20.352382 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:20.352377    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18434bd7-41d8-4fd6-b897-9d51433dc9fc-dshm\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:08:20.352382 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:20.352387    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll9lx\" (UniqueName: \"kubernetes.io/projected/18434bd7-41d8-4fd6-b897-9d51433dc9fc-kube-api-access-ll9lx\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:08:20.910729 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:20.910673    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg" podUID="57f82bf0-495d-4338-806d-446f25c183e2" containerName="main" containerID="cri-o://6ed28f7c7f8a9d9232d7f64a8f31de40502acf5aa9e883ee66f9490d18e955a8" gracePeriod=45
Jul 29 20:08:21.043677 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:21.043641    2557 generic.go:356] "Generic (PLEG): container finished" podID="57f82bf0-495d-4338-806d-446f25c183e2" containerID="6ed28f7c7f8a9d9232d7f64a8f31de40502acf5aa9e883ee66f9490d18e955a8" exitCode=0
Jul 29 20:08:21.043797 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:21.043708    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg" event={"ID":"57f82bf0-495d-4338-806d-446f25c183e2","Type":"ContainerDied","Data":"6ed28f7c7f8a9d9232d7f64a8f31de40502acf5aa9e883ee66f9490d18e955a8"}
Jul 29 20:08:21.045139 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:21.045119    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds"
Jul 29 20:08:21.045243 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:21.045154    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds" event={"ID":"18434bd7-41d8-4fd6-b897-9d51433dc9fc","Type":"ContainerDied","Data":"48e1ac410e5c118bab0a041b24441d4bdc3169f94fe95793adc88396c2770d00"}
Jul 29 20:08:21.045243 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:21.045198    2557 scope.go:117] "RemoveContainer" containerID="2495fd4e5cad9b7ec702619274184a1776c6dca96043163791eb1af9db7e9845"
Jul 29 20:08:21.067818 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:21.067794    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds"]
Jul 29 20:08:21.072537 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:21.072514    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-641e39/rate-test-model-641e39-kserve-5c74cb6d5f-bcpds"]
Jul 29 20:08:21.175289 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:21.175236    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg"
Jul 29 20:08:21.258654 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:21.258621    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6v44w\" (UniqueName: \"kubernetes.io/projected/57f82bf0-495d-4338-806d-446f25c183e2-kube-api-access-6v44w\") pod \"57f82bf0-495d-4338-806d-446f25c183e2\" (UID: \"57f82bf0-495d-4338-806d-446f25c183e2\") "
Jul 29 20:08:21.258812 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:21.258666    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f82bf0-495d-4338-806d-446f25c183e2-tmp-dir\") pod \"57f82bf0-495d-4338-806d-446f25c183e2\" (UID: \"57f82bf0-495d-4338-806d-446f25c183e2\") "
Jul 29 20:08:21.258812 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:21.258688    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57f82bf0-495d-4338-806d-446f25c183e2-home\") pod \"57f82bf0-495d-4338-806d-446f25c183e2\" (UID: \"57f82bf0-495d-4338-806d-446f25c183e2\") "
Jul 29 20:08:21.258812 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:21.258735    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57f82bf0-495d-4338-806d-446f25c183e2-tls-certs\") pod \"57f82bf0-495d-4338-806d-446f25c183e2\" (UID: \"57f82bf0-495d-4338-806d-446f25c183e2\") "
Jul 29 20:08:21.259019 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:21.258819    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f82bf0-495d-4338-806d-446f25c183e2-dshm\") pod \"57f82bf0-495d-4338-806d-446f25c183e2\" (UID: \"57f82bf0-495d-4338-806d-446f25c183e2\") "
Jul 29 20:08:21.259019 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:21.258835    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f82bf0-495d-4338-806d-446f25c183e2-model-cache\") pod \"57f82bf0-495d-4338-806d-446f25c183e2\" (UID: \"57f82bf0-495d-4338-806d-446f25c183e2\") "
Jul 29 20:08:21.259019 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:21.258951    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f82bf0-495d-4338-806d-446f25c183e2-home" (OuterVolumeSpecName: "home") pod "57f82bf0-495d-4338-806d-446f25c183e2" (UID: "57f82bf0-495d-4338-806d-446f25c183e2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:08:21.259019 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:21.258990    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f82bf0-495d-4338-806d-446f25c183e2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "57f82bf0-495d-4338-806d-446f25c183e2" (UID: "57f82bf0-495d-4338-806d-446f25c183e2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:08:21.259235 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:21.259140    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f82bf0-495d-4338-806d-446f25c183e2-tmp-dir\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:08:21.259235 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:21.259162    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57f82bf0-495d-4338-806d-446f25c183e2-home\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:08:21.259235 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:21.259171    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f82bf0-495d-4338-806d-446f25c183e2-model-cache" (OuterVolumeSpecName: "model-cache") pod "57f82bf0-495d-4338-806d-446f25c183e2" (UID: "57f82bf0-495d-4338-806d-446f25c183e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:08:21.260806 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:21.260779    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f82bf0-495d-4338-806d-446f25c183e2-dshm" (OuterVolumeSpecName: "dshm") pod "57f82bf0-495d-4338-806d-446f25c183e2" (UID: "57f82bf0-495d-4338-806d-446f25c183e2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:08:21.260914 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:21.260834    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f82bf0-495d-4338-806d-446f25c183e2-kube-api-access-6v44w" (OuterVolumeSpecName: "kube-api-access-6v44w") pod "57f82bf0-495d-4338-806d-446f25c183e2" (UID: "57f82bf0-495d-4338-806d-446f25c183e2"). InnerVolumeSpecName "kube-api-access-6v44w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:08:21.260914 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:21.260895    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57f82bf0-495d-4338-806d-446f25c183e2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "57f82bf0-495d-4338-806d-446f25c183e2" (UID: "57f82bf0-495d-4338-806d-446f25c183e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:08:21.360563 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:21.360533    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f82bf0-495d-4338-806d-446f25c183e2-dshm\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:08:21.360563 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:21.360559    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f82bf0-495d-4338-806d-446f25c183e2-model-cache\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:08:21.360563 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:21.360570    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v44w\" (UniqueName: \"kubernetes.io/projected/57f82bf0-495d-4338-806d-446f25c183e2-kube-api-access-6v44w\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:08:21.360777 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:21.360579    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57f82bf0-495d-4338-806d-446f25c183e2-tls-certs\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:08:21.678286 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:21.678254    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18434bd7-41d8-4fd6-b897-9d51433dc9fc" path="/var/lib/kubelet/pods/18434bd7-41d8-4fd6-b897-9d51433dc9fc/volumes"
Jul 29 20:08:22.052082 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:22.051989    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg"
Jul 29 20:08:22.052082 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:22.051995    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg" event={"ID":"57f82bf0-495d-4338-806d-446f25c183e2","Type":"ContainerDied","Data":"7d60b752fcfac88e366051369f06bf82c6183c87e2265c3c2ffb457376198985"}
Jul 29 20:08:22.052585 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:22.052121    2557 scope.go:117] "RemoveContainer" containerID="6ed28f7c7f8a9d9232d7f64a8f31de40502acf5aa9e883ee66f9490d18e955a8"
Jul 29 20:08:22.069262 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:22.069235    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg"]
Jul 29 20:08:22.075305 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:22.075281    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-4c787c/rate-test-model-4c787c-kserve-58775bcf67-zndsg"]
Jul 29 20:08:23.678331 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:23.678295    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57f82bf0-495d-4338-806d-446f25c183e2" path="/var/lib/kubelet/pods/57f82bf0-495d-4338-806d-446f25c183e2/volumes"
Jul 29 20:08:30.336301 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:30.336270    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n"]
Jul 29 20:08:30.336756 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:30.336664    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57f82bf0-495d-4338-806d-446f25c183e2" containerName="main"
Jul 29 20:08:30.336756 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:30.336678    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="57f82bf0-495d-4338-806d-446f25c183e2" containerName="main"
Jul 29 20:08:30.336756 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:30.336699    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18434bd7-41d8-4fd6-b897-9d51433dc9fc" containerName="main"
Jul 29 20:08:30.336756 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:30.336704    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="18434bd7-41d8-4fd6-b897-9d51433dc9fc" containerName="main"
Jul 29 20:08:30.336893 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:30.336764    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="57f82bf0-495d-4338-806d-446f25c183e2" containerName="main"
Jul 29 20:08:30.336893 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:30.336776    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="18434bd7-41d8-4fd6-b897-9d51433dc9fc" containerName="main"
Jul 29 20:08:30.341023 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:30.341003    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n"
Jul 29 20:08:30.344647 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:30.344625    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f0b78f-serving-cert\""
Jul 29 20:08:30.354824 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:30.354797    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n"]
Jul 29 20:08:30.442002 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:30.441941    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e61b2c1-44ec-4153-8ca9-a852b2882b08-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n\" (UID: \"5e61b2c1-44ec-4153-8ca9-a852b2882b08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n"
Jul 29 20:08:30.442173 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:30.442059    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e61b2c1-44ec-4153-8ca9-a852b2882b08-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n\" (UID: \"5e61b2c1-44ec-4153-8ca9-a852b2882b08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n"
Jul 29 20:08:30.442173 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:30.442097    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwjdz\" (UniqueName: \"kubernetes.io/projected/5e61b2c1-44ec-4153-8ca9-a852b2882b08-kube-api-access-bwjdz\") pod \"maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n\" (UID: \"5e61b2c1-44ec-4153-8ca9-a852b2882b08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n"
Jul 29 20:08:30.542645 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:30.542612    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e61b2c1-44ec-4153-8ca9-a852b2882b08-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n\" (UID: \"5e61b2c1-44ec-4153-8ca9-a852b2882b08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n"
Jul 29 20:08:30.542801 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:30.542678    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e61b2c1-44ec-4153-8ca9-a852b2882b08-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n\" (UID: \"5e61b2c1-44ec-4153-8ca9-a852b2882b08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n"
Jul 29 20:08:30.542801 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:30.542725    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwjdz\" (UniqueName: \"kubernetes.io/projected/5e61b2c1-44ec-4153-8ca9-a852b2882b08-kube-api-access-bwjdz\") pod \"maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n\" (UID: \"5e61b2c1-44ec-4153-8ca9-a852b2882b08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n"
Jul 29 20:08:30.543449 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:30.543420    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e61b2c1-44ec-4153-8ca9-a852b2882b08-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n\" (UID: \"5e61b2c1-44ec-4153-8ca9-a852b2882b08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n"
Jul 29 20:08:30.545193 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:30.545170    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e61b2c1-44ec-4153-8ca9-a852b2882b08-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n\" (UID: \"5e61b2c1-44ec-4153-8ca9-a852b2882b08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n"
Jul 29 20:08:30.560740 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:30.560712    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwjdz\" (UniqueName: \"kubernetes.io/projected/5e61b2c1-44ec-4153-8ca9-a852b2882b08-kube-api-access-bwjdz\") pod \"maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n\" (UID: \"5e61b2c1-44ec-4153-8ca9-a852b2882b08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n"
Jul 29 20:08:30.651203 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:30.651175    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n"
Jul 29 20:08:30.793029 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:30.792999    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n"]
Jul 29 20:08:30.794915 ip-10-0-128-173 kubenswrapper[2557]: W0729 20:08:30.794888    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e61b2c1_44ec_4153_8ca9_a852b2882b08.slice/crio-d2b095f01497fc04d7a13455ef2e78c3bf40b46d074eaf0c80cfe5edeacdca96 WatchSource:0}: Error finding container d2b095f01497fc04d7a13455ef2e78c3bf40b46d074eaf0c80cfe5edeacdca96: Status 404 returned error can't find the container with id d2b095f01497fc04d7a13455ef2e78c3bf40b46d074eaf0c80cfe5edeacdca96
Jul 29 20:08:30.796211 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:30.796193    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 20:08:31.099498 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:31.099419    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n" event={"ID":"5e61b2c1-44ec-4153-8ca9-a852b2882b08","Type":"ContainerStarted","Data":"d2b095f01497fc04d7a13455ef2e78c3bf40b46d074eaf0c80cfe5edeacdca96"}
Jul 29 20:08:31.274626 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:31.274597    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b78f-6d94d85f88-jsgdq"]
Jul 29 20:08:31.279267 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:31.279248    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b78f-6d94d85f88-jsgdq"
Jul 29 20:08:31.282346 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:31.282320    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f0b78f-dockercfg-dt676\""
Jul 29 20:08:31.282472 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:31.282360    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f0b78f\""
Jul 29 20:08:31.291335 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:31.291309    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b78f-6d94d85f88-jsgdq"]
Jul 29 20:08:31.344691 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:31.344653    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0b78f-68b67497b8-6w6sn"]
Jul 29 20:08:31.348216 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:31.348194    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b78f-68b67497b8-6w6sn"
Jul 29 20:08:31.350665 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:31.350600    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f81544c9-87bf-444e-8946-5f9f0d082f1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0b78f-6d94d85f88-jsgdq\" (UID: \"f81544c9-87bf-444e-8946-5f9f0d082f1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b78f-6d94d85f88-jsgdq"
Jul 29 20:08:31.350665 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:31.350634    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kvkr\" (UniqueName: \"kubernetes.io/projected/f81544c9-87bf-444e-8946-5f9f0d082f1b-kube-api-access-2kvkr\") pod \"payload-pre-processing-e2e-ipp-a-f0b78f-6d94d85f88-jsgdq\" (UID: \"f81544c9-87bf-444e-8946-5f9f0d082f1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b78f-6d94d85f88-jsgdq"
Jul 29 20:08:31.357067 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:31.357046    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0b78f-68b67497b8-6w6sn"]
Jul 29 20:08:31.451896 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:31.451872    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f81544c9-87bf-444e-8946-5f9f0d082f1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0b78f-6d94d85f88-jsgdq\" (UID: \"f81544c9-87bf-444e-8946-5f9f0d082f1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b78f-6d94d85f88-jsgdq"
Jul 29 20:08:31.452003 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:31.451910    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kvkr\" (UniqueName: \"kubernetes.io/projected/f81544c9-87bf-444e-8946-5f9f0d082f1b-kube-api-access-2kvkr\") pod \"payload-pre-processing-e2e-ipp-a-f0b78f-6d94d85f88-jsgdq\" (UID: \"f81544c9-87bf-444e-8946-5f9f0d082f1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b78f-6d94d85f88-jsgdq"
Jul 29 20:08:31.452003 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:31.451937    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f48b6661-c056-4615-abbd-731a5a79a4b3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0b78f-68b67497b8-6w6sn\" (UID: \"f48b6661-c056-4615-abbd-731a5a79a4b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b78f-68b67497b8-6w6sn"
Jul 29 20:08:31.452095 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:31.452003    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwpxj\" (UniqueName: \"kubernetes.io/projected/f48b6661-c056-4615-abbd-731a5a79a4b3-kube-api-access-jwpxj\") pod \"payload-processing-e2e-ipp-a-f0b78f-68b67497b8-6w6sn\" (UID: \"f48b6661-c056-4615-abbd-731a5a79a4b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b78f-68b67497b8-6w6sn"
Jul 29 20:08:31.452785 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:31.452761    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f81544c9-87bf-444e-8946-5f9f0d082f1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0b78f-6d94d85f88-jsgdq\" (UID: \"f81544c9-87bf-444e-8946-5f9f0d082f1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b78f-6d94d85f88-jsgdq"
Jul 29 20:08:31.467446 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:31.467425    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kvkr\" (UniqueName: \"kubernetes.io/projected/f81544c9-87bf-444e-8946-5f9f0d082f1b-kube-api-access-2kvkr\") pod \"payload-pre-processing-e2e-ipp-a-f0b78f-6d94d85f88-jsgdq\" (UID: \"f81544c9-87bf-444e-8946-5f9f0d082f1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b78f-6d94d85f88-jsgdq"
Jul 29 20:08:31.553193 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:31.553162    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f48b6661-c056-4615-abbd-731a5a79a4b3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0b78f-68b67497b8-6w6sn\" (UID: \"f48b6661-c056-4615-abbd-731a5a79a4b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b78f-68b67497b8-6w6sn"
Jul 29 20:08:31.553364 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:31.553239    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwpxj\" (UniqueName: \"kubernetes.io/projected/f48b6661-c056-4615-abbd-731a5a79a4b3-kube-api-access-jwpxj\") pod \"payload-processing-e2e-ipp-a-f0b78f-68b67497b8-6w6sn\" (UID: \"f48b6661-c056-4615-abbd-731a5a79a4b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b78f-68b67497b8-6w6sn"
Jul 29 20:08:31.554073 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:31.554049    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f48b6661-c056-4615-abbd-731a5a79a4b3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0b78f-68b67497b8-6w6sn\" (UID: \"f48b6661-c056-4615-abbd-731a5a79a4b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b78f-68b67497b8-6w6sn"
Jul 29 20:08:31.565270 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:31.565246    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwpxj\" (UniqueName: \"kubernetes.io/projected/f48b6661-c056-4615-abbd-731a5a79a4b3-kube-api-access-jwpxj\") pod \"payload-processing-e2e-ipp-a-f0b78f-68b67497b8-6w6sn\" (UID: \"f48b6661-c056-4615-abbd-731a5a79a4b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b78f-68b67497b8-6w6sn"
Jul 29 20:08:31.590249 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:31.590227    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b78f-6d94d85f88-jsgdq"
Jul 29 20:08:31.659496 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:31.659459    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b78f-68b67497b8-6w6sn"
Jul 29 20:08:31.744017 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:31.743990    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b78f-6d94d85f88-jsgdq"]
Jul 29 20:08:31.746681 ip-10-0-128-173 kubenswrapper[2557]: W0729 20:08:31.746648    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf81544c9_87bf_444e_8946_5f9f0d082f1b.slice/crio-be16f308527e2031d2e943e1caa303fe5cf2ac9890c5d376e1ed7111986710d8 WatchSource:0}: Error finding container be16f308527e2031d2e943e1caa303fe5cf2ac9890c5d376e1ed7111986710d8: Status 404 returned error can't find the container with id be16f308527e2031d2e943e1caa303fe5cf2ac9890c5d376e1ed7111986710d8
Jul 29 20:08:32.020428 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:32.020400    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0b78f-68b67497b8-6w6sn"]
Jul 29 20:08:32.022258 ip-10-0-128-173 kubenswrapper[2557]: W0729 20:08:32.022230    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf48b6661_c056_4615_abbd_731a5a79a4b3.slice/crio-fb39d1d40b236d3763d08f2f7a7712ac29695a78be7618497beebf887468e888 WatchSource:0}: Error finding container fb39d1d40b236d3763d08f2f7a7712ac29695a78be7618497beebf887468e888: Status 404 returned error can't find the container with id fb39d1d40b236d3763d08f2f7a7712ac29695a78be7618497beebf887468e888
Jul 29 20:08:32.105911 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:32.105882    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b78f-68b67497b8-6w6sn" event={"ID":"f48b6661-c056-4615-abbd-731a5a79a4b3","Type":"ContainerStarted","Data":"fb39d1d40b236d3763d08f2f7a7712ac29695a78be7618497beebf887468e888"}
Jul 29 20:08:32.107461 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:32.107433    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b78f-6d94d85f88-jsgdq" event={"ID":"f81544c9-87bf-444e-8946-5f9f0d082f1b","Type":"ContainerStarted","Data":"72429b653ac67d922a78333821d428d13dd8a6017ef269bf4cd58ce582d450ec"}
Jul 29 20:08:32.107602 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:32.107489    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b78f-6d94d85f88-jsgdq" event={"ID":"f81544c9-87bf-444e-8946-5f9f0d082f1b","Type":"ContainerStarted","Data":"be16f308527e2031d2e943e1caa303fe5cf2ac9890c5d376e1ed7111986710d8"}
Jul 29 20:08:32.107693 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:32.107607    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b78f-6d94d85f88-jsgdq"
Jul 29 20:08:32.109015 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:32.108989    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n" event={"ID":"5e61b2c1-44ec-4153-8ca9-a852b2882b08","Type":"ContainerStarted","Data":"5bbddadcff74d5e5c7a3e99a1ba160a0c2550b0d585306e8060b04c0bab266c3"}
Jul 29 20:08:32.109106 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:32.109077    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n"
Jul 29 20:08:32.132798 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:32.132757    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b78f-6d94d85f88-jsgdq" podStartSLOduration=1.1327436419999999 podStartE2EDuration="1.132743642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:08:32.129596871 +0000 UTC m=+3082.957095929" watchObservedRunningTime="2026-07-29 20:08:32.132743642 +0000 UTC m=+3082.960242700"
Jul 29 20:08:32.157247 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:32.157201    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n" podStartSLOduration=1.548384735 podStartE2EDuration="2.157185593s" totalImagesPullingTime="608.800858ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:08:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:08:32.155319153 +0000 UTC m=+3082.982818230" watchObservedRunningTime="2026-07-29 20:08:32.157185593 +0000 UTC m=+3082.984684689"
Jul 29 20:08:33.115590 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:33.115550    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b78f-68b67497b8-6w6sn" event={"ID":"f48b6661-c056-4615-abbd-731a5a79a4b3","Type":"ContainerStarted","Data":"22602dce0e944976eec7c674af3e609b61f944c102d90e8a6e1a778c65986323"}
Jul 29 20:08:33.116070 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:33.115691    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b78f-68b67497b8-6w6sn"
Jul 29 20:08:33.132588 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:33.132534    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b78f-68b67497b8-6w6sn" podStartSLOduration=2.132515887 podStartE2EDuration="2.132515887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:08:33.131335 +0000 UTC m=+3083.958834060" watchObservedRunningTime="2026-07-29 20:08:33.132515887 +0000 UTC m=+3083.960014947"
Jul 29 20:08:38.120719 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:38.120687    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n"
Jul 29 20:08:43.117984 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:43.117928    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b78f-6d94d85f88-jsgdq"
Jul 29 20:08:44.123116 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:44.123088    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b78f-68b67497b8-6w6sn"
Jul 29 20:08:59.116894 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:59.116860    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb"]
Jul 29 20:08:59.121028 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:59.121002    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb"
Jul 29 20:08:59.123680 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:59.123653    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-2d9c39-serving-cert\""
Jul 29 20:08:59.129854 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:59.129829    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb"]
Jul 29 20:08:59.200423 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:59.200384    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6335181c-9345-47f5-a85d-6af9dae251a5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb\" (UID: \"6335181c-9345-47f5-a85d-6af9dae251a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb"
Jul 29 20:08:59.200629 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:59.200446    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzdxd\" (UniqueName: \"kubernetes.io/projected/6335181c-9345-47f5-a85d-6af9dae251a5-kube-api-access-zzdxd\") pod \"maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb\" (UID: \"6335181c-9345-47f5-a85d-6af9dae251a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb"
Jul 29 20:08:59.200629 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:59.200533    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6335181c-9345-47f5-a85d-6af9dae251a5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb\" (UID: \"6335181c-9345-47f5-a85d-6af9dae251a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb"
Jul 29 20:08:59.302111 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:59.302068    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzdxd\" (UniqueName: \"kubernetes.io/projected/6335181c-9345-47f5-a85d-6af9dae251a5-kube-api-access-zzdxd\") pod \"maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb\" (UID: \"6335181c-9345-47f5-a85d-6af9dae251a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb"
Jul 29 20:08:59.302311 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:59.302129    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6335181c-9345-47f5-a85d-6af9dae251a5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb\" (UID: \"6335181c-9345-47f5-a85d-6af9dae251a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb"
Jul 29 20:08:59.302311 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:59.302217    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6335181c-9345-47f5-a85d-6af9dae251a5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb\" (UID: \"6335181c-9345-47f5-a85d-6af9dae251a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb"
Jul 29 20:08:59.302407 ip-10-0-128-173 kubenswrapper[2557]: E0729 20:08:59.302359    2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d9c39-serving-cert: secret "maas-api-e2e-ipp-b-2d9c39-serving-cert" not found
Jul 29 20:08:59.302446 ip-10-0-128-173 kubenswrapper[2557]: E0729 20:08:59.302430    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6335181c-9345-47f5-a85d-6af9dae251a5-maas-api-tls podName:6335181c-9345-47f5-a85d-6af9dae251a5 nodeName:}" failed. No retries permitted until 2026-07-29 20:08:59.802414478 +0000 UTC m=+3110.629913515 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6335181c-9345-47f5-a85d-6af9dae251a5-maas-api-tls") pod "maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb" (UID: "6335181c-9345-47f5-a85d-6af9dae251a5") : secret "maas-api-e2e-ipp-b-2d9c39-serving-cert" not found
Jul 29 20:08:59.302867 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:59.302846    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6335181c-9345-47f5-a85d-6af9dae251a5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb\" (UID: \"6335181c-9345-47f5-a85d-6af9dae251a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb"
Jul 29 20:08:59.311800 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:59.311776    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzdxd\" (UniqueName: \"kubernetes.io/projected/6335181c-9345-47f5-a85d-6af9dae251a5-kube-api-access-zzdxd\") pod \"maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb\" (UID: \"6335181c-9345-47f5-a85d-6af9dae251a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb"
Jul 29 20:08:59.808119 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:59.808081    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6335181c-9345-47f5-a85d-6af9dae251a5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb\" (UID: \"6335181c-9345-47f5-a85d-6af9dae251a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb"
Jul 29 20:08:59.810674 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:59.810645    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6335181c-9345-47f5-a85d-6af9dae251a5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb\" (UID: \"6335181c-9345-47f5-a85d-6af9dae251a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb"
Jul 29 20:08:59.810991 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:59.810954    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d9c39-6d969b5668-54lb5"]
Jul 29 20:08:59.814860 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:59.814842    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d9c39-6d969b5668-54lb5"
Jul 29 20:08:59.818306 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:59.818283    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2d9c39-dockercfg-qsnh5\""
Jul 29 20:08:59.818417 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:59.818291    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2d9c39\""
Jul 29 20:08:59.825284 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:59.825263    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d9c39-6d969b5668-54lb5"]
Jul 29 20:08:59.909442 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:59.909400    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z95fl\" (UniqueName: \"kubernetes.io/projected/137db5d6-330f-4daa-bb00-8753572a5cd9-kube-api-access-z95fl\") pod \"payload-processing-e2e-ipp-b-2d9c39-6d969b5668-54lb5\" (UID: \"137db5d6-330f-4daa-bb00-8753572a5cd9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d9c39-6d969b5668-54lb5"
Jul 29 20:08:59.909597 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:08:59.909464    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137db5d6-330f-4daa-bb00-8753572a5cd9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2d9c39-6d969b5668-54lb5\" (UID: \"137db5d6-330f-4daa-bb00-8753572a5cd9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d9c39-6d969b5668-54lb5"
Jul 29 20:09:00.010984 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:00.010930    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z95fl\" (UniqueName: \"kubernetes.io/projected/137db5d6-330f-4daa-bb00-8753572a5cd9-kube-api-access-z95fl\") pod \"payload-processing-e2e-ipp-b-2d9c39-6d969b5668-54lb5\" (UID: \"137db5d6-330f-4daa-bb00-8753572a5cd9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d9c39-6d969b5668-54lb5"
Jul 29 20:09:00.011168 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:00.011003    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137db5d6-330f-4daa-bb00-8753572a5cd9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2d9c39-6d969b5668-54lb5\" (UID: \"137db5d6-330f-4daa-bb00-8753572a5cd9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d9c39-6d969b5668-54lb5"
Jul 29 20:09:00.011708 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:00.011690    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137db5d6-330f-4daa-bb00-8753572a5cd9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2d9c39-6d969b5668-54lb5\" (UID: \"137db5d6-330f-4daa-bb00-8753572a5cd9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d9c39-6d969b5668-54lb5"
Jul 29 20:09:00.018704 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:00.018682    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z95fl\" (UniqueName: \"kubernetes.io/projected/137db5d6-330f-4daa-bb00-8753572a5cd9-kube-api-access-z95fl\") pod \"payload-processing-e2e-ipp-b-2d9c39-6d969b5668-54lb5\" (UID: \"137db5d6-330f-4daa-bb00-8753572a5cd9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d9c39-6d969b5668-54lb5"
Jul 29 20:09:00.035493 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:00.035469    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb"
Jul 29 20:09:00.133311 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:00.133280    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d9c39-6d969b5668-54lb5"
Jul 29 20:09:00.280706 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:00.280679    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d9c39-6d969b5668-54lb5"]
Jul 29 20:09:00.282224 ip-10-0-128-173 kubenswrapper[2557]: W0729 20:09:00.282187    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod137db5d6_330f_4daa_bb00_8753572a5cd9.slice/crio-f776d5d1c0f0489f75fc1517b74f7e3085f6eb2ae711ee0a32c6e65b28be031e WatchSource:0}: Error finding container f776d5d1c0f0489f75fc1517b74f7e3085f6eb2ae711ee0a32c6e65b28be031e: Status 404 returned error can't find the container with id f776d5d1c0f0489f75fc1517b74f7e3085f6eb2ae711ee0a32c6e65b28be031e
Jul 29 20:09:00.379289 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:00.379259    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb"]
Jul 29 20:09:00.380957 ip-10-0-128-173 kubenswrapper[2557]: W0729 20:09:00.380931    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6335181c_9345_47f5_a85d_6af9dae251a5.slice/crio-4038e8d8f7931c60407f761e132bb436acdcecf4f477427f32e12fff33b5d580 WatchSource:0}: Error finding container 4038e8d8f7931c60407f761e132bb436acdcecf4f477427f32e12fff33b5d580: Status 404 returned error can't find the container with id 4038e8d8f7931c60407f761e132bb436acdcecf4f477427f32e12fff33b5d580
Jul 29 20:09:01.260179 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:01.260140    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d9c39-6d969b5668-54lb5" event={"ID":"137db5d6-330f-4daa-bb00-8753572a5cd9","Type":"ContainerStarted","Data":"fb9fe089c79d7ff27204706fa26b8c853c66ffeceefc44ce54df58c8631495d9"}
Jul 29 20:09:01.260626 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:01.260206    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d9c39-6d969b5668-54lb5" event={"ID":"137db5d6-330f-4daa-bb00-8753572a5cd9","Type":"ContainerStarted","Data":"f776d5d1c0f0489f75fc1517b74f7e3085f6eb2ae711ee0a32c6e65b28be031e"}
Jul 29 20:09:01.260626 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:01.260235    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d9c39-6d969b5668-54lb5"
Jul 29 20:09:01.261875 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:01.261848    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb" event={"ID":"6335181c-9345-47f5-a85d-6af9dae251a5","Type":"ContainerStarted","Data":"f009896fdaee0c39ef96bd3b2e4bcc1f77ca4d928bb8916bc9dbaf034e19ce59"}
Jul 29 20:09:01.262030 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:01.261898    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb" event={"ID":"6335181c-9345-47f5-a85d-6af9dae251a5","Type":"ContainerStarted","Data":"4038e8d8f7931c60407f761e132bb436acdcecf4f477427f32e12fff33b5d580"}
Jul 29 20:09:01.262030 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:01.261927    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb"
Jul 29 20:09:01.279018 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:01.278953    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d9c39-6d969b5668-54lb5" podStartSLOduration=2.278936985 podStartE2EDuration="2.278936985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:08:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:09:01.275949673 +0000 UTC m=+3112.103448731" watchObservedRunningTime="2026-07-29 20:09:01.278936985 +0000 UTC m=+3112.106436042"
Jul 29 20:09:01.300167 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:01.300123    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb" podStartSLOduration=1.8503927999999998 podStartE2EDuration="2.300107395s" totalImagesPullingTime="449.714595ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:08:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:09:01.298743507 +0000 UTC m=+3112.126242578" watchObservedRunningTime="2026-07-29 20:09:01.300107395 +0000 UTC m=+3112.127606500"
Jul 29 20:09:07.273779 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:07.273744    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb"
Jul 29 20:09:12.269462 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:12.269433    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d9c39-6d969b5668-54lb5"
Jul 29 20:09:35.527240 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:35.527154    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-2d9c39-6d969b5668-54lb5_137db5d6-330f-4daa-bb00-8753572a5cd9/payload-processing/0.log"
Jul 29 20:09:48.837007 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:48.836947    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx"]
Jul 29 20:09:48.841262 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:48.841241    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx"
Jul 29 20:09:48.847476 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:48.847406    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-f0b78f\"/\"default-dockercfg-6h5jz\""
Jul 29 20:09:48.847476 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:48.847432    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-f0b78f\"/\"kube-root-ca.crt\""
Jul 29 20:09:48.847686 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:48.847529    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-f0b78f\"/\"openshift-service-ca.crt\""
Jul 29 20:09:48.847686 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:48.847433    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-f0b78f\"/\"ipp-route-f0b78f-kserve-self-signed-certs\""
Jul 29 20:09:48.851921 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:48.851899    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx"]
Jul 29 20:09:48.966906 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:48.966877    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66872169-2625-4828-9bf1-5ced2de250a8-tls-certs\") pod \"ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx\" (UID: \"66872169-2625-4828-9bf1-5ced2de250a8\") " pod="ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx"
Jul 29 20:09:48.966906 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:48.966911    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6jgj\" (UniqueName: \"kubernetes.io/projected/66872169-2625-4828-9bf1-5ced2de250a8-kube-api-access-b6jgj\") pod \"ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx\" (UID: \"66872169-2625-4828-9bf1-5ced2de250a8\") " pod="ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx"
Jul 29 20:09:48.967148 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:48.966927    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66872169-2625-4828-9bf1-5ced2de250a8-model-cache\") pod \"ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx\" (UID: \"66872169-2625-4828-9bf1-5ced2de250a8\") " pod="ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx"
Jul 29 20:09:48.967148 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:48.967010    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66872169-2625-4828-9bf1-5ced2de250a8-tmp-dir\") pod \"ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx\" (UID: \"66872169-2625-4828-9bf1-5ced2de250a8\") " pod="ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx"
Jul 29 20:09:48.967148 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:48.967053    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66872169-2625-4828-9bf1-5ced2de250a8-home\") pod \"ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx\" (UID: \"66872169-2625-4828-9bf1-5ced2de250a8\") " pod="ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx"
Jul 29 20:09:48.967148 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:48.967078    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66872169-2625-4828-9bf1-5ced2de250a8-dshm\") pod \"ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx\" (UID: \"66872169-2625-4828-9bf1-5ced2de250a8\") " pod="ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx"
Jul 29 20:09:49.067914 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:49.067883    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6jgj\" (UniqueName: \"kubernetes.io/projected/66872169-2625-4828-9bf1-5ced2de250a8-kube-api-access-b6jgj\") pod \"ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx\" (UID: \"66872169-2625-4828-9bf1-5ced2de250a8\") " pod="ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx"
Jul 29 20:09:49.067914 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:49.067913    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66872169-2625-4828-9bf1-5ced2de250a8-model-cache\") pod \"ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx\" (UID: \"66872169-2625-4828-9bf1-5ced2de250a8\") " pod="ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx"
Jul 29 20:09:49.068158 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:49.067945    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66872169-2625-4828-9bf1-5ced2de250a8-tmp-dir\") pod \"ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx\" (UID: \"66872169-2625-4828-9bf1-5ced2de250a8\") " pod="ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx"
Jul 29 20:09:49.068158 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:49.067991    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66872169-2625-4828-9bf1-5ced2de250a8-home\") pod \"ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx\" (UID: \"66872169-2625-4828-9bf1-5ced2de250a8\") " pod="ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx"
Jul 29 20:09:49.068158 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:49.068017    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66872169-2625-4828-9bf1-5ced2de250a8-dshm\") pod \"ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx\" (UID: \"66872169-2625-4828-9bf1-5ced2de250a8\") " pod="ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx"
Jul 29 20:09:49.068158 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:49.068074    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66872169-2625-4828-9bf1-5ced2de250a8-tls-certs\") pod \"ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx\" (UID: \"66872169-2625-4828-9bf1-5ced2de250a8\") " pod="ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx"
Jul 29 20:09:49.068414 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:49.068393    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66872169-2625-4828-9bf1-5ced2de250a8-model-cache\") pod \"ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx\" (UID: \"66872169-2625-4828-9bf1-5ced2de250a8\") " pod="ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx"
Jul 29 20:09:49.068496 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:49.068434    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66872169-2625-4828-9bf1-5ced2de250a8-tmp-dir\") pod \"ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx\" (UID: \"66872169-2625-4828-9bf1-5ced2de250a8\") " pod="ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx"
Jul 29 20:09:49.068496 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:49.068463    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66872169-2625-4828-9bf1-5ced2de250a8-home\") pod \"ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx\" (UID: \"66872169-2625-4828-9bf1-5ced2de250a8\") " pod="ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx"
Jul 29 20:09:49.070231 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:49.070209    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66872169-2625-4828-9bf1-5ced2de250a8-dshm\") pod \"ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx\" (UID: \"66872169-2625-4828-9bf1-5ced2de250a8\") " pod="ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx"
Jul 29 20:09:49.070398 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:49.070379    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66872169-2625-4828-9bf1-5ced2de250a8-tls-certs\") pod \"ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx\" (UID: \"66872169-2625-4828-9bf1-5ced2de250a8\") " pod="ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx"
Jul 29 20:09:49.076611 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:49.076588    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6jgj\" (UniqueName: \"kubernetes.io/projected/66872169-2625-4828-9bf1-5ced2de250a8-kube-api-access-b6jgj\") pod \"ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx\" (UID: \"66872169-2625-4828-9bf1-5ced2de250a8\") " pod="ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx"
Jul 29 20:09:49.152390 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:49.152354    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx"
Jul 29 20:09:49.287708 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:49.287576    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx"]
Jul 29 20:09:49.290118 ip-10-0-128-173 kubenswrapper[2557]: W0729 20:09:49.290087    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66872169_2625_4828_9bf1_5ced2de250a8.slice/crio-fe1ee7b45c10ce760ae58d315c05dab5f5a3599cebb0d73706d5983cbb48c5e3 WatchSource:0}: Error finding container fe1ee7b45c10ce760ae58d315c05dab5f5a3599cebb0d73706d5983cbb48c5e3: Status 404 returned error can't find the container with id fe1ee7b45c10ce760ae58d315c05dab5f5a3599cebb0d73706d5983cbb48c5e3
Jul 29 20:09:49.512605 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:49.512518    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx" event={"ID":"66872169-2625-4828-9bf1-5ced2de250a8","Type":"ContainerStarted","Data":"88bf73258176132d2b7e1894751303afa27f6f6fc814d8b109c587bc20e52ed0"}
Jul 29 20:09:49.512605 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:49.512558    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx" event={"ID":"66872169-2625-4828-9bf1-5ced2de250a8","Type":"ContainerStarted","Data":"fe1ee7b45c10ce760ae58d315c05dab5f5a3599cebb0d73706d5983cbb48c5e3"}
Jul 29 20:09:49.532690 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:49.532634    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx" podStartSLOduration=1.532617927 podStartE2EDuration="1.532617927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:09:49.530746901 +0000 UTC m=+3160.358245961" watchObservedRunningTime="2026-07-29 20:09:49.532617927 +0000 UTC m=+3160.360116981"
Jul 29 20:09:59.153144 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:59.153109    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx"
Jul 29 20:09:59.153525 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:59.153163    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx"
Jul 29 20:09:59.166113 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:59.166082    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx"
Jul 29 20:09:59.572039 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:09:59.571948    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx"
Jul 29 20:10:40.738248 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:10:40.738219    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-f0b78f-68b67497b8-6w6sn_f48b6661-c056-4615-abbd-731a5a79a4b3/payload-processing/0.log"
Jul 29 20:10:40.984395 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:10:40.984364    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-2d9c39-6d969b5668-54lb5_137db5d6-330f-4daa-bb00-8753572a5cd9/payload-processing/0.log"
Jul 29 20:10:46.864328 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:10:46.864289    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5d7141-697ddb6d84-wh7bt"]
Jul 29 20:10:46.868896 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:10:46.868867    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5d7141-697ddb6d84-wh7bt"
Jul 29 20:10:46.871706 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:10:46.871682    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-5d7141-dockercfg-d5slm\""
Jul 29 20:10:46.871927 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:10:46.871903    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-5d7141\""
Jul 29 20:10:46.878192 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:10:46.877804    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5d7141-697ddb6d84-wh7bt"]
Jul 29 20:10:47.007298 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:10:47.007266    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jwsb\" (UniqueName: \"kubernetes.io/projected/c4f65c63-6b6e-4bb6-9701-ee7e52851a82-kube-api-access-9jwsb\") pod \"payload-processing-e2e-ipp-cleanup-5d7141-697ddb6d84-wh7bt\" (UID: \"c4f65c63-6b6e-4bb6-9701-ee7e52851a82\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5d7141-697ddb6d84-wh7bt"
Jul 29 20:10:47.007461 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:10:47.007306    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f65c63-6b6e-4bb6-9701-ee7e52851a82-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-5d7141-697ddb6d84-wh7bt\" (UID: \"c4f65c63-6b6e-4bb6-9701-ee7e52851a82\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5d7141-697ddb6d84-wh7bt"
Jul 29 20:10:47.108505 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:10:47.108465    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jwsb\" (UniqueName: \"kubernetes.io/projected/c4f65c63-6b6e-4bb6-9701-ee7e52851a82-kube-api-access-9jwsb\") pod \"payload-processing-e2e-ipp-cleanup-5d7141-697ddb6d84-wh7bt\" (UID: \"c4f65c63-6b6e-4bb6-9701-ee7e52851a82\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5d7141-697ddb6d84-wh7bt"
Jul 29 20:10:47.108505 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:10:47.108505    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f65c63-6b6e-4bb6-9701-ee7e52851a82-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-5d7141-697ddb6d84-wh7bt\" (UID: \"c4f65c63-6b6e-4bb6-9701-ee7e52851a82\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5d7141-697ddb6d84-wh7bt"
Jul 29 20:10:47.109219 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:10:47.109200    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f65c63-6b6e-4bb6-9701-ee7e52851a82-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-5d7141-697ddb6d84-wh7bt\" (UID: \"c4f65c63-6b6e-4bb6-9701-ee7e52851a82\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5d7141-697ddb6d84-wh7bt"
Jul 29 20:10:47.117549 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:10:47.117493    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jwsb\" (UniqueName: \"kubernetes.io/projected/c4f65c63-6b6e-4bb6-9701-ee7e52851a82-kube-api-access-9jwsb\") pod \"payload-processing-e2e-ipp-cleanup-5d7141-697ddb6d84-wh7bt\" (UID: \"c4f65c63-6b6e-4bb6-9701-ee7e52851a82\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5d7141-697ddb6d84-wh7bt"
Jul 29 20:10:47.182012 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:10:47.181981    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5d7141-697ddb6d84-wh7bt"
Jul 29 20:10:47.333373 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:10:47.333347    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5d7141-697ddb6d84-wh7bt"]
Jul 29 20:10:47.334814 ip-10-0-128-173 kubenswrapper[2557]: W0729 20:10:47.334781    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4f65c63_6b6e_4bb6_9701_ee7e52851a82.slice/crio-5d20f2a582343770dff80c3d1e58ca7698a13ff7a6c132251231bb1c54057fb1 WatchSource:0}: Error finding container 5d20f2a582343770dff80c3d1e58ca7698a13ff7a6c132251231bb1c54057fb1: Status 404 returned error can't find the container with id 5d20f2a582343770dff80c3d1e58ca7698a13ff7a6c132251231bb1c54057fb1
Jul 29 20:10:47.793929 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:10:47.793886    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5d7141-697ddb6d84-wh7bt" event={"ID":"c4f65c63-6b6e-4bb6-9701-ee7e52851a82","Type":"ContainerStarted","Data":"18f3ee730cfe63a16ee561e16d1821a15f7d40d1f336d698ece3b01fc6cd6660"}
Jul 29 20:10:47.794134 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:10:47.793942    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5d7141-697ddb6d84-wh7bt" event={"ID":"c4f65c63-6b6e-4bb6-9701-ee7e52851a82","Type":"ContainerStarted","Data":"5d20f2a582343770dff80c3d1e58ca7698a13ff7a6c132251231bb1c54057fb1"}
Jul 29 20:10:47.794134 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:10:47.793983    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5d7141-697ddb6d84-wh7bt"
Jul 29 20:10:47.811735 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:10:47.811687    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5d7141-697ddb6d84-wh7bt" podStartSLOduration=1.811673257 podStartE2EDuration="1.811673257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:10:47.809908223 +0000 UTC m=+3218.637407292" watchObservedRunningTime="2026-07-29 20:10:47.811673257 +0000 UTC m=+3218.639172314"
Jul 29 20:10:58.801347 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:10:58.801312    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5d7141-697ddb6d84-wh7bt"
Jul 29 20:11:16.802774 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:16.802731    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5d7141-697ddb6d84-wh7bt"]
Jul 29 20:11:16.803401 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:16.803015    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5d7141-697ddb6d84-wh7bt" podUID="c4f65c63-6b6e-4bb6-9701-ee7e52851a82" containerName="payload-processing" containerID="cri-o://18f3ee730cfe63a16ee561e16d1821a15f7d40d1f336d698ece3b01fc6cd6660" gracePeriod=30
Jul 29 20:11:16.945135 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:16.945100    2557 generic.go:356] "Generic (PLEG): container finished" podID="c4f65c63-6b6e-4bb6-9701-ee7e52851a82" containerID="18f3ee730cfe63a16ee561e16d1821a15f7d40d1f336d698ece3b01fc6cd6660" exitCode=0
Jul 29 20:11:16.945300 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:16.945164    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5d7141-697ddb6d84-wh7bt" event={"ID":"c4f65c63-6b6e-4bb6-9701-ee7e52851a82","Type":"ContainerDied","Data":"18f3ee730cfe63a16ee561e16d1821a15f7d40d1f336d698ece3b01fc6cd6660"}
Jul 29 20:11:17.050274 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:17.050250    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5d7141-697ddb6d84-wh7bt"
Jul 29 20:11:17.195214 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:17.195179    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jwsb\" (UniqueName: \"kubernetes.io/projected/c4f65c63-6b6e-4bb6-9701-ee7e52851a82-kube-api-access-9jwsb\") pod \"c4f65c63-6b6e-4bb6-9701-ee7e52851a82\" (UID: \"c4f65c63-6b6e-4bb6-9701-ee7e52851a82\") "
Jul 29 20:11:17.195387 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:17.195237    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f65c63-6b6e-4bb6-9701-ee7e52851a82-plugins-config-volume\") pod \"c4f65c63-6b6e-4bb6-9701-ee7e52851a82\" (UID: \"c4f65c63-6b6e-4bb6-9701-ee7e52851a82\") "
Jul 29 20:11:17.195683 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:17.195664    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f65c63-6b6e-4bb6-9701-ee7e52851a82-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c4f65c63-6b6e-4bb6-9701-ee7e52851a82" (UID: "c4f65c63-6b6e-4bb6-9701-ee7e52851a82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:11:17.197364 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:17.197344    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f65c63-6b6e-4bb6-9701-ee7e52851a82-kube-api-access-9jwsb" (OuterVolumeSpecName: "kube-api-access-9jwsb") pod "c4f65c63-6b6e-4bb6-9701-ee7e52851a82" (UID: "c4f65c63-6b6e-4bb6-9701-ee7e52851a82"). InnerVolumeSpecName "kube-api-access-9jwsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:11:17.296233 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:17.296196    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jwsb\" (UniqueName: \"kubernetes.io/projected/c4f65c63-6b6e-4bb6-9701-ee7e52851a82-kube-api-access-9jwsb\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:11:17.296233 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:17.296228    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f65c63-6b6e-4bb6-9701-ee7e52851a82-plugins-config-volume\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:11:17.959843 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:17.959760    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5d7141-697ddb6d84-wh7bt" event={"ID":"c4f65c63-6b6e-4bb6-9701-ee7e52851a82","Type":"ContainerDied","Data":"5d20f2a582343770dff80c3d1e58ca7698a13ff7a6c132251231bb1c54057fb1"}
Jul 29 20:11:17.962347 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:17.959846    2557 scope.go:117] "RemoveContainer" containerID="18f3ee730cfe63a16ee561e16d1821a15f7d40d1f336d698ece3b01fc6cd6660"
Jul 29 20:11:17.962347 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:17.959773    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5d7141-697ddb6d84-wh7bt"
Jul 29 20:11:17.980840 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:17.980806    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5d7141-697ddb6d84-wh7bt"]
Jul 29 20:11:17.988326 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:17.988301    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5d7141-697ddb6d84-wh7bt"]
Jul 29 20:11:19.679452 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:19.679419    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4f65c63-6b6e-4bb6-9701-ee7e52851a82" path="/var/lib/kubelet/pods/c4f65c63-6b6e-4bb6-9701-ee7e52851a82/volumes"
Jul 29 20:11:42.422556 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:42.422522    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n"]
Jul 29 20:11:42.423171 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:42.422746    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n" podUID="5e61b2c1-44ec-4153-8ca9-a852b2882b08" containerName="maas-api" containerID="cri-o://5bbddadcff74d5e5c7a3e99a1ba160a0c2550b0d585306e8060b04c0bab266c3" gracePeriod=30
Jul 29 20:11:42.502719 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:42.502687    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0b78f-68b67497b8-6w6sn"]
Jul 29 20:11:42.502974 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:42.502930    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b78f-68b67497b8-6w6sn" podUID="f48b6661-c056-4615-abbd-731a5a79a4b3" containerName="payload-processing" containerID="cri-o://22602dce0e944976eec7c674af3e609b61f944c102d90e8a6e1a778c65986323" gracePeriod=30
Jul 29 20:11:42.513595 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:42.513568    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b78f-6d94d85f88-jsgdq"]
Jul 29 20:11:42.513803 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:42.513766    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b78f-6d94d85f88-jsgdq" podUID="f81544c9-87bf-444e-8946-5f9f0d082f1b" containerName="payload-pre-processing" containerID="cri-o://72429b653ac67d922a78333821d428d13dd8a6017ef269bf4cd58ce582d450ec" gracePeriod=30
Jul 29 20:11:42.769285 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:42.769256    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n"
Jul 29 20:11:42.801792 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:42.801763    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b78f-68b67497b8-6w6sn"
Jul 29 20:11:42.812142 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:42.812119    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b78f-6d94d85f88-jsgdq"
Jul 29 20:11:42.836274 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:42.836246    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e61b2c1-44ec-4153-8ca9-a852b2882b08-maas-api-tls\") pod \"5e61b2c1-44ec-4153-8ca9-a852b2882b08\" (UID: \"5e61b2c1-44ec-4153-8ca9-a852b2882b08\") "
Jul 29 20:11:42.836457 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:42.836290    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f48b6661-c056-4615-abbd-731a5a79a4b3-plugins-config-volume\") pod \"f48b6661-c056-4615-abbd-731a5a79a4b3\" (UID: \"f48b6661-c056-4615-abbd-731a5a79a4b3\") "
Jul 29 20:11:42.836457 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:42.836364    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwjdz\" (UniqueName: \"kubernetes.io/projected/5e61b2c1-44ec-4153-8ca9-a852b2882b08-kube-api-access-bwjdz\") pod \"5e61b2c1-44ec-4153-8ca9-a852b2882b08\" (UID: \"5e61b2c1-44ec-4153-8ca9-a852b2882b08\") "
Jul 29 20:11:42.836457 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:42.836409    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2kvkr\" (UniqueName: \"kubernetes.io/projected/f81544c9-87bf-444e-8946-5f9f0d082f1b-kube-api-access-2kvkr\") pod \"f81544c9-87bf-444e-8946-5f9f0d082f1b\" (UID: \"f81544c9-87bf-444e-8946-5f9f0d082f1b\") "
Jul 29 20:11:42.836457 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:42.836437    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e61b2c1-44ec-4153-8ca9-a852b2882b08-openshift-service-ca\") pod \"5e61b2c1-44ec-4153-8ca9-a852b2882b08\" (UID: \"5e61b2c1-44ec-4153-8ca9-a852b2882b08\") "
Jul 29 20:11:42.836687 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:42.836496    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwpxj\" (UniqueName: \"kubernetes.io/projected/f48b6661-c056-4615-abbd-731a5a79a4b3-kube-api-access-jwpxj\") pod \"f48b6661-c056-4615-abbd-731a5a79a4b3\" (UID: \"f48b6661-c056-4615-abbd-731a5a79a4b3\") "
Jul 29 20:11:42.836687 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:42.836532    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f81544c9-87bf-444e-8946-5f9f0d082f1b-plugins-config-volume\") pod \"f81544c9-87bf-444e-8946-5f9f0d082f1b\" (UID: \"f81544c9-87bf-444e-8946-5f9f0d082f1b\") "
Jul 29 20:11:42.837136 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:42.836809    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f48b6661-c056-4615-abbd-731a5a79a4b3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f48b6661-c056-4615-abbd-731a5a79a4b3" (UID: "f48b6661-c056-4615-abbd-731a5a79a4b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:11:42.837136 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:42.836903    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f48b6661-c056-4615-abbd-731a5a79a4b3-plugins-config-volume\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:11:42.837336 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:42.837184    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e61b2c1-44ec-4153-8ca9-a852b2882b08-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5e61b2c1-44ec-4153-8ca9-a852b2882b08" (UID: "5e61b2c1-44ec-4153-8ca9-a852b2882b08"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:11:42.837336 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:42.837188    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f81544c9-87bf-444e-8946-5f9f0d082f1b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f81544c9-87bf-444e-8946-5f9f0d082f1b" (UID: "f81544c9-87bf-444e-8946-5f9f0d082f1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:11:42.838909 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:42.838881    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e61b2c1-44ec-4153-8ca9-a852b2882b08-kube-api-access-bwjdz" (OuterVolumeSpecName: "kube-api-access-bwjdz") pod "5e61b2c1-44ec-4153-8ca9-a852b2882b08" (UID: "5e61b2c1-44ec-4153-8ca9-a852b2882b08"). InnerVolumeSpecName "kube-api-access-bwjdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:11:42.839105 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:42.838982    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e61b2c1-44ec-4153-8ca9-a852b2882b08-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5e61b2c1-44ec-4153-8ca9-a852b2882b08" (UID: "5e61b2c1-44ec-4153-8ca9-a852b2882b08"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:11:42.839105 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:42.838994    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f81544c9-87bf-444e-8946-5f9f0d082f1b-kube-api-access-2kvkr" (OuterVolumeSpecName: "kube-api-access-2kvkr") pod "f81544c9-87bf-444e-8946-5f9f0d082f1b" (UID: "f81544c9-87bf-444e-8946-5f9f0d082f1b"). InnerVolumeSpecName "kube-api-access-2kvkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:11:42.839306 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:42.839280    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f48b6661-c056-4615-abbd-731a5a79a4b3-kube-api-access-jwpxj" (OuterVolumeSpecName: "kube-api-access-jwpxj") pod "f48b6661-c056-4615-abbd-731a5a79a4b3" (UID: "f48b6661-c056-4615-abbd-731a5a79a4b3"). InnerVolumeSpecName "kube-api-access-jwpxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:11:42.937689 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:42.937602    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwjdz\" (UniqueName: \"kubernetes.io/projected/5e61b2c1-44ec-4153-8ca9-a852b2882b08-kube-api-access-bwjdz\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:11:42.937689 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:42.937633    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kvkr\" (UniqueName: \"kubernetes.io/projected/f81544c9-87bf-444e-8946-5f9f0d082f1b-kube-api-access-2kvkr\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:11:42.937689 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:42.937644    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e61b2c1-44ec-4153-8ca9-a852b2882b08-openshift-service-ca\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:11:42.937689 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:42.937653    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwpxj\" (UniqueName: \"kubernetes.io/projected/f48b6661-c056-4615-abbd-731a5a79a4b3-kube-api-access-jwpxj\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:11:42.937689 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:42.937661    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f81544c9-87bf-444e-8946-5f9f0d082f1b-plugins-config-volume\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:11:42.937689 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:42.937670    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e61b2c1-44ec-4153-8ca9-a852b2882b08-maas-api-tls\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:11:43.090257 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:43.090222    2557 generic.go:356] "Generic (PLEG): container finished" podID="5e61b2c1-44ec-4153-8ca9-a852b2882b08" containerID="5bbddadcff74d5e5c7a3e99a1ba160a0c2550b0d585306e8060b04c0bab266c3" exitCode=0
Jul 29 20:11:43.090451 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:43.090265    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n"
Jul 29 20:11:43.090451 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:43.090310    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n" event={"ID":"5e61b2c1-44ec-4153-8ca9-a852b2882b08","Type":"ContainerDied","Data":"5bbddadcff74d5e5c7a3e99a1ba160a0c2550b0d585306e8060b04c0bab266c3"}
Jul 29 20:11:43.090451 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:43.090373    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n" event={"ID":"5e61b2c1-44ec-4153-8ca9-a852b2882b08","Type":"ContainerDied","Data":"d2b095f01497fc04d7a13455ef2e78c3bf40b46d074eaf0c80cfe5edeacdca96"}
Jul 29 20:11:43.090451 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:43.090404    2557 scope.go:117] "RemoveContainer" containerID="5bbddadcff74d5e5c7a3e99a1ba160a0c2550b0d585306e8060b04c0bab266c3"
Jul 29 20:11:43.091818 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:43.091792    2557 generic.go:356] "Generic (PLEG): container finished" podID="f48b6661-c056-4615-abbd-731a5a79a4b3" containerID="22602dce0e944976eec7c674af3e609b61f944c102d90e8a6e1a778c65986323" exitCode=0
Jul 29 20:11:43.091946 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:43.091836    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b78f-68b67497b8-6w6sn"
Jul 29 20:11:43.091946 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:43.091875    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b78f-68b67497b8-6w6sn" event={"ID":"f48b6661-c056-4615-abbd-731a5a79a4b3","Type":"ContainerDied","Data":"22602dce0e944976eec7c674af3e609b61f944c102d90e8a6e1a778c65986323"}
Jul 29 20:11:43.091946 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:43.091912    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b78f-68b67497b8-6w6sn" event={"ID":"f48b6661-c056-4615-abbd-731a5a79a4b3","Type":"ContainerDied","Data":"fb39d1d40b236d3763d08f2f7a7712ac29695a78be7618497beebf887468e888"}
Jul 29 20:11:43.093389 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:43.093362    2557 generic.go:356] "Generic (PLEG): container finished" podID="f81544c9-87bf-444e-8946-5f9f0d082f1b" containerID="72429b653ac67d922a78333821d428d13dd8a6017ef269bf4cd58ce582d450ec" exitCode=0
Jul 29 20:11:43.093506 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:43.093409    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b78f-6d94d85f88-jsgdq"
Jul 29 20:11:43.093506 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:43.093431    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b78f-6d94d85f88-jsgdq" event={"ID":"f81544c9-87bf-444e-8946-5f9f0d082f1b","Type":"ContainerDied","Data":"72429b653ac67d922a78333821d428d13dd8a6017ef269bf4cd58ce582d450ec"}
Jul 29 20:11:43.093506 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:43.093468    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b78f-6d94d85f88-jsgdq" event={"ID":"f81544c9-87bf-444e-8946-5f9f0d082f1b","Type":"ContainerDied","Data":"be16f308527e2031d2e943e1caa303fe5cf2ac9890c5d376e1ed7111986710d8"}
Jul 29 20:11:43.100056 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:43.099987    2557 scope.go:117] "RemoveContainer" containerID="5bbddadcff74d5e5c7a3e99a1ba160a0c2550b0d585306e8060b04c0bab266c3"
Jul 29 20:11:43.100318 ip-10-0-128-173 kubenswrapper[2557]: E0729 20:11:43.100290    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bbddadcff74d5e5c7a3e99a1ba160a0c2550b0d585306e8060b04c0bab266c3\": container with ID starting with 5bbddadcff74d5e5c7a3e99a1ba160a0c2550b0d585306e8060b04c0bab266c3 not found: ID does not exist" containerID="5bbddadcff74d5e5c7a3e99a1ba160a0c2550b0d585306e8060b04c0bab266c3"
Jul 29 20:11:43.100398 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:43.100331    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bbddadcff74d5e5c7a3e99a1ba160a0c2550b0d585306e8060b04c0bab266c3"} err="failed to get container status \"5bbddadcff74d5e5c7a3e99a1ba160a0c2550b0d585306e8060b04c0bab266c3\": rpc error: code = NotFound desc = could not find container \"5bbddadcff74d5e5c7a3e99a1ba160a0c2550b0d585306e8060b04c0bab266c3\": container with ID starting with 5bbddadcff74d5e5c7a3e99a1ba160a0c2550b0d585306e8060b04c0bab266c3 not found: ID does not exist"
Jul 29 20:11:43.100398 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:43.100353    2557 scope.go:117] "RemoveContainer" containerID="22602dce0e944976eec7c674af3e609b61f944c102d90e8a6e1a778c65986323"
Jul 29 20:11:43.109819 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:43.109805    2557 scope.go:117] "RemoveContainer" containerID="22602dce0e944976eec7c674af3e609b61f944c102d90e8a6e1a778c65986323"
Jul 29 20:11:43.110112 ip-10-0-128-173 kubenswrapper[2557]: E0729 20:11:43.110091    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22602dce0e944976eec7c674af3e609b61f944c102d90e8a6e1a778c65986323\": container with ID starting with 22602dce0e944976eec7c674af3e609b61f944c102d90e8a6e1a778c65986323 not found: ID does not exist" containerID="22602dce0e944976eec7c674af3e609b61f944c102d90e8a6e1a778c65986323"
Jul 29 20:11:43.110166 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:43.110120    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22602dce0e944976eec7c674af3e609b61f944c102d90e8a6e1a778c65986323"} err="failed to get container status \"22602dce0e944976eec7c674af3e609b61f944c102d90e8a6e1a778c65986323\": rpc error: code = NotFound desc = could not find container \"22602dce0e944976eec7c674af3e609b61f944c102d90e8a6e1a778c65986323\": container with ID starting with 22602dce0e944976eec7c674af3e609b61f944c102d90e8a6e1a778c65986323 not found: ID does not exist"
Jul 29 20:11:43.110166 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:43.110138    2557 scope.go:117] "RemoveContainer" containerID="72429b653ac67d922a78333821d428d13dd8a6017ef269bf4cd58ce582d450ec"
Jul 29 20:11:43.117748 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:43.117730    2557 scope.go:117] "RemoveContainer" containerID="72429b653ac67d922a78333821d428d13dd8a6017ef269bf4cd58ce582d450ec"
Jul 29 20:11:43.118033 ip-10-0-128-173 kubenswrapper[2557]: E0729 20:11:43.118015    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72429b653ac67d922a78333821d428d13dd8a6017ef269bf4cd58ce582d450ec\": container with ID starting with 72429b653ac67d922a78333821d428d13dd8a6017ef269bf4cd58ce582d450ec not found: ID does not exist" containerID="72429b653ac67d922a78333821d428d13dd8a6017ef269bf4cd58ce582d450ec"
Jul 29 20:11:43.118080 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:43.118041    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72429b653ac67d922a78333821d428d13dd8a6017ef269bf4cd58ce582d450ec"} err="failed to get container status \"72429b653ac67d922a78333821d428d13dd8a6017ef269bf4cd58ce582d450ec\": rpc error: code = NotFound desc = could not find container \"72429b653ac67d922a78333821d428d13dd8a6017ef269bf4cd58ce582d450ec\": container with ID starting with 72429b653ac67d922a78333821d428d13dd8a6017ef269bf4cd58ce582d450ec not found: ID does not exist"
Jul 29 20:11:43.138243 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:43.138220    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0b78f-68b67497b8-6w6sn"]
Jul 29 20:11:43.150092 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:43.150072    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0b78f-68b67497b8-6w6sn"]
Jul 29 20:11:43.177819 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:43.177733    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n"]
Jul 29 20:11:43.183202 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:43.183158    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0b78f-56ddf4c888-8jq2n"]
Jul 29 20:11:43.199237 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:43.199171    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b78f-6d94d85f88-jsgdq"]
Jul 29 20:11:43.206697 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:43.206675    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b78f-6d94d85f88-jsgdq"]
Jul 29 20:11:43.678599 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:43.678557    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e61b2c1-44ec-4153-8ca9-a852b2882b08" path="/var/lib/kubelet/pods/5e61b2c1-44ec-4153-8ca9-a852b2882b08/volumes"
Jul 29 20:11:43.679156 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:43.679135    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f48b6661-c056-4615-abbd-731a5a79a4b3" path="/var/lib/kubelet/pods/f48b6661-c056-4615-abbd-731a5a79a4b3/volumes"
Jul 29 20:11:43.679534 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:43.679516    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f81544c9-87bf-444e-8946-5f9f0d082f1b" path="/var/lib/kubelet/pods/f81544c9-87bf-444e-8946-5f9f0d082f1b/volumes"
Jul 29 20:11:48.097299 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:11:48.097263    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx"]
Jul 29 20:12:03.116230 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:03.116170    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx" podUID="66872169-2625-4828-9bf1-5ced2de250a8" containerName="main" containerID="cri-o://88bf73258176132d2b7e1894751303afa27f6f6fc814d8b109c587bc20e52ed0" gracePeriod=45
Jul 29 20:12:03.369279 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:03.369224    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx"
Jul 29 20:12:03.537457 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:03.537422    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66872169-2625-4828-9bf1-5ced2de250a8-home\") pod \"66872169-2625-4828-9bf1-5ced2de250a8\" (UID: \"66872169-2625-4828-9bf1-5ced2de250a8\") "
Jul 29 20:12:03.537724 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:03.537533    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6jgj\" (UniqueName: \"kubernetes.io/projected/66872169-2625-4828-9bf1-5ced2de250a8-kube-api-access-b6jgj\") pod \"66872169-2625-4828-9bf1-5ced2de250a8\" (UID: \"66872169-2625-4828-9bf1-5ced2de250a8\") "
Jul 29 20:12:03.537724 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:03.537553    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66872169-2625-4828-9bf1-5ced2de250a8-dshm\") pod \"66872169-2625-4828-9bf1-5ced2de250a8\" (UID: \"66872169-2625-4828-9bf1-5ced2de250a8\") "
Jul 29 20:12:03.537724 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:03.537576    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66872169-2625-4828-9bf1-5ced2de250a8-tmp-dir\") pod \"66872169-2625-4828-9bf1-5ced2de250a8\" (UID: \"66872169-2625-4828-9bf1-5ced2de250a8\") "
Jul 29 20:12:03.537724 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:03.537613    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66872169-2625-4828-9bf1-5ced2de250a8-model-cache\") pod \"66872169-2625-4828-9bf1-5ced2de250a8\" (UID: \"66872169-2625-4828-9bf1-5ced2de250a8\") "
Jul 29 20:12:03.537724 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:03.537644    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66872169-2625-4828-9bf1-5ced2de250a8-tls-certs\") pod \"66872169-2625-4828-9bf1-5ced2de250a8\" (UID: \"66872169-2625-4828-9bf1-5ced2de250a8\") "
Jul 29 20:12:03.537724 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:03.537694    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66872169-2625-4828-9bf1-5ced2de250a8-home" (OuterVolumeSpecName: "home") pod "66872169-2625-4828-9bf1-5ced2de250a8" (UID: "66872169-2625-4828-9bf1-5ced2de250a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:12:03.538038 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:03.537864    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66872169-2625-4828-9bf1-5ced2de250a8-model-cache" (OuterVolumeSpecName: "model-cache") pod "66872169-2625-4828-9bf1-5ced2de250a8" (UID: "66872169-2625-4828-9bf1-5ced2de250a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:12:03.538038 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:03.537882    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66872169-2625-4828-9bf1-5ced2de250a8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "66872169-2625-4828-9bf1-5ced2de250a8" (UID: "66872169-2625-4828-9bf1-5ced2de250a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:12:03.538038 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:03.538010    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66872169-2625-4828-9bf1-5ced2de250a8-home\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:12:03.538038 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:03.538026    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66872169-2625-4828-9bf1-5ced2de250a8-tmp-dir\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:12:03.538038 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:03.538038    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66872169-2625-4828-9bf1-5ced2de250a8-model-cache\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:12:03.539846 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:03.539826    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66872169-2625-4828-9bf1-5ced2de250a8-dshm" (OuterVolumeSpecName: "dshm") pod "66872169-2625-4828-9bf1-5ced2de250a8" (UID: "66872169-2625-4828-9bf1-5ced2de250a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:12:03.539924 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:03.539886    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66872169-2625-4828-9bf1-5ced2de250a8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "66872169-2625-4828-9bf1-5ced2de250a8" (UID: "66872169-2625-4828-9bf1-5ced2de250a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:12:03.539984 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:03.539933    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66872169-2625-4828-9bf1-5ced2de250a8-kube-api-access-b6jgj" (OuterVolumeSpecName: "kube-api-access-b6jgj") pod "66872169-2625-4828-9bf1-5ced2de250a8" (UID: "66872169-2625-4828-9bf1-5ced2de250a8"). InnerVolumeSpecName "kube-api-access-b6jgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:12:03.638516 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:03.638485    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66872169-2625-4828-9bf1-5ced2de250a8-tls-certs\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:12:03.638516 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:03.638516    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6jgj\" (UniqueName: \"kubernetes.io/projected/66872169-2625-4828-9bf1-5ced2de250a8-kube-api-access-b6jgj\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:12:03.638695 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:03.638527    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66872169-2625-4828-9bf1-5ced2de250a8-dshm\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:12:04.202694 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:04.202657    2557 generic.go:356] "Generic (PLEG): container finished" podID="66872169-2625-4828-9bf1-5ced2de250a8" containerID="88bf73258176132d2b7e1894751303afa27f6f6fc814d8b109c587bc20e52ed0" exitCode=0
Jul 29 20:12:04.202694 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:04.202697    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx"
Jul 29 20:12:04.203333 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:04.202736    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx" event={"ID":"66872169-2625-4828-9bf1-5ced2de250a8","Type":"ContainerDied","Data":"88bf73258176132d2b7e1894751303afa27f6f6fc814d8b109c587bc20e52ed0"}
Jul 29 20:12:04.203333 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:04.202786    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx" event={"ID":"66872169-2625-4828-9bf1-5ced2de250a8","Type":"ContainerDied","Data":"fe1ee7b45c10ce760ae58d315c05dab5f5a3599cebb0d73706d5983cbb48c5e3"}
Jul 29 20:12:04.203333 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:04.202811    2557 scope.go:117] "RemoveContainer" containerID="88bf73258176132d2b7e1894751303afa27f6f6fc814d8b109c587bc20e52ed0"
Jul 29 20:12:04.211129 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:04.211107    2557 scope.go:117] "RemoveContainer" containerID="88bf73258176132d2b7e1894751303afa27f6f6fc814d8b109c587bc20e52ed0"
Jul 29 20:12:04.211378 ip-10-0-128-173 kubenswrapper[2557]: E0729 20:12:04.211358    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88bf73258176132d2b7e1894751303afa27f6f6fc814d8b109c587bc20e52ed0\": container with ID starting with 88bf73258176132d2b7e1894751303afa27f6f6fc814d8b109c587bc20e52ed0 not found: ID does not exist" containerID="88bf73258176132d2b7e1894751303afa27f6f6fc814d8b109c587bc20e52ed0"
Jul 29 20:12:04.211450 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:04.211391    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88bf73258176132d2b7e1894751303afa27f6f6fc814d8b109c587bc20e52ed0"} err="failed to get container status \"88bf73258176132d2b7e1894751303afa27f6f6fc814d8b109c587bc20e52ed0\": rpc error: code = NotFound desc = could not find container \"88bf73258176132d2b7e1894751303afa27f6f6fc814d8b109c587bc20e52ed0\": container with ID starting with 88bf73258176132d2b7e1894751303afa27f6f6fc814d8b109c587bc20e52ed0 not found: ID does not exist"
Jul 29 20:12:04.225888 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:04.225858    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx"]
Jul 29 20:12:04.228344 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:04.228323    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-f0b78f/ipp-route-f0b78f-kserve-64fdbd8cff-cjvzx"]
Jul 29 20:12:05.678854 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:05.678821    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66872169-2625-4828-9bf1-5ced2de250a8" path="/var/lib/kubelet/pods/66872169-2625-4828-9bf1-5ced2de250a8/volumes"
Jul 29 20:12:42.950904 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:42.950713    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb"]
Jul 29 20:12:42.951550 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:42.950998    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb" podUID="6335181c-9345-47f5-a85d-6af9dae251a5" containerName="maas-api" containerID="cri-o://f009896fdaee0c39ef96bd3b2e4bcc1f77ca4d928bb8916bc9dbaf034e19ce59" gracePeriod=30
Jul 29 20:12:43.018365 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.018329    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d9c39-6d969b5668-54lb5"]
Jul 29 20:12:43.018632 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.018583    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d9c39-6d969b5668-54lb5" podUID="137db5d6-330f-4daa-bb00-8753572a5cd9" containerName="payload-processing" containerID="cri-o://fb9fe089c79d7ff27204706fa26b8c853c66ffeceefc44ce54df58c8631495d9" gracePeriod=30
Jul 29 20:12:43.299850 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.299822    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb"
Jul 29 20:12:43.302747 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.302724    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d9c39-6d969b5668-54lb5"
Jul 29 20:12:43.383805 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.383777    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6335181c-9345-47f5-a85d-6af9dae251a5-openshift-service-ca\") pod \"6335181c-9345-47f5-a85d-6af9dae251a5\" (UID: \"6335181c-9345-47f5-a85d-6af9dae251a5\") "
Jul 29 20:12:43.383994 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.383844    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzdxd\" (UniqueName: \"kubernetes.io/projected/6335181c-9345-47f5-a85d-6af9dae251a5-kube-api-access-zzdxd\") pod \"6335181c-9345-47f5-a85d-6af9dae251a5\" (UID: \"6335181c-9345-47f5-a85d-6af9dae251a5\") "
Jul 29 20:12:43.383994 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.383922    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6335181c-9345-47f5-a85d-6af9dae251a5-maas-api-tls\") pod \"6335181c-9345-47f5-a85d-6af9dae251a5\" (UID: \"6335181c-9345-47f5-a85d-6af9dae251a5\") "
Jul 29 20:12:43.383994 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.383957    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z95fl\" (UniqueName: \"kubernetes.io/projected/137db5d6-330f-4daa-bb00-8753572a5cd9-kube-api-access-z95fl\") pod \"137db5d6-330f-4daa-bb00-8753572a5cd9\" (UID: \"137db5d6-330f-4daa-bb00-8753572a5cd9\") "
Jul 29 20:12:43.383994 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.383991    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137db5d6-330f-4daa-bb00-8753572a5cd9-plugins-config-volume\") pod \"137db5d6-330f-4daa-bb00-8753572a5cd9\" (UID: \"137db5d6-330f-4daa-bb00-8753572a5cd9\") "
Jul 29 20:12:43.384375 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.384339    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6335181c-9345-47f5-a85d-6af9dae251a5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6335181c-9345-47f5-a85d-6af9dae251a5" (UID: "6335181c-9345-47f5-a85d-6af9dae251a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:12:43.384511 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.384420    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137db5d6-330f-4daa-bb00-8753572a5cd9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "137db5d6-330f-4daa-bb00-8753572a5cd9" (UID: "137db5d6-330f-4daa-bb00-8753572a5cd9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:12:43.386119 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.386090    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6335181c-9345-47f5-a85d-6af9dae251a5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6335181c-9345-47f5-a85d-6af9dae251a5" (UID: "6335181c-9345-47f5-a85d-6af9dae251a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:12:43.386234 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.386135    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137db5d6-330f-4daa-bb00-8753572a5cd9-kube-api-access-z95fl" (OuterVolumeSpecName: "kube-api-access-z95fl") pod "137db5d6-330f-4daa-bb00-8753572a5cd9" (UID: "137db5d6-330f-4daa-bb00-8753572a5cd9"). InnerVolumeSpecName "kube-api-access-z95fl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:12:43.386234 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.386135    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6335181c-9345-47f5-a85d-6af9dae251a5-kube-api-access-zzdxd" (OuterVolumeSpecName: "kube-api-access-zzdxd") pod "6335181c-9345-47f5-a85d-6af9dae251a5" (UID: "6335181c-9345-47f5-a85d-6af9dae251a5"). InnerVolumeSpecName "kube-api-access-zzdxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:12:43.394160 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.394135    2557 generic.go:356] "Generic (PLEG): container finished" podID="137db5d6-330f-4daa-bb00-8753572a5cd9" containerID="fb9fe089c79d7ff27204706fa26b8c853c66ffeceefc44ce54df58c8631495d9" exitCode=0
Jul 29 20:12:43.394292 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.394195    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d9c39-6d969b5668-54lb5"
Jul 29 20:12:43.394292 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.394231    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d9c39-6d969b5668-54lb5" event={"ID":"137db5d6-330f-4daa-bb00-8753572a5cd9","Type":"ContainerDied","Data":"fb9fe089c79d7ff27204706fa26b8c853c66ffeceefc44ce54df58c8631495d9"}
Jul 29 20:12:43.394292 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.394289    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d9c39-6d969b5668-54lb5" event={"ID":"137db5d6-330f-4daa-bb00-8753572a5cd9","Type":"ContainerDied","Data":"f776d5d1c0f0489f75fc1517b74f7e3085f6eb2ae711ee0a32c6e65b28be031e"}
Jul 29 20:12:43.394473 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.394316    2557 scope.go:117] "RemoveContainer" containerID="fb9fe089c79d7ff27204706fa26b8c853c66ffeceefc44ce54df58c8631495d9"
Jul 29 20:12:43.395699 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.395673    2557 generic.go:356] "Generic (PLEG): container finished" podID="6335181c-9345-47f5-a85d-6af9dae251a5" containerID="f009896fdaee0c39ef96bd3b2e4bcc1f77ca4d928bb8916bc9dbaf034e19ce59" exitCode=0
Jul 29 20:12:43.395816 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.395717    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb"
Jul 29 20:12:43.395816 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.395765    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb" event={"ID":"6335181c-9345-47f5-a85d-6af9dae251a5","Type":"ContainerDied","Data":"f009896fdaee0c39ef96bd3b2e4bcc1f77ca4d928bb8916bc9dbaf034e19ce59"}
Jul 29 20:12:43.395816 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.395799    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb" event={"ID":"6335181c-9345-47f5-a85d-6af9dae251a5","Type":"ContainerDied","Data":"4038e8d8f7931c60407f761e132bb436acdcecf4f477427f32e12fff33b5d580"}
Jul 29 20:12:43.403345 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.403325    2557 scope.go:117] "RemoveContainer" containerID="fb9fe089c79d7ff27204706fa26b8c853c66ffeceefc44ce54df58c8631495d9"
Jul 29 20:12:43.403755 ip-10-0-128-173 kubenswrapper[2557]: E0729 20:12:43.403655    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb9fe089c79d7ff27204706fa26b8c853c66ffeceefc44ce54df58c8631495d9\": container with ID starting with fb9fe089c79d7ff27204706fa26b8c853c66ffeceefc44ce54df58c8631495d9 not found: ID does not exist" containerID="fb9fe089c79d7ff27204706fa26b8c853c66ffeceefc44ce54df58c8631495d9"
Jul 29 20:12:43.403755 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.403686    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb9fe089c79d7ff27204706fa26b8c853c66ffeceefc44ce54df58c8631495d9"} err="failed to get container status \"fb9fe089c79d7ff27204706fa26b8c853c66ffeceefc44ce54df58c8631495d9\": rpc error: code = NotFound desc = could not find container \"fb9fe089c79d7ff27204706fa26b8c853c66ffeceefc44ce54df58c8631495d9\": container with ID starting with fb9fe089c79d7ff27204706fa26b8c853c66ffeceefc44ce54df58c8631495d9 not found: ID does not exist"
Jul 29 20:12:43.403755 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.403709    2557 scope.go:117] "RemoveContainer" containerID="f009896fdaee0c39ef96bd3b2e4bcc1f77ca4d928bb8916bc9dbaf034e19ce59"
Jul 29 20:12:43.413085 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.413066    2557 scope.go:117] "RemoveContainer" containerID="f009896fdaee0c39ef96bd3b2e4bcc1f77ca4d928bb8916bc9dbaf034e19ce59"
Jul 29 20:12:43.413333 ip-10-0-128-173 kubenswrapper[2557]: E0729 20:12:43.413315    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f009896fdaee0c39ef96bd3b2e4bcc1f77ca4d928bb8916bc9dbaf034e19ce59\": container with ID starting with f009896fdaee0c39ef96bd3b2e4bcc1f77ca4d928bb8916bc9dbaf034e19ce59 not found: ID does not exist" containerID="f009896fdaee0c39ef96bd3b2e4bcc1f77ca4d928bb8916bc9dbaf034e19ce59"
Jul 29 20:12:43.413398 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.413340    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f009896fdaee0c39ef96bd3b2e4bcc1f77ca4d928bb8916bc9dbaf034e19ce59"} err="failed to get container status \"f009896fdaee0c39ef96bd3b2e4bcc1f77ca4d928bb8916bc9dbaf034e19ce59\": rpc error: code = NotFound desc = could not find container \"f009896fdaee0c39ef96bd3b2e4bcc1f77ca4d928bb8916bc9dbaf034e19ce59\": container with ID starting with f009896fdaee0c39ef96bd3b2e4bcc1f77ca4d928bb8916bc9dbaf034e19ce59 not found: ID does not exist"
Jul 29 20:12:43.420588 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.420566    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d9c39-6d969b5668-54lb5"]
Jul 29 20:12:43.427490 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.427469    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d9c39-6d969b5668-54lb5"]
Jul 29 20:12:43.442406 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.442380    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb"]
Jul 29 20:12:43.445859 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.445839    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d9c39-d6bd7c54-xprmb"]
Jul 29 20:12:43.485757 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.485689    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6335181c-9345-47f5-a85d-6af9dae251a5-openshift-service-ca\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:12:43.485757 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.485715    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzdxd\" (UniqueName: \"kubernetes.io/projected/6335181c-9345-47f5-a85d-6af9dae251a5-kube-api-access-zzdxd\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:12:43.485757 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.485726    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6335181c-9345-47f5-a85d-6af9dae251a5-maas-api-tls\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:12:43.485757 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.485738    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z95fl\" (UniqueName: \"kubernetes.io/projected/137db5d6-330f-4daa-bb00-8753572a5cd9-kube-api-access-z95fl\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:12:43.485757 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.485746    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137db5d6-330f-4daa-bb00-8753572a5cd9-plugins-config-volume\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:12:43.679512 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.679477    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="137db5d6-330f-4daa-bb00-8753572a5cd9" path="/var/lib/kubelet/pods/137db5d6-330f-4daa-bb00-8753572a5cd9/volumes"
Jul 29 20:12:43.679903 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:43.679888    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6335181c-9345-47f5-a85d-6af9dae251a5" path="/var/lib/kubelet/pods/6335181c-9345-47f5-a85d-6af9dae251a5/volumes"
Jul 29 20:12:50.229190 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:50.229149    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-641e39-7vx9s"]
Jul 29 20:12:50.229654 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:50.229585    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f81544c9-87bf-444e-8946-5f9f0d082f1b" containerName="payload-pre-processing"
Jul 29 20:12:50.229654 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:50.229598    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f81544c9-87bf-444e-8946-5f9f0d082f1b" containerName="payload-pre-processing"
Jul 29 20:12:50.229654 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:50.229610    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e61b2c1-44ec-4153-8ca9-a852b2882b08" containerName="maas-api"
Jul 29 20:12:50.229654 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:50.229616    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e61b2c1-44ec-4153-8ca9-a852b2882b08" containerName="maas-api"
Jul 29 20:12:50.229654 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:50.229629    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="137db5d6-330f-4daa-bb00-8753572a5cd9" containerName="payload-processing"
Jul 29 20:12:50.229654 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:50.229638    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="137db5d6-330f-4daa-bb00-8753572a5cd9" containerName="payload-processing"
Jul 29 20:12:50.229654 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:50.229650    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66872169-2625-4828-9bf1-5ced2de250a8" containerName="main"
Jul 29 20:12:50.229654 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:50.229656    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="66872169-2625-4828-9bf1-5ced2de250a8" containerName="main"
Jul 29 20:12:50.229904 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:50.229662    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4f65c63-6b6e-4bb6-9701-ee7e52851a82" containerName="payload-processing"
Jul 29 20:12:50.229904 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:50.229668    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4f65c63-6b6e-4bb6-9701-ee7e52851a82" containerName="payload-processing"
Jul 29 20:12:50.229904 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:50.229676    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f48b6661-c056-4615-abbd-731a5a79a4b3" containerName="payload-processing"
Jul 29 20:12:50.229904 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:50.229681    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f48b6661-c056-4615-abbd-731a5a79a4b3" containerName="payload-processing"
Jul 29 20:12:50.229904 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:50.229694    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6335181c-9345-47f5-a85d-6af9dae251a5" containerName="maas-api"
Jul 29 20:12:50.229904 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:50.229699    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6335181c-9345-47f5-a85d-6af9dae251a5" containerName="maas-api"
Jul 29 20:12:50.229904 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:50.229758    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6335181c-9345-47f5-a85d-6af9dae251a5" containerName="maas-api"
Jul 29 20:12:50.229904 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:50.229766    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e61b2c1-44ec-4153-8ca9-a852b2882b08" containerName="maas-api"
Jul 29 20:12:50.229904 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:50.229772    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="66872169-2625-4828-9bf1-5ced2de250a8" containerName="main"
Jul 29 20:12:50.229904 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:50.229776    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4f65c63-6b6e-4bb6-9701-ee7e52851a82" containerName="payload-processing"
Jul 29 20:12:50.229904 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:50.229784    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f48b6661-c056-4615-abbd-731a5a79a4b3" containerName="payload-processing"
Jul 29 20:12:50.229904 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:50.229796    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f81544c9-87bf-444e-8946-5f9f0d082f1b" containerName="payload-pre-processing"
Jul 29 20:12:50.229904 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:50.229803    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="137db5d6-330f-4daa-bb00-8753572a5cd9" containerName="payload-processing"
Jul 29 20:12:50.235321 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:50.235302    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-641e39-7vx9s"
Jul 29 20:12:50.238710 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:50.238682    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jsq2r\""
Jul 29 20:12:50.240949 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:50.240920    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-641e39-7vx9s"]
Jul 29 20:12:50.348586 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:50.348548    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d07dba-bf2e-49bc-9a0f-73208e5c2221-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-641e39-7vx9s\" (UID: \"41d07dba-bf2e-49bc-9a0f-73208e5c2221\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-641e39-7vx9s"
Jul 29 20:12:50.449814 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:50.449779    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d07dba-bf2e-49bc-9a0f-73208e5c2221-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-641e39-7vx9s\" (UID: \"41d07dba-bf2e-49bc-9a0f-73208e5c2221\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-641e39-7vx9s"
Jul 29 20:12:50.450401 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:50.450382    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d07dba-bf2e-49bc-9a0f-73208e5c2221-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-641e39-7vx9s\" (UID: \"41d07dba-bf2e-49bc-9a0f-73208e5c2221\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-641e39-7vx9s"
Jul 29 20:12:50.548130 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:50.548047    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-641e39-7vx9s"
Jul 29 20:12:50.686131 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:50.686101    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-641e39-7vx9s"]
Jul 29 20:12:50.688708 ip-10-0-128-173 kubenswrapper[2557]: W0729 20:12:50.688680    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d07dba_bf2e_49bc_9a0f_73208e5c2221.slice/crio-4fe7c0d26ea8ec1e62aaab8b58086cec1271eeca05936969bd39212f01a68a73 WatchSource:0}: Error finding container 4fe7c0d26ea8ec1e62aaab8b58086cec1271eeca05936969bd39212f01a68a73: Status 404 returned error can't find the container with id 4fe7c0d26ea8ec1e62aaab8b58086cec1271eeca05936969bd39212f01a68a73
Jul 29 20:12:51.437463 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:51.437430    2557 generic.go:356] "Generic (PLEG): container finished" podID="41d07dba-bf2e-49bc-9a0f-73208e5c2221" containerID="5c14a592de450db3b93a66f8139c3d55f57da3c1138b9850b027f7f5ea0347aa" exitCode=0
Jul 29 20:12:51.437872 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:51.437517    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-641e39-7vx9s" event={"ID":"41d07dba-bf2e-49bc-9a0f-73208e5c2221","Type":"ContainerDied","Data":"5c14a592de450db3b93a66f8139c3d55f57da3c1138b9850b027f7f5ea0347aa"}
Jul 29 20:12:51.437872 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:51.437585    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-641e39-7vx9s" event={"ID":"41d07dba-bf2e-49bc-9a0f-73208e5c2221","Type":"ContainerStarted","Data":"4fe7c0d26ea8ec1e62aaab8b58086cec1271eeca05936969bd39212f01a68a73"}
Jul 29 20:12:52.581367 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:52.581344    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-641e39-7vx9s"
Jul 29 20:12:52.670888 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:52.670857    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d07dba-bf2e-49bc-9a0f-73208e5c2221-maas-api-service-ca\") pod \"41d07dba-bf2e-49bc-9a0f-73208e5c2221\" (UID: \"41d07dba-bf2e-49bc-9a0f-73208e5c2221\") "
Jul 29 20:12:52.671231 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:52.671209    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d07dba-bf2e-49bc-9a0f-73208e5c2221-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "41d07dba-bf2e-49bc-9a0f-73208e5c2221" (UID: "41d07dba-bf2e-49bc-9a0f-73208e5c2221"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:12:52.772206 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:52.772101    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d07dba-bf2e-49bc-9a0f-73208e5c2221-maas-api-service-ca\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:12:53.448445 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:53.448409    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-641e39-7vx9s"
Jul 29 20:12:53.448618 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:53.448447    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-641e39-7vx9s" event={"ID":"41d07dba-bf2e-49bc-9a0f-73208e5c2221","Type":"ContainerDied","Data":"4fe7c0d26ea8ec1e62aaab8b58086cec1271eeca05936969bd39212f01a68a73"}
Jul 29 20:12:53.448618 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:12:53.448492    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fe7c0d26ea8ec1e62aaab8b58086cec1271eeca05936969bd39212f01a68a73"
Jul 29 20:13:00.914511 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:00.914468    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2"]
Jul 29 20:13:00.915024 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:00.914777    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2" podUID="e4769186-0b54-46b4-bb07-ecc3a5b92e94" containerName="maas-api" containerID="cri-o://03beaba7553bd230ae735346dcc46f4b84d083f7640929e9ff19b0ed34b04e52" gracePeriod=30
Jul 29 20:13:00.996371 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:00.996337    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-641e39-5cb6cb867d-c4fdv"]
Jul 29 20:13:00.996682 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:00.996616    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-641e39-5cb6cb867d-c4fdv" podUID="a31200ff-316c-439b-821d-0a6f939ac40e" containerName="payload-processing" containerID="cri-o://01b8ec7cf67373ed1e3c8ec39b473623ebe5118f1737bcbf74f78285eb6ed0f1" gracePeriod=30
Jul 29 20:13:01.009765 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.009577    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-641e39-787b6966b8-rwjd7"]
Jul 29 20:13:01.010150 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.010117    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-641e39-787b6966b8-rwjd7" podUID="e2e3a032-66b8-4274-9aa5-3da2c46f6835" containerName="payload-pre-processing" containerID="cri-o://b2f1a5cc278921296e20d794e813e70f4fab6f855e8b991a633ded26d1a209d5" gracePeriod=30
Jul 29 20:13:01.187820 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.187785    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-641e39-7vx9s"]
Jul 29 20:13:01.194000 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.193954    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-641e39-7vx9s"]
Jul 29 20:13:01.256683 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.256650    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2"
Jul 29 20:13:01.273454 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.273428    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-641e39-5cb6cb867d-c4fdv"
Jul 29 20:13:01.286912 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.286887    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-641e39-787b6966b8-rwjd7"
Jul 29 20:13:01.455017 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.454901    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e3a032-66b8-4274-9aa5-3da2c46f6835-plugins-config-volume\") pod \"e2e3a032-66b8-4274-9aa5-3da2c46f6835\" (UID: \"e2e3a032-66b8-4274-9aa5-3da2c46f6835\") "
Jul 29 20:13:01.455017 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.454951    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a31200ff-316c-439b-821d-0a6f939ac40e-plugins-config-volume\") pod \"a31200ff-316c-439b-821d-0a6f939ac40e\" (UID: \"a31200ff-316c-439b-821d-0a6f939ac40e\") "
Jul 29 20:13:01.455017 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.455002    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2299\" (UniqueName: \"kubernetes.io/projected/e4769186-0b54-46b4-bb07-ecc3a5b92e94-kube-api-access-v2299\") pod \"e4769186-0b54-46b4-bb07-ecc3a5b92e94\" (UID: \"e4769186-0b54-46b4-bb07-ecc3a5b92e94\") "
Jul 29 20:13:01.455332 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.455030    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4769186-0b54-46b4-bb07-ecc3a5b92e94-openshift-service-ca\") pod \"e4769186-0b54-46b4-bb07-ecc3a5b92e94\" (UID: \"e4769186-0b54-46b4-bb07-ecc3a5b92e94\") "
Jul 29 20:13:01.455332 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.455059    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnkqp\" (UniqueName: \"kubernetes.io/projected/e2e3a032-66b8-4274-9aa5-3da2c46f6835-kube-api-access-hnkqp\") pod \"e2e3a032-66b8-4274-9aa5-3da2c46f6835\" (UID: \"e2e3a032-66b8-4274-9aa5-3da2c46f6835\") "
Jul 29 20:13:01.455332 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.455097    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgc6z\" (UniqueName: \"kubernetes.io/projected/a31200ff-316c-439b-821d-0a6f939ac40e-kube-api-access-vgc6z\") pod \"a31200ff-316c-439b-821d-0a6f939ac40e\" (UID: \"a31200ff-316c-439b-821d-0a6f939ac40e\") "
Jul 29 20:13:01.455332 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.455203    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4769186-0b54-46b4-bb07-ecc3a5b92e94-maas-api-tls\") pod \"e4769186-0b54-46b4-bb07-ecc3a5b92e94\" (UID: \"e4769186-0b54-46b4-bb07-ecc3a5b92e94\") "
Jul 29 20:13:01.455536 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.455413    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2e3a032-66b8-4274-9aa5-3da2c46f6835-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e2e3a032-66b8-4274-9aa5-3da2c46f6835" (UID: "e2e3a032-66b8-4274-9aa5-3da2c46f6835"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:13:01.455536 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.455416    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4769186-0b54-46b4-bb07-ecc3a5b92e94-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e4769186-0b54-46b4-bb07-ecc3a5b92e94" (UID: "e4769186-0b54-46b4-bb07-ecc3a5b92e94"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:13:01.455536 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.455491    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a31200ff-316c-439b-821d-0a6f939ac40e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a31200ff-316c-439b-821d-0a6f939ac40e" (UID: "a31200ff-316c-439b-821d-0a6f939ac40e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:13:01.455677 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.455635    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e3a032-66b8-4274-9aa5-3da2c46f6835-plugins-config-volume\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:13:01.455677 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.455654    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a31200ff-316c-439b-821d-0a6f939ac40e-plugins-config-volume\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:13:01.455677 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.455669    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4769186-0b54-46b4-bb07-ecc3a5b92e94-openshift-service-ca\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:13:01.457319 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.457294    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4769186-0b54-46b4-bb07-ecc3a5b92e94-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e4769186-0b54-46b4-bb07-ecc3a5b92e94" (UID: "e4769186-0b54-46b4-bb07-ecc3a5b92e94"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:13:01.457491 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.457469    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4769186-0b54-46b4-bb07-ecc3a5b92e94-kube-api-access-v2299" (OuterVolumeSpecName: "kube-api-access-v2299") pod "e4769186-0b54-46b4-bb07-ecc3a5b92e94" (UID: "e4769186-0b54-46b4-bb07-ecc3a5b92e94"). InnerVolumeSpecName "kube-api-access-v2299". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:13:01.457745 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.457716    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2e3a032-66b8-4274-9aa5-3da2c46f6835-kube-api-access-hnkqp" (OuterVolumeSpecName: "kube-api-access-hnkqp") pod "e2e3a032-66b8-4274-9aa5-3da2c46f6835" (UID: "e2e3a032-66b8-4274-9aa5-3da2c46f6835"). InnerVolumeSpecName "kube-api-access-hnkqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:13:01.457745 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.457722    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a31200ff-316c-439b-821d-0a6f939ac40e-kube-api-access-vgc6z" (OuterVolumeSpecName: "kube-api-access-vgc6z") pod "a31200ff-316c-439b-821d-0a6f939ac40e" (UID: "a31200ff-316c-439b-821d-0a6f939ac40e"). InnerVolumeSpecName "kube-api-access-vgc6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:13:01.489727 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.489693    2557 generic.go:356] "Generic (PLEG): container finished" podID="a31200ff-316c-439b-821d-0a6f939ac40e" containerID="01b8ec7cf67373ed1e3c8ec39b473623ebe5118f1737bcbf74f78285eb6ed0f1" exitCode=0
Jul 29 20:13:01.489907 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.489747    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-641e39-5cb6cb867d-c4fdv"
Jul 29 20:13:01.489907 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.489771    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-641e39-5cb6cb867d-c4fdv" event={"ID":"a31200ff-316c-439b-821d-0a6f939ac40e","Type":"ContainerDied","Data":"01b8ec7cf67373ed1e3c8ec39b473623ebe5118f1737bcbf74f78285eb6ed0f1"}
Jul 29 20:13:01.489907 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.489836    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-641e39-5cb6cb867d-c4fdv" event={"ID":"a31200ff-316c-439b-821d-0a6f939ac40e","Type":"ContainerDied","Data":"184dc1924a0e461dd2f3425d2db63bb98f8b127c0010e5c59a21ac64e6d56196"}
Jul 29 20:13:01.489907 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.489867    2557 scope.go:117] "RemoveContainer" containerID="01b8ec7cf67373ed1e3c8ec39b473623ebe5118f1737bcbf74f78285eb6ed0f1"
Jul 29 20:13:01.491129 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.491106    2557 generic.go:356] "Generic (PLEG): container finished" podID="e4769186-0b54-46b4-bb07-ecc3a5b92e94" containerID="03beaba7553bd230ae735346dcc46f4b84d083f7640929e9ff19b0ed34b04e52" exitCode=0
Jul 29 20:13:01.491240 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.491174    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2"
Jul 29 20:13:01.491240 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.491197    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2" event={"ID":"e4769186-0b54-46b4-bb07-ecc3a5b92e94","Type":"ContainerDied","Data":"03beaba7553bd230ae735346dcc46f4b84d083f7640929e9ff19b0ed34b04e52"}
Jul 29 20:13:01.491362 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.491263    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2" event={"ID":"e4769186-0b54-46b4-bb07-ecc3a5b92e94","Type":"ContainerDied","Data":"09d4817fe849c03a0cc53df18d136dcc9e998ad737f3249bd2a361580ea479a5"}
Jul 29 20:13:01.492569 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.492549    2557 generic.go:356] "Generic (PLEG): container finished" podID="e2e3a032-66b8-4274-9aa5-3da2c46f6835" containerID="b2f1a5cc278921296e20d794e813e70f4fab6f855e8b991a633ded26d1a209d5" exitCode=0
Jul 29 20:13:01.492677 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.492589    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-641e39-787b6966b8-rwjd7" event={"ID":"e2e3a032-66b8-4274-9aa5-3da2c46f6835","Type":"ContainerDied","Data":"b2f1a5cc278921296e20d794e813e70f4fab6f855e8b991a633ded26d1a209d5"}
Jul 29 20:13:01.492677 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.492601    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-641e39-787b6966b8-rwjd7"
Jul 29 20:13:01.492677 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.492620    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-641e39-787b6966b8-rwjd7" event={"ID":"e2e3a032-66b8-4274-9aa5-3da2c46f6835","Type":"ContainerDied","Data":"9526c32df6e9c8d0fef2c09f2fccae067496df5cf1737160993507d60e652420"}
Jul 29 20:13:01.500978 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.500941    2557 scope.go:117] "RemoveContainer" containerID="01b8ec7cf67373ed1e3c8ec39b473623ebe5118f1737bcbf74f78285eb6ed0f1"
Jul 29 20:13:01.501266 ip-10-0-128-173 kubenswrapper[2557]: E0729 20:13:01.501248    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01b8ec7cf67373ed1e3c8ec39b473623ebe5118f1737bcbf74f78285eb6ed0f1\": container with ID starting with 01b8ec7cf67373ed1e3c8ec39b473623ebe5118f1737bcbf74f78285eb6ed0f1 not found: ID does not exist" containerID="01b8ec7cf67373ed1e3c8ec39b473623ebe5118f1737bcbf74f78285eb6ed0f1"
Jul 29 20:13:01.501335 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.501280    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01b8ec7cf67373ed1e3c8ec39b473623ebe5118f1737bcbf74f78285eb6ed0f1"} err="failed to get container status \"01b8ec7cf67373ed1e3c8ec39b473623ebe5118f1737bcbf74f78285eb6ed0f1\": rpc error: code = NotFound desc = could not find container \"01b8ec7cf67373ed1e3c8ec39b473623ebe5118f1737bcbf74f78285eb6ed0f1\": container with ID starting with 01b8ec7cf67373ed1e3c8ec39b473623ebe5118f1737bcbf74f78285eb6ed0f1 not found: ID does not exist"
Jul 29 20:13:01.501335 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.501303    2557 scope.go:117] "RemoveContainer" containerID="03beaba7553bd230ae735346dcc46f4b84d083f7640929e9ff19b0ed34b04e52"
Jul 29 20:13:01.509658 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.509636    2557 scope.go:117] "RemoveContainer" containerID="03beaba7553bd230ae735346dcc46f4b84d083f7640929e9ff19b0ed34b04e52"
Jul 29 20:13:01.510015 ip-10-0-128-173 kubenswrapper[2557]: E0729 20:13:01.509992    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03beaba7553bd230ae735346dcc46f4b84d083f7640929e9ff19b0ed34b04e52\": container with ID starting with 03beaba7553bd230ae735346dcc46f4b84d083f7640929e9ff19b0ed34b04e52 not found: ID does not exist" containerID="03beaba7553bd230ae735346dcc46f4b84d083f7640929e9ff19b0ed34b04e52"
Jul 29 20:13:01.510107 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.510021    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03beaba7553bd230ae735346dcc46f4b84d083f7640929e9ff19b0ed34b04e52"} err="failed to get container status \"03beaba7553bd230ae735346dcc46f4b84d083f7640929e9ff19b0ed34b04e52\": rpc error: code = NotFound desc = could not find container \"03beaba7553bd230ae735346dcc46f4b84d083f7640929e9ff19b0ed34b04e52\": container with ID starting with 03beaba7553bd230ae735346dcc46f4b84d083f7640929e9ff19b0ed34b04e52 not found: ID does not exist"
Jul 29 20:13:01.510107 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.510039    2557 scope.go:117] "RemoveContainer" containerID="b2f1a5cc278921296e20d794e813e70f4fab6f855e8b991a633ded26d1a209d5"
Jul 29 20:13:01.517771 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.517750    2557 scope.go:117] "RemoveContainer" containerID="b2f1a5cc278921296e20d794e813e70f4fab6f855e8b991a633ded26d1a209d5"
Jul 29 20:13:01.518081 ip-10-0-128-173 kubenswrapper[2557]: E0729 20:13:01.518055    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2f1a5cc278921296e20d794e813e70f4fab6f855e8b991a633ded26d1a209d5\": container with ID starting with b2f1a5cc278921296e20d794e813e70f4fab6f855e8b991a633ded26d1a209d5 not found: ID does not exist" containerID="b2f1a5cc278921296e20d794e813e70f4fab6f855e8b991a633ded26d1a209d5"
Jul 29 20:13:01.518147 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.518086    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f1a5cc278921296e20d794e813e70f4fab6f855e8b991a633ded26d1a209d5"} err="failed to get container status \"b2f1a5cc278921296e20d794e813e70f4fab6f855e8b991a633ded26d1a209d5\": rpc error: code = NotFound desc = could not find container \"b2f1a5cc278921296e20d794e813e70f4fab6f855e8b991a633ded26d1a209d5\": container with ID starting with b2f1a5cc278921296e20d794e813e70f4fab6f855e8b991a633ded26d1a209d5 not found: ID does not exist"
Jul 29 20:13:01.536993 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.536945    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-641e39-5cb6cb867d-c4fdv"]
Jul 29 20:13:01.542841 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.542813    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-641e39-5cb6cb867d-c4fdv"]
Jul 29 20:13:01.556989 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.556950    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgc6z\" (UniqueName: \"kubernetes.io/projected/a31200ff-316c-439b-821d-0a6f939ac40e-kube-api-access-vgc6z\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:13:01.556989 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.556989    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4769186-0b54-46b4-bb07-ecc3a5b92e94-maas-api-tls\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:13:01.557129 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.556999    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2299\" (UniqueName: \"kubernetes.io/projected/e4769186-0b54-46b4-bb07-ecc3a5b92e94-kube-api-access-v2299\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:13:01.557129 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.557008    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnkqp\" (UniqueName: \"kubernetes.io/projected/e2e3a032-66b8-4274-9aa5-3da2c46f6835-kube-api-access-hnkqp\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:13:01.564514 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.564486    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-641e39-787b6966b8-rwjd7"]
Jul 29 20:13:01.568012 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.567987    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-641e39-787b6966b8-rwjd7"]
Jul 29 20:13:01.579917 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.579890    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2"]
Jul 29 20:13:01.584297 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.584277    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-641e39-5f54cbd799-dpbs2"]
Jul 29 20:13:01.679797 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.679764    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41d07dba-bf2e-49bc-9a0f-73208e5c2221" path="/var/lib/kubelet/pods/41d07dba-bf2e-49bc-9a0f-73208e5c2221/volumes"
Jul 29 20:13:01.680147 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.680133    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a31200ff-316c-439b-821d-0a6f939ac40e" path="/var/lib/kubelet/pods/a31200ff-316c-439b-821d-0a6f939ac40e/volumes"
Jul 29 20:13:01.680450 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.680437    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2e3a032-66b8-4274-9aa5-3da2c46f6835" path="/var/lib/kubelet/pods/e2e3a032-66b8-4274-9aa5-3da2c46f6835/volumes"
Jul 29 20:13:01.680775 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:01.680763    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4769186-0b54-46b4-bb07-ecc3a5b92e94" path="/var/lib/kubelet/pods/e4769186-0b54-46b4-bb07-ecc3a5b92e94/volumes"
Jul 29 20:13:08.378005 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:08.377944    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4c787c-4fhwk"]
Jul 29 20:13:08.378505 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:08.378483    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a31200ff-316c-439b-821d-0a6f939ac40e" containerName="payload-processing"
Jul 29 20:13:08.378582 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:08.378509    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a31200ff-316c-439b-821d-0a6f939ac40e" containerName="payload-processing"
Jul 29 20:13:08.378582 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:08.378523    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4769186-0b54-46b4-bb07-ecc3a5b92e94" containerName="maas-api"
Jul 29 20:13:08.378582 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:08.378532    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4769186-0b54-46b4-bb07-ecc3a5b92e94" containerName="maas-api"
Jul 29 20:13:08.378582 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:08.378543    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d07dba-bf2e-49bc-9a0f-73208e5c2221" containerName="revoke-keys"
Jul 29 20:13:08.378582 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:08.378552    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d07dba-bf2e-49bc-9a0f-73208e5c2221" containerName="revoke-keys"
Jul 29 20:13:08.378582 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:08.378564    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2e3a032-66b8-4274-9aa5-3da2c46f6835" containerName="payload-pre-processing"
Jul 29 20:13:08.378894 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:08.378586    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2e3a032-66b8-4274-9aa5-3da2c46f6835" containerName="payload-pre-processing"
Jul 29 20:13:08.378894 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:08.378703    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a31200ff-316c-439b-821d-0a6f939ac40e" containerName="payload-processing"
Jul 29 20:13:08.378894 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:08.378715    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2e3a032-66b8-4274-9aa5-3da2c46f6835" containerName="payload-pre-processing"
Jul 29 20:13:08.378894 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:08.378725    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4769186-0b54-46b4-bb07-ecc3a5b92e94" containerName="maas-api"
Jul 29 20:13:08.378894 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:08.378737    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="41d07dba-bf2e-49bc-9a0f-73208e5c2221" containerName="revoke-keys"
Jul 29 20:13:08.383579 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:08.383558    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4c787c-4fhwk"
Jul 29 20:13:08.386775 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:08.386755    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jsq2r\""
Jul 29 20:13:08.389623 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:08.389599    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4c787c-4fhwk"]
Jul 29 20:13:08.406829 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:08.406802    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9867fcd4-3edc-4e04-8531-2acf7c1434f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-4c787c-4fhwk\" (UID: \"9867fcd4-3edc-4e04-8531-2acf7c1434f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4c787c-4fhwk"
Jul 29 20:13:08.508203 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:08.508170    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9867fcd4-3edc-4e04-8531-2acf7c1434f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-4c787c-4fhwk\" (UID: \"9867fcd4-3edc-4e04-8531-2acf7c1434f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4c787c-4fhwk"
Jul 29 20:13:08.508818 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:08.508797    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9867fcd4-3edc-4e04-8531-2acf7c1434f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-4c787c-4fhwk\" (UID: \"9867fcd4-3edc-4e04-8531-2acf7c1434f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4c787c-4fhwk"
Jul 29 20:13:08.694796 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:08.694756    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4c787c-4fhwk"
Jul 29 20:13:08.825759 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:08.825732    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4c787c-4fhwk"]
Jul 29 20:13:08.828193 ip-10-0-128-173 kubenswrapper[2557]: W0729 20:13:08.828162    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9867fcd4_3edc_4e04_8531_2acf7c1434f6.slice/crio-6faca32844a60f7e2250792ae93d416a1b4878b63bb8ebdfac773e10ac8e08ad WatchSource:0}: Error finding container 6faca32844a60f7e2250792ae93d416a1b4878b63bb8ebdfac773e10ac8e08ad: Status 404 returned error can't find the container with id 6faca32844a60f7e2250792ae93d416a1b4878b63bb8ebdfac773e10ac8e08ad
Jul 29 20:13:09.535784 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:09.535746    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4c787c-4fhwk" event={"ID":"9867fcd4-3edc-4e04-8531-2acf7c1434f6","Type":"ContainerStarted","Data":"49fc1d651fd49545c6c0650eb7490e5f7dadbf3d1ff19d43f3c9c9aa3c5f5ede"}
Jul 29 20:13:09.536258 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:09.535836    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4c787c-4fhwk" event={"ID":"9867fcd4-3edc-4e04-8531-2acf7c1434f6","Type":"ContainerStarted","Data":"6faca32844a60f7e2250792ae93d416a1b4878b63bb8ebdfac773e10ac8e08ad"}
Jul 29 20:13:09.556131 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:09.556080    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4c787c-4fhwk" podStartSLOduration=1.556064772 podStartE2EDuration="1.556064772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:13:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:13:09.555095094 +0000 UTC m=+3360.382594153" watchObservedRunningTime="2026-07-29 20:13:09.556064772 +0000 UTC m=+3360.383563830"
Jul 29 20:13:10.542271 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:10.542235    2557 generic.go:356] "Generic (PLEG): container finished" podID="9867fcd4-3edc-4e04-8531-2acf7c1434f6" containerID="49fc1d651fd49545c6c0650eb7490e5f7dadbf3d1ff19d43f3c9c9aa3c5f5ede" exitCode=0
Jul 29 20:13:10.542693 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:10.542338    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4c787c-4fhwk" event={"ID":"9867fcd4-3edc-4e04-8531-2acf7c1434f6","Type":"ContainerDied","Data":"49fc1d651fd49545c6c0650eb7490e5f7dadbf3d1ff19d43f3c9c9aa3c5f5ede"}
Jul 29 20:13:11.679682 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:11.679656    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4c787c-4fhwk"
Jul 29 20:13:11.739633 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:11.739603    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9867fcd4-3edc-4e04-8531-2acf7c1434f6-maas-api-service-ca\") pod \"9867fcd4-3edc-4e04-8531-2acf7c1434f6\" (UID: \"9867fcd4-3edc-4e04-8531-2acf7c1434f6\") "
Jul 29 20:13:11.739944 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:11.739924    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9867fcd4-3edc-4e04-8531-2acf7c1434f6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9867fcd4-3edc-4e04-8531-2acf7c1434f6" (UID: "9867fcd4-3edc-4e04-8531-2acf7c1434f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:13:11.840746 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:11.840679    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9867fcd4-3edc-4e04-8531-2acf7c1434f6-maas-api-service-ca\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:13:12.552974 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:12.552938    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4c787c-4fhwk"
Jul 29 20:13:12.553198 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:12.552992    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4c787c-4fhwk" event={"ID":"9867fcd4-3edc-4e04-8531-2acf7c1434f6","Type":"ContainerDied","Data":"6faca32844a60f7e2250792ae93d416a1b4878b63bb8ebdfac773e10ac8e08ad"}
Jul 29 20:13:12.553198 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:12.553038    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6faca32844a60f7e2250792ae93d416a1b4878b63bb8ebdfac773e10ac8e08ad"
Jul 29 20:13:19.027237 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.027201    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l"]
Jul 29 20:13:19.027697 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.027408    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l" podUID="5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8" containerName="maas-api" containerID="cri-o://d7175b952a2dff5e7c3b77201f79f34f17ba6974925fc0042850934ba2551b2f" gracePeriod=30
Jul 29 20:13:19.089797 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.089758    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-4c787c-5489f58b58-hddhc"]
Jul 29 20:13:19.090069 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.090036    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-4c787c-5489f58b58-hddhc" podUID="74179647-d44f-48c3-b934-ba7a2226bc0a" containerName="payload-processing" containerID="cri-o://8a6edd4524fa0c1bb01597ee541e1092ab9f6c9c396b43819d9912e9ffc21b21" gracePeriod=30
Jul 29 20:13:19.101957 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.101923    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-4c787c-8f587d5f6-n64xz"]
Jul 29 20:13:19.103430 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.103398    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4c787c-8f587d5f6-n64xz" podUID="15466f4c-344d-48b3-9487-dfd21dab978e" containerName="payload-pre-processing" containerID="cri-o://a576e044a71312db3356a661cd3818e3058aa1e3677c49455202702437b9b1c5" gracePeriod=30
Jul 29 20:13:19.255435 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.255366    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4c787c-4fhwk"]
Jul 29 20:13:19.257772 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.257742    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4c787c-4fhwk"]
Jul 29 20:13:19.383627 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.383602    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l"
Jul 29 20:13:19.387652 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.387632    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-4c787c-5489f58b58-hddhc"
Jul 29 20:13:19.419640 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.419620    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4c787c-8f587d5f6-n64xz"
Jul 29 20:13:19.511211 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.511178    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8-openshift-service-ca\") pod \"5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8\" (UID: \"5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8\") "
Jul 29 20:13:19.511417 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.511223    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rsjp\" (UniqueName: \"kubernetes.io/projected/15466f4c-344d-48b3-9487-dfd21dab978e-kube-api-access-5rsjp\") pod \"15466f4c-344d-48b3-9487-dfd21dab978e\" (UID: \"15466f4c-344d-48b3-9487-dfd21dab978e\") "
Jul 29 20:13:19.511417 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.511266    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74179647-d44f-48c3-b934-ba7a2226bc0a-plugins-config-volume\") pod \"74179647-d44f-48c3-b934-ba7a2226bc0a\" (UID: \"74179647-d44f-48c3-b934-ba7a2226bc0a\") "
Jul 29 20:13:19.511417 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.511345    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8-maas-api-tls\") pod \"5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8\" (UID: \"5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8\") "
Jul 29 20:13:19.511417 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.511369    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qphr2\" (UniqueName: \"kubernetes.io/projected/5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8-kube-api-access-qphr2\") pod \"5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8\" (UID: \"5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8\") "
Jul 29 20:13:19.511417 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.511411    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15466f4c-344d-48b3-9487-dfd21dab978e-plugins-config-volume\") pod \"15466f4c-344d-48b3-9487-dfd21dab978e\" (UID: \"15466f4c-344d-48b3-9487-dfd21dab978e\") "
Jul 29 20:13:19.511703 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.511445    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kscw6\" (UniqueName: \"kubernetes.io/projected/74179647-d44f-48c3-b934-ba7a2226bc0a-kube-api-access-kscw6\") pod \"74179647-d44f-48c3-b934-ba7a2226bc0a\" (UID: \"74179647-d44f-48c3-b934-ba7a2226bc0a\") "
Jul 29 20:13:19.511757 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.511691    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8" (UID: "5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:13:19.511827 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.511799    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74179647-d44f-48c3-b934-ba7a2226bc0a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "74179647-d44f-48c3-b934-ba7a2226bc0a" (UID: "74179647-d44f-48c3-b934-ba7a2226bc0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:13:19.511827 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.511807    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15466f4c-344d-48b3-9487-dfd21dab978e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "15466f4c-344d-48b3-9487-dfd21dab978e" (UID: "15466f4c-344d-48b3-9487-dfd21dab978e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:13:19.513754 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.513725    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74179647-d44f-48c3-b934-ba7a2226bc0a-kube-api-access-kscw6" (OuterVolumeSpecName: "kube-api-access-kscw6") pod "74179647-d44f-48c3-b934-ba7a2226bc0a" (UID: "74179647-d44f-48c3-b934-ba7a2226bc0a"). InnerVolumeSpecName "kube-api-access-kscw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:13:19.513860 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.513756    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8-kube-api-access-qphr2" (OuterVolumeSpecName: "kube-api-access-qphr2") pod "5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8" (UID: "5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8"). InnerVolumeSpecName "kube-api-access-qphr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:13:19.513860 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.513773    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15466f4c-344d-48b3-9487-dfd21dab978e-kube-api-access-5rsjp" (OuterVolumeSpecName: "kube-api-access-5rsjp") pod "15466f4c-344d-48b3-9487-dfd21dab978e" (UID: "15466f4c-344d-48b3-9487-dfd21dab978e"). InnerVolumeSpecName "kube-api-access-5rsjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:13:19.513860 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.513797    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8" (UID: "5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:13:19.589846 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.589763    2557 generic.go:356] "Generic (PLEG): container finished" podID="74179647-d44f-48c3-b934-ba7a2226bc0a" containerID="8a6edd4524fa0c1bb01597ee541e1092ab9f6c9c396b43819d9912e9ffc21b21" exitCode=0
Jul 29 20:13:19.589846 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.589809    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-4c787c-5489f58b58-hddhc"
Jul 29 20:13:19.590070 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.589855    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-4c787c-5489f58b58-hddhc" event={"ID":"74179647-d44f-48c3-b934-ba7a2226bc0a","Type":"ContainerDied","Data":"8a6edd4524fa0c1bb01597ee541e1092ab9f6c9c396b43819d9912e9ffc21b21"}
Jul 29 20:13:19.590070 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.589945    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-4c787c-5489f58b58-hddhc" event={"ID":"74179647-d44f-48c3-b934-ba7a2226bc0a","Type":"ContainerDied","Data":"ffd7243bb21e5f9a734d119c744e7c16d49db5217a8df7aab870e35d40b048b6"}
Jul 29 20:13:19.590070 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.589999    2557 scope.go:117] "RemoveContainer" containerID="8a6edd4524fa0c1bb01597ee541e1092ab9f6c9c396b43819d9912e9ffc21b21"
Jul 29 20:13:19.591306 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.591281    2557 generic.go:356] "Generic (PLEG): container finished" podID="5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8" containerID="d7175b952a2dff5e7c3b77201f79f34f17ba6974925fc0042850934ba2551b2f" exitCode=0
Jul 29 20:13:19.591416 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.591326    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l"
Jul 29 20:13:19.591416 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.591367    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l" event={"ID":"5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8","Type":"ContainerDied","Data":"d7175b952a2dff5e7c3b77201f79f34f17ba6974925fc0042850934ba2551b2f"}
Jul 29 20:13:19.591510 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.591414    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l" event={"ID":"5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8","Type":"ContainerDied","Data":"73b25d8299a07e84f64d6c6112c835adf2fe3de6c9ed7b22543b6cc07f5a5269"}
Jul 29 20:13:19.592742 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.592717    2557 generic.go:356] "Generic (PLEG): container finished" podID="15466f4c-344d-48b3-9487-dfd21dab978e" containerID="a576e044a71312db3356a661cd3818e3058aa1e3677c49455202702437b9b1c5" exitCode=0
Jul 29 20:13:19.592881 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.592755    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4c787c-8f587d5f6-n64xz"
Jul 29 20:13:19.592881 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.592763    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4c787c-8f587d5f6-n64xz" event={"ID":"15466f4c-344d-48b3-9487-dfd21dab978e","Type":"ContainerDied","Data":"a576e044a71312db3356a661cd3818e3058aa1e3677c49455202702437b9b1c5"}
Jul 29 20:13:19.592881 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.592799    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4c787c-8f587d5f6-n64xz" event={"ID":"15466f4c-344d-48b3-9487-dfd21dab978e","Type":"ContainerDied","Data":"6a460c67b31432e4b1c30859a2ddc0cf9073880cd610fee8cf11412adac1dd12"}
Jul 29 20:13:19.601224 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.601131    2557 scope.go:117] "RemoveContainer" containerID="8a6edd4524fa0c1bb01597ee541e1092ab9f6c9c396b43819d9912e9ffc21b21"
Jul 29 20:13:19.601495 ip-10-0-128-173 kubenswrapper[2557]: E0729 20:13:19.601467    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a6edd4524fa0c1bb01597ee541e1092ab9f6c9c396b43819d9912e9ffc21b21\": container with ID starting with 8a6edd4524fa0c1bb01597ee541e1092ab9f6c9c396b43819d9912e9ffc21b21 not found: ID does not exist" containerID="8a6edd4524fa0c1bb01597ee541e1092ab9f6c9c396b43819d9912e9ffc21b21"
Jul 29 20:13:19.601559 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.601506    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a6edd4524fa0c1bb01597ee541e1092ab9f6c9c396b43819d9912e9ffc21b21"} err="failed to get container status \"8a6edd4524fa0c1bb01597ee541e1092ab9f6c9c396b43819d9912e9ffc21b21\": rpc error: code = NotFound desc = could not find container \"8a6edd4524fa0c1bb01597ee541e1092ab9f6c9c396b43819d9912e9ffc21b21\": container with ID starting with 8a6edd4524fa0c1bb01597ee541e1092ab9f6c9c396b43819d9912e9ffc21b21 not found: ID does not exist"
Jul 29 20:13:19.601559 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.601528    2557 scope.go:117] "RemoveContainer" containerID="d7175b952a2dff5e7c3b77201f79f34f17ba6974925fc0042850934ba2551b2f"
Jul 29 20:13:19.609694 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.609677    2557 scope.go:117] "RemoveContainer" containerID="d7175b952a2dff5e7c3b77201f79f34f17ba6974925fc0042850934ba2551b2f"
Jul 29 20:13:19.609934 ip-10-0-128-173 kubenswrapper[2557]: E0729 20:13:19.609914    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7175b952a2dff5e7c3b77201f79f34f17ba6974925fc0042850934ba2551b2f\": container with ID starting with d7175b952a2dff5e7c3b77201f79f34f17ba6974925fc0042850934ba2551b2f not found: ID does not exist" containerID="d7175b952a2dff5e7c3b77201f79f34f17ba6974925fc0042850934ba2551b2f"
Jul 29 20:13:19.610018 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.609941    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7175b952a2dff5e7c3b77201f79f34f17ba6974925fc0042850934ba2551b2f"} err="failed to get container status \"d7175b952a2dff5e7c3b77201f79f34f17ba6974925fc0042850934ba2551b2f\": rpc error: code = NotFound desc = could not find container \"d7175b952a2dff5e7c3b77201f79f34f17ba6974925fc0042850934ba2551b2f\": container with ID starting with d7175b952a2dff5e7c3b77201f79f34f17ba6974925fc0042850934ba2551b2f not found: ID does not exist"
Jul 29 20:13:19.610018 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.609957    2557 scope.go:117] "RemoveContainer" containerID="a576e044a71312db3356a661cd3818e3058aa1e3677c49455202702437b9b1c5"
Jul 29 20:13:19.612188 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.612167    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74179647-d44f-48c3-b934-ba7a2226bc0a-plugins-config-volume\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:13:19.612247 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.612192    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8-maas-api-tls\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:13:19.612247 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.612206    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qphr2\" (UniqueName: \"kubernetes.io/projected/5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8-kube-api-access-qphr2\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:13:19.612247 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.612218    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15466f4c-344d-48b3-9487-dfd21dab978e-plugins-config-volume\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:13:19.612247 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.612229    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kscw6\" (UniqueName: \"kubernetes.io/projected/74179647-d44f-48c3-b934-ba7a2226bc0a-kube-api-access-kscw6\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:13:19.612247 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.612240    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8-openshift-service-ca\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:13:19.612410 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.612252    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rsjp\" (UniqueName: \"kubernetes.io/projected/15466f4c-344d-48b3-9487-dfd21dab978e-kube-api-access-5rsjp\") on node \"ip-10-0-128-173.ec2.internal\" DevicePath \"\""
Jul 29 20:13:19.617649 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.617629    2557 scope.go:117] "RemoveContainer" containerID="a576e044a71312db3356a661cd3818e3058aa1e3677c49455202702437b9b1c5"
Jul 29 20:13:19.617927 ip-10-0-128-173 kubenswrapper[2557]: E0729 20:13:19.617906    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a576e044a71312db3356a661cd3818e3058aa1e3677c49455202702437b9b1c5\": container with ID starting with a576e044a71312db3356a661cd3818e3058aa1e3677c49455202702437b9b1c5 not found: ID does not exist" containerID="a576e044a71312db3356a661cd3818e3058aa1e3677c49455202702437b9b1c5"
Jul 29 20:13:19.618021 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.617935    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a576e044a71312db3356a661cd3818e3058aa1e3677c49455202702437b9b1c5"} err="failed to get container status \"a576e044a71312db3356a661cd3818e3058aa1e3677c49455202702437b9b1c5\": rpc error: code = NotFound desc = could not find container \"a576e044a71312db3356a661cd3818e3058aa1e3677c49455202702437b9b1c5\": container with ID starting with a576e044a71312db3356a661cd3818e3058aa1e3677c49455202702437b9b1c5 not found: ID does not exist"
Jul 29 20:13:19.619202 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.619180    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-4c787c-5489f58b58-hddhc"]
Jul 29 20:13:19.620796 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.620776    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-4c787c-5489f58b58-hddhc"]
Jul 29 20:13:19.630675 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.630655    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-4c787c-8f587d5f6-n64xz"]
Jul 29 20:13:19.634160 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.634141    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-4c787c-8f587d5f6-n64xz"]
Jul 29 20:13:19.646589 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.646570    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l"]
Jul 29 20:13:19.650568 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.650551    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4c787c-5d787fc7cb-fbv8l"]
Jul 29 20:13:19.678776 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.678754    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15466f4c-344d-48b3-9487-dfd21dab978e" path="/var/lib/kubelet/pods/15466f4c-344d-48b3-9487-dfd21dab978e/volumes"
Jul 29 20:13:19.679157 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.679143    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8" path="/var/lib/kubelet/pods/5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8/volumes"
Jul 29 20:13:19.679469 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.679456    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74179647-d44f-48c3-b934-ba7a2226bc0a" path="/var/lib/kubelet/pods/74179647-d44f-48c3-b934-ba7a2226bc0a/volumes"
Jul 29 20:13:19.679736 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:19.679725    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9867fcd4-3edc-4e04-8531-2acf7c1434f6" path="/var/lib/kubelet/pods/9867fcd4-3edc-4e04-8531-2acf7c1434f6/volumes"
Jul 29 20:13:32.682772 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:32.682743    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d84699fcf-kk7qz_f034f84d-5306-43a8-914d-0a91c0d60b4e/copy-manifests/0.log"
Jul 29 20:13:32.699553 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:32.699525    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d84699fcf-kk7qz_f034f84d-5306-43a8-914d-0a91c0d60b4e/manager/0.log"
Jul 29 20:13:33.271617 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:33.271586    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29755890-6hnnk_ab14f273-7fc8-4074-854e-de0fb06a8737/cleanup/0.log"
Jul 29 20:13:33.380517 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:33.380488    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29755905-48dzc_6421eb66-5559-4396-a8e3-3be2e7a9ece1/cleanup/0.log"
Jul 29 20:13:33.590364 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:33.590280    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-cwr5x_2aa53dc2-24c4-4801-bbca-b84ac909f19e/postgres/0.log"
Jul 29 20:13:35.147866 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:35.147834    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-9bw6q_f43acfbd-67be-4d90-9985-e3b3382274c1/kuadrant-console-plugin/0.log"
Jul 29 20:13:36.266654 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:36.266618    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d594574db-l29np_3518470c-c9c6-4fce-b7ee-399de7938787/kube-auth-proxy/0.log"
Jul 29 20:13:36.487804 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:36.487772    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-tj96m_31e35ba1-a9ef-4cb3-b1a2-aa983501d6bf/payload-pre-processing/0.log"
Jul 29 20:13:36.714756 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:36.714732    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7fc6497669-xk7lp_dd24fa62-fd72-4e18-af46-6930625f22da/router/0.log"
Jul 29 20:13:37.030676 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:37.030601    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-q4s4s_f6c448d8-f996-4179-b52a-e637a7622d14/main/0.log"
Jul 29 20:13:37.559955 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:37.559923    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-6mwvq_df575b63-388f-45b7-aa75-a2833c64c645/main/0.log"
Jul 29 20:13:42.734103 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:42.734066    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7np8j/must-gather-htdg8"]
Jul 29 20:13:42.734551 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:42.734386    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15466f4c-344d-48b3-9487-dfd21dab978e" containerName="payload-pre-processing"
Jul 29 20:13:42.734551 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:42.734396    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="15466f4c-344d-48b3-9487-dfd21dab978e" containerName="payload-pre-processing"
Jul 29 20:13:42.734551 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:42.734407    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74179647-d44f-48c3-b934-ba7a2226bc0a" containerName="payload-processing"
Jul 29 20:13:42.734551 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:42.734413    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="74179647-d44f-48c3-b934-ba7a2226bc0a" containerName="payload-processing"
Jul 29 20:13:42.734551 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:42.734425    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8" containerName="maas-api"
Jul 29 20:13:42.734551 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:42.734430    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8" containerName="maas-api"
Jul 29 20:13:42.734551 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:42.734446    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9867fcd4-3edc-4e04-8531-2acf7c1434f6" containerName="revoke-keys"
Jul 29 20:13:42.734551 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:42.734452    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9867fcd4-3edc-4e04-8531-2acf7c1434f6" containerName="revoke-keys"
Jul 29 20:13:42.734551 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:42.734503    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="74179647-d44f-48c3-b934-ba7a2226bc0a" containerName="payload-processing"
Jul 29 20:13:42.734551 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:42.734509    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9867fcd4-3edc-4e04-8531-2acf7c1434f6" containerName="revoke-keys"
Jul 29 20:13:42.734551 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:42.734516    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f3fe360-665e-4e28-a8f0-a29b7d0cc1f8" containerName="maas-api"
Jul 29 20:13:42.734551 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:42.734522    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="15466f4c-344d-48b3-9487-dfd21dab978e" containerName="payload-pre-processing"
Jul 29 20:13:42.737372 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:42.737356    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7np8j/must-gather-htdg8"
Jul 29 20:13:42.740808 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:42.740787    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7np8j\"/\"default-dockercfg-xfh45\""
Jul 29 20:13:42.741578 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:42.741561    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7np8j\"/\"kube-root-ca.crt\""
Jul 29 20:13:42.741643 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:42.741589    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7np8j\"/\"openshift-service-ca.crt\""
Jul 29 20:13:42.751486 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:42.751461    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7np8j/must-gather-htdg8"]
Jul 29 20:13:42.824944 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:42.824910    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7fgx\" (UniqueName: \"kubernetes.io/projected/89df24a6-4ba6-4855-b228-d97b7cc95178-kube-api-access-j7fgx\") pod \"must-gather-htdg8\" (UID: \"89df24a6-4ba6-4855-b228-d97b7cc95178\") " pod="openshift-must-gather-7np8j/must-gather-htdg8"
Jul 29 20:13:42.825143 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:42.824994    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/89df24a6-4ba6-4855-b228-d97b7cc95178-must-gather-output\") pod \"must-gather-htdg8\" (UID: \"89df24a6-4ba6-4855-b228-d97b7cc95178\") " pod="openshift-must-gather-7np8j/must-gather-htdg8"
Jul 29 20:13:42.926028 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:42.925987    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7fgx\" (UniqueName: \"kubernetes.io/projected/89df24a6-4ba6-4855-b228-d97b7cc95178-kube-api-access-j7fgx\") pod \"must-gather-htdg8\" (UID: \"89df24a6-4ba6-4855-b228-d97b7cc95178\") " pod="openshift-must-gather-7np8j/must-gather-htdg8"
Jul 29 20:13:42.926186 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:42.926076    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/89df24a6-4ba6-4855-b228-d97b7cc95178-must-gather-output\") pod \"must-gather-htdg8\" (UID: \"89df24a6-4ba6-4855-b228-d97b7cc95178\") " pod="openshift-must-gather-7np8j/must-gather-htdg8"
Jul 29 20:13:42.926412 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:42.926395    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/89df24a6-4ba6-4855-b228-d97b7cc95178-must-gather-output\") pod \"must-gather-htdg8\" (UID: \"89df24a6-4ba6-4855-b228-d97b7cc95178\") " pod="openshift-must-gather-7np8j/must-gather-htdg8"
Jul 29 20:13:42.936589 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:42.936563    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7fgx\" (UniqueName: \"kubernetes.io/projected/89df24a6-4ba6-4855-b228-d97b7cc95178-kube-api-access-j7fgx\") pod \"must-gather-htdg8\" (UID: \"89df24a6-4ba6-4855-b228-d97b7cc95178\") " pod="openshift-must-gather-7np8j/must-gather-htdg8"
Jul 29 20:13:43.046000 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:43.045911    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7np8j/must-gather-htdg8"
Jul 29 20:13:43.178065 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:43.178039    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7np8j/must-gather-htdg8"]
Jul 29 20:13:43.180491 ip-10-0-128-173 kubenswrapper[2557]: W0729 20:13:43.180461    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89df24a6_4ba6_4855_b228_d97b7cc95178.slice/crio-b744fe6e406e7522969a5c9ce3c64b15052eeb1868cfadd9c4d2f26c2d206664 WatchSource:0}: Error finding container b744fe6e406e7522969a5c9ce3c64b15052eeb1868cfadd9c4d2f26c2d206664: Status 404 returned error can't find the container with id b744fe6e406e7522969a5c9ce3c64b15052eeb1868cfadd9c4d2f26c2d206664
Jul 29 20:13:43.182292 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:43.182275    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 20:13:43.703934 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:43.703901    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7np8j/must-gather-htdg8" event={"ID":"89df24a6-4ba6-4855-b228-d97b7cc95178","Type":"ContainerStarted","Data":"b744fe6e406e7522969a5c9ce3c64b15052eeb1868cfadd9c4d2f26c2d206664"}
Jul 29 20:13:44.713592 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:44.712714    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7np8j/must-gather-htdg8" event={"ID":"89df24a6-4ba6-4855-b228-d97b7cc95178","Type":"ContainerStarted","Data":"16496d52a62ec033107b00d1e0b98cb4050c85ee7d45561171c0c382203b30bf"}
Jul 29 20:13:44.713592 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:44.712768    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7np8j/must-gather-htdg8" event={"ID":"89df24a6-4ba6-4855-b228-d97b7cc95178","Type":"ContainerStarted","Data":"887e398a7f4f0371e3572dc4b4f8e8a869f9fc7ee2f06f6be01d2a08eb48ce68"}
Jul 29 20:13:44.740650 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:44.740573    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7np8j/must-gather-htdg8" podStartSLOduration=1.884609773 podStartE2EDuration="2.740551719s" totalImagesPullingTime="855.941946ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:13:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:13:44.738386355 +0000 UTC m=+3395.565885414" watchObservedRunningTime="2026-07-29 20:13:44.740551719 +0000 UTC m=+3395.568050779"
Jul 29 20:13:45.720611 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:45.720575    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-v5x8q_93168299-ac4f-4fdd-8ac3-b69edb385d71/global-pull-secret-syncer/0.log"
Jul 29 20:13:45.870325 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:45.870292    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6jmnn_639a786b-08de-453b-8733-28f54e9a59ff/konnectivity-agent/0.log"
Jul 29 20:13:45.935843 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:45.935811    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-173.ec2.internal_70a21ba21338b6ba758e27d2cca1c0ab/haproxy/0.log"
Jul 29 20:13:50.135498 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:50.135464    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-9bw6q_f43acfbd-67be-4d90-9985-e3b3382274c1/kuadrant-console-plugin/0.log"
Jul 29 20:13:52.587530 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:52.587489    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-s7m8g_8329219a-75ca-4323-a2e1-42d46d14b6d8/monitoring-plugin/0.log"
Jul 29 20:13:52.850209 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:52.850177    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v8jvg_5e58f89f-c42e-4fe3-9d4f-14e805228f11/node-exporter/0.log"
Jul 29 20:13:52.874737 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:52.874675    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v8jvg_5e58f89f-c42e-4fe3-9d4f-14e805228f11/kube-rbac-proxy/0.log"
Jul 29 20:13:52.898476 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:52.898445    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v8jvg_5e58f89f-c42e-4fe3-9d4f-14e805228f11/init-textfile/0.log"
Jul 29 20:13:53.063995 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:53.063877    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eba3dd68-ff9d-472d-b498-a0d7af3ae175/prometheus/0.log"
Jul 29 20:13:53.093044 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:53.093007    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eba3dd68-ff9d-472d-b498-a0d7af3ae175/config-reloader/0.log"
Jul 29 20:13:53.118659 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:53.118623    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eba3dd68-ff9d-472d-b498-a0d7af3ae175/thanos-sidecar/0.log"
Jul 29 20:13:53.141852 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:53.141802    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eba3dd68-ff9d-472d-b498-a0d7af3ae175/kube-rbac-proxy-web/0.log"
Jul 29 20:13:53.171918 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:53.171887    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eba3dd68-ff9d-472d-b498-a0d7af3ae175/kube-rbac-proxy/0.log"
Jul 29 20:13:53.213024 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:53.212997    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eba3dd68-ff9d-472d-b498-a0d7af3ae175/kube-rbac-proxy-thanos/0.log"
Jul 29 20:13:53.242934 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:53.242889    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eba3dd68-ff9d-472d-b498-a0d7af3ae175/init-config-reloader/0.log"
Jul 29 20:13:53.282443 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:53.282415    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-frhw9_d7bc168b-7b7c-4242-b505-92500fd9a3a3/prometheus-operator/0.log"
Jul 29 20:13:53.308147 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:53.308113    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-frhw9_d7bc168b-7b7c-4242-b505-92500fd9a3a3/kube-rbac-proxy/0.log"
Jul 29 20:13:54.865856 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:54.865823    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-dh7rf_259710be-9c9b-4457-81c2-301f7d7cb110/networking-console-plugin/0.log"
Jul 29 20:13:56.060826 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:56.060799    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5b57854b96-s5rfx_ef5d5ef6-6f90-4b28-afeb-1487cfa7a4ec/console/0.log"
Jul 29 20:13:56.096671 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:56.096643    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-84drj_b92baea5-f190-4e2d-9e6e-2fcfc1a61176/download-server/0.log"
Jul 29 20:13:56.202071 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:56.202033    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7np8j/perf-node-gather-daemonset-ljszz"]
Jul 29 20:13:56.211247 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:56.211219    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7np8j/perf-node-gather-daemonset-ljszz"
Jul 29 20:13:56.218302 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:56.218273    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7np8j/perf-node-gather-daemonset-ljszz"]
Jul 29 20:13:56.268333 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:56.268293    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8263b59b-6c31-4ddb-a4d8-b69e18d62b42-proc\") pod \"perf-node-gather-daemonset-ljszz\" (UID: \"8263b59b-6c31-4ddb-a4d8-b69e18d62b42\") " pod="openshift-must-gather-7np8j/perf-node-gather-daemonset-ljszz"
Jul 29 20:13:56.268333 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:56.268337    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8263b59b-6c31-4ddb-a4d8-b69e18d62b42-lib-modules\") pod \"perf-node-gather-daemonset-ljszz\" (UID: \"8263b59b-6c31-4ddb-a4d8-b69e18d62b42\") " pod="openshift-must-gather-7np8j/perf-node-gather-daemonset-ljszz"
Jul 29 20:13:56.268557 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:56.268412    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8263b59b-6c31-4ddb-a4d8-b69e18d62b42-sys\") pod \"perf-node-gather-daemonset-ljszz\" (UID: \"8263b59b-6c31-4ddb-a4d8-b69e18d62b42\") " pod="openshift-must-gather-7np8j/perf-node-gather-daemonset-ljszz"
Jul 29 20:13:56.268557 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:56.268457    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhck2\" (UniqueName: \"kubernetes.io/projected/8263b59b-6c31-4ddb-a4d8-b69e18d62b42-kube-api-access-vhck2\") pod \"perf-node-gather-daemonset-ljszz\" (UID: \"8263b59b-6c31-4ddb-a4d8-b69e18d62b42\") " pod="openshift-must-gather-7np8j/perf-node-gather-daemonset-ljszz"
Jul 29 20:13:56.268557 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:56.268490    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8263b59b-6c31-4ddb-a4d8-b69e18d62b42-podres\") pod \"perf-node-gather-daemonset-ljszz\" (UID: \"8263b59b-6c31-4ddb-a4d8-b69e18d62b42\") " pod="openshift-must-gather-7np8j/perf-node-gather-daemonset-ljszz"
Jul 29 20:13:56.369218 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:56.369130    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8263b59b-6c31-4ddb-a4d8-b69e18d62b42-proc\") pod \"perf-node-gather-daemonset-ljszz\" (UID: \"8263b59b-6c31-4ddb-a4d8-b69e18d62b42\") " pod="openshift-must-gather-7np8j/perf-node-gather-daemonset-ljszz"
Jul 29 20:13:56.369218 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:56.369172    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8263b59b-6c31-4ddb-a4d8-b69e18d62b42-lib-modules\") pod \"perf-node-gather-daemonset-ljszz\" (UID: \"8263b59b-6c31-4ddb-a4d8-b69e18d62b42\") " pod="openshift-must-gather-7np8j/perf-node-gather-daemonset-ljszz"
Jul 29 20:13:56.369218 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:56.369198    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8263b59b-6c31-4ddb-a4d8-b69e18d62b42-sys\") pod \"perf-node-gather-daemonset-ljszz\" (UID: \"8263b59b-6c31-4ddb-a4d8-b69e18d62b42\") " pod="openshift-must-gather-7np8j/perf-node-gather-daemonset-ljszz"
Jul 29 20:13:56.369502 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:56.369236    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhck2\" (UniqueName: \"kubernetes.io/projected/8263b59b-6c31-4ddb-a4d8-b69e18d62b42-kube-api-access-vhck2\") pod \"perf-node-gather-daemonset-ljszz\" (UID: \"8263b59b-6c31-4ddb-a4d8-b69e18d62b42\") " pod="openshift-must-gather-7np8j/perf-node-gather-daemonset-ljszz"
Jul 29 20:13:56.369502 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:56.369266    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8263b59b-6c31-4ddb-a4d8-b69e18d62b42-proc\") pod \"perf-node-gather-daemonset-ljszz\" (UID: \"8263b59b-6c31-4ddb-a4d8-b69e18d62b42\") " pod="openshift-must-gather-7np8j/perf-node-gather-daemonset-ljszz"
Jul 29 20:13:56.369502 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:56.369272    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8263b59b-6c31-4ddb-a4d8-b69e18d62b42-podres\") pod \"perf-node-gather-daemonset-ljszz\" (UID: \"8263b59b-6c31-4ddb-a4d8-b69e18d62b42\") " pod="openshift-must-gather-7np8j/perf-node-gather-daemonset-ljszz"
Jul 29 20:13:56.369502 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:56.369341    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8263b59b-6c31-4ddb-a4d8-b69e18d62b42-podres\") pod \"perf-node-gather-daemonset-ljszz\" (UID: \"8263b59b-6c31-4ddb-a4d8-b69e18d62b42\") " pod="openshift-must-gather-7np8j/perf-node-gather-daemonset-ljszz"
Jul 29 20:13:56.369502 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:56.369356    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8263b59b-6c31-4ddb-a4d8-b69e18d62b42-sys\") pod \"perf-node-gather-daemonset-ljszz\" (UID: \"8263b59b-6c31-4ddb-a4d8-b69e18d62b42\") " pod="openshift-must-gather-7np8j/perf-node-gather-daemonset-ljszz"
Jul 29 20:13:56.369502 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:56.369339    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8263b59b-6c31-4ddb-a4d8-b69e18d62b42-lib-modules\") pod \"perf-node-gather-daemonset-ljszz\" (UID: \"8263b59b-6c31-4ddb-a4d8-b69e18d62b42\") " pod="openshift-must-gather-7np8j/perf-node-gather-daemonset-ljszz"
Jul 29 20:13:56.379684 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:56.379657    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhck2\" (UniqueName: \"kubernetes.io/projected/8263b59b-6c31-4ddb-a4d8-b69e18d62b42-kube-api-access-vhck2\") pod \"perf-node-gather-daemonset-ljszz\" (UID: \"8263b59b-6c31-4ddb-a4d8-b69e18d62b42\") " pod="openshift-must-gather-7np8j/perf-node-gather-daemonset-ljszz"
Jul 29 20:13:56.536760 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:56.536721    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7np8j/perf-node-gather-daemonset-ljszz"
Jul 29 20:13:56.734097 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:56.732621    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7np8j/perf-node-gather-daemonset-ljszz"]
Jul 29 20:13:56.797672 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:56.797593    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7np8j/perf-node-gather-daemonset-ljszz" event={"ID":"8263b59b-6c31-4ddb-a4d8-b69e18d62b42","Type":"ContainerStarted","Data":"8273f26dbdf9bf8089cf5fe1ba9f80b7364d5a3a6a50ed652561885efe2fbf25"}
Jul 29 20:13:57.807686 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:57.807594    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7np8j/perf-node-gather-daemonset-ljszz" event={"ID":"8263b59b-6c31-4ddb-a4d8-b69e18d62b42","Type":"ContainerStarted","Data":"13f21951b3840e67d5c70d5a959ab4dc364853a9f087786d4414d60036ecca09"}
Jul 29 20:13:57.808162 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:57.807819    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7np8j/perf-node-gather-daemonset-ljszz"
Jul 29 20:13:57.828932 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:57.828862    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7np8j/perf-node-gather-daemonset-ljszz" podStartSLOduration=1.828841476 podStartE2EDuration="1.828841476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:13:57.827906919 +0000 UTC m=+3408.655405978" watchObservedRunningTime="2026-07-29 20:13:57.828841476 +0000 UTC m=+3408.656340535"
Jul 29 20:13:58.021506 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:58.021477    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kq4zj_ffafbd2c-ca03-47e9-ab0d-cd6125697e9e/dns/0.log"
Jul 29 20:13:58.043248 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:58.043218    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kq4zj_ffafbd2c-ca03-47e9-ab0d-cd6125697e9e/kube-rbac-proxy/0.log"
Jul 29 20:13:58.176428 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:58.176403    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vft97_f737cfa7-db52-409a-b5c3-f704b9170fa4/dns-node-resolver/0.log"
Jul 29 20:13:58.800656 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:58.800621    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-frvmp_d160081b-f780-4c60-85f9-04b9a6d1c3d9/node-ca/0.log"
Jul 29 20:13:59.979197 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:13:59.979163    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d594574db-l29np_3518470c-c9c6-4fce-b7ee-399de7938787/kube-auth-proxy/0.log"
Jul 29 20:14:00.068728 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:14:00.068695    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-tj96m_31e35ba1-a9ef-4cb3-b1a2-aa983501d6bf/payload-pre-processing/0.log"
Jul 29 20:14:00.125439 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:14:00.125401    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7fc6497669-xk7lp_dd24fa62-fd72-4e18-af46-6930625f22da/router/0.log"
Jul 29 20:14:01.048956 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:14:01.048924    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-q724q_7266743b-b72b-41da-8110-b47af371be5c/serve-healthcheck-canary/0.log"
Jul 29 20:14:01.537147 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:14:01.537111    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-cfmh8_2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac/insights-operator/0.log"
Jul 29 20:14:01.539684 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:14:01.539660    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-cfmh8_2eacb5ca-ad77-4c2c-8fca-7f99a1c8c3ac/insights-operator/1.log"
Jul 29 20:14:01.559598 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:14:01.559550    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bpq7l_6e8a45f8-a6cf-4d48-b1cb-92a91e190c50/kube-rbac-proxy/0.log"
Jul 29 20:14:01.589561 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:14:01.589532    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bpq7l_6e8a45f8-a6cf-4d48-b1cb-92a91e190c50/exporter/0.log"
Jul 29 20:14:01.611406 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:14:01.611382    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bpq7l_6e8a45f8-a6cf-4d48-b1cb-92a91e190c50/extractor/0.log"
Jul 29 20:14:03.823183 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:14:03.823150    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7np8j/perf-node-gather-daemonset-ljszz"
Jul 29 20:14:04.620209 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:14:04.620171    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d84699fcf-kk7qz_f034f84d-5306-43a8-914d-0a91c0d60b4e/manager/0.log"
Jul 29 20:14:04.636366 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:14:04.636335    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d84699fcf-kk7qz_f034f84d-5306-43a8-914d-0a91c0d60b4e/copy-manifests/0.log"
Jul 29 20:14:06.193198 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:14:06.193072    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-ffdbfbbc8-nd2q7_189375cf-ed59-4af0-8b17-0813e2a39795/manager/0.log"
Jul 29 20:14:06.446716 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:14:06.446597    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d594574db-l29np_3518470c-c9c6-4fce-b7ee-399de7938787/kube-auth-proxy/0.log"
Jul 29 20:14:06.525123 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:14:06.525092    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-tj96m_31e35ba1-a9ef-4cb3-b1a2-aa983501d6bf/payload-pre-processing/0.log"
Jul 29 20:14:06.561758 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:14:06.561730    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7fc6497669-xk7lp_dd24fa62-fd72-4e18-af46-6930625f22da/router/0.log"
Jul 29 20:14:08.118777 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:14:08.118752    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d594574db-l29np_3518470c-c9c6-4fce-b7ee-399de7938787/kube-auth-proxy/0.log"
Jul 29 20:14:08.196277 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:14:08.196229    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-tj96m_31e35ba1-a9ef-4cb3-b1a2-aa983501d6bf/payload-pre-processing/0.log"
Jul 29 20:14:08.235659 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:14:08.235588    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7fc6497669-xk7lp_dd24fa62-fd72-4e18-af46-6930625f22da/router/0.log"
Jul 29 20:14:12.914263 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:14:12.914233    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v67cp_08c90b03-c2a2-478b-8d59-bb71b15e96f8/kube-multus-additional-cni-plugins/0.log"
Jul 29 20:14:12.932350 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:14:12.932281    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v67cp_08c90b03-c2a2-478b-8d59-bb71b15e96f8/egress-router-binary-copy/0.log"
Jul 29 20:14:12.953770 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:14:12.953744    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v67cp_08c90b03-c2a2-478b-8d59-bb71b15e96f8/cni-plugins/0.log"
Jul 29 20:14:12.972408 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:14:12.972385    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v67cp_08c90b03-c2a2-478b-8d59-bb71b15e96f8/bond-cni-plugin/0.log"
Jul 29 20:14:12.994054 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:14:12.994033    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v67cp_08c90b03-c2a2-478b-8d59-bb71b15e96f8/routeoverride-cni/0.log"
Jul 29 20:14:13.012171 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:14:13.012150    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v67cp_08c90b03-c2a2-478b-8d59-bb71b15e96f8/whereabouts-cni-bincopy/0.log"
Jul 29 20:14:13.034633 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:14:13.034607    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v67cp_08c90b03-c2a2-478b-8d59-bb71b15e96f8/whereabouts-cni/0.log"
Jul 29 20:14:13.076678 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:14:13.076652    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-c528d_4c408e45-c274-45fa-9aab-8c6a4c6fdd4e/kube-multus/0.log"
Jul 29 20:14:13.161125 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:14:13.161097    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-cw75m_a4f202a3-5dc7-416f-8793-e857e6086174/network-metrics-daemon/0.log"
Jul 29 20:14:13.181146 ip-10-0-128-173 kubenswrapper[2557]: I0729 20:14:13.181121    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-cw75m_a4f202a3-5dc7-416f-8793-e857e6086174/kube-rbac-proxy/0.log"