Sep 10 20:35:59.925464 ip-10-0-141-185 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 20:36:00.348551 ip-10-0-141-185 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 20:36:00.348551 ip-10-0-141-185 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 20:36:00.348551 ip-10-0-141-185 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 20:36:00.348551 ip-10-0-141-185 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 20:36:00.348551 ip-10-0-141-185 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 20:36:00.349614 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.349550    2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 20:36:00.355258 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355244    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:36:00.355258 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355259    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:36:00.355323 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355263    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:36:00.355323 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355267    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:36:00.355323 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355270    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:36:00.355323 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355278    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:36:00.355323 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355281    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:36:00.355323 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355284    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:36:00.355323 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355287    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:36:00.355323 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355290    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:36:00.355323 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355293    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:36:00.355323 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355296    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:36:00.355323 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355298    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:36:00.355323 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355301    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:36:00.355323 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355303    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:36:00.355323 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355306    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:36:00.355323 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355309    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:36:00.355323 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355311    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:36:00.355323 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355314    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:36:00.355323 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355316    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:36:00.355323 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355319    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:36:00.355323 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355322    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:36:00.355800 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355324    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:36:00.355800 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355327    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:36:00.355800 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355330    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:36:00.355800 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355333    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:36:00.355800 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355336    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:36:00.355800 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355339    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:36:00.355800 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355342    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:36:00.355800 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355345    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:36:00.355800 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355347    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:36:00.355800 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355349    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:36:00.355800 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355352    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:36:00.355800 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355355    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:36:00.355800 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355357    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:36:00.355800 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355360    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:36:00.355800 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355362    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:36:00.355800 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355365    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:36:00.355800 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355367    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:36:00.355800 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355370    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:36:00.355800 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355372    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:36:00.355800 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355374    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:36:00.356330 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355377    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:36:00.356330 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355379    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:36:00.356330 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355381    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:36:00.356330 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355384    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:36:00.356330 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355386    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:36:00.356330 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355389    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:36:00.356330 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355391    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:36:00.356330 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355394    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:36:00.356330 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355398    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:36:00.356330 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355402    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:36:00.356330 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355405    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:36:00.356330 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355408    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:36:00.356330 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355411    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:36:00.356330 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355414    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:36:00.356330 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355416    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:36:00.356330 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355419    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:36:00.356330 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355421    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:36:00.356330 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355424    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:36:00.356330 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355426    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:36:00.356786 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355429    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:36:00.356786 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355431    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:36:00.356786 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355434    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:36:00.356786 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355437    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:36:00.356786 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355440    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:36:00.356786 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355442    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:36:00.356786 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355445    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:36:00.356786 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355447    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:36:00.356786 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355450    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:36:00.356786 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355452    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:36:00.356786 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355455    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:36:00.356786 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355457    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:36:00.356786 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355460    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:36:00.356786 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355462    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:36:00.356786 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355465    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:36:00.356786 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355467    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:36:00.356786 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355470    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:36:00.356786 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355472    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:36:00.356786 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355475    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:36:00.357272 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355478    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:36:00.357272 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355481    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:36:00.357272 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355483    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:36:00.357272 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355486    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:36:00.357272 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355489    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:36:00.357272 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355493    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:36:00.357272 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355496    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:36:00.357272 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355500    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:36:00.357272 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355503    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:36:00.357272 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355506    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:36:00.357272 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355509    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:36:00.357272 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355512    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:36:00.357272 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355515    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:36:00.357272 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355519    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:36:00.357272 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355522    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:36:00.357272 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355525    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:36:00.357272 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355528    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:36:00.357272 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355531    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:36:00.357272 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.355534    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:36:00.357764 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356210    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:36:00.357764 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356216    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:36:00.357764 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356219    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:36:00.357764 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356222    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:36:00.357764 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356225    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:36:00.357764 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356228    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:36:00.357764 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356231    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:36:00.357764 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356234    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:36:00.357764 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356237    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:36:00.357764 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356240    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:36:00.357764 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356243    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:36:00.357764 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356245    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:36:00.357764 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356248    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:36:00.357764 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356251    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:36:00.357764 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356253    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:36:00.357764 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356256    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:36:00.357764 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356259    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:36:00.357764 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356261    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:36:00.357764 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356264    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:36:00.357764 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356268    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:36:00.358308 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356271    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:36:00.358308 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356274    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:36:00.358308 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356277    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:36:00.358308 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356280    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:36:00.358308 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356282    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:36:00.358308 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356285    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:36:00.358308 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356287    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:36:00.358308 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356290    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:36:00.358308 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356293    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:36:00.358308 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356295    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:36:00.358308 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356298    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:36:00.358308 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356301    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:36:00.358308 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356304    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:36:00.358308 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356306    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:36:00.358308 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356311    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:36:00.358308 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356314    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:36:00.358308 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356317    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:36:00.358308 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356320    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:36:00.358308 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356323    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:36:00.358778 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356325    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:36:00.358778 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356328    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:36:00.358778 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356332    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:36:00.358778 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356335    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:36:00.358778 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356337    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:36:00.358778 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356340    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:36:00.358778 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356343    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:36:00.358778 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356346    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:36:00.358778 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356348    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:36:00.358778 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356351    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:36:00.358778 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356353    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:36:00.358778 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356356    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:36:00.358778 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356359    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:36:00.358778 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356362    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:36:00.358778 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356365    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:36:00.358778 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356368    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:36:00.358778 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356370    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:36:00.358778 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356373    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:36:00.358778 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356375    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:36:00.358778 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356378    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:36:00.359286 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356381    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:36:00.359286 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356383    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:36:00.359286 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356386    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:36:00.359286 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356388    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:36:00.359286 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356391    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:36:00.359286 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356394    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:36:00.359286 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356396    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:36:00.359286 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356399    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:36:00.359286 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356401    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:36:00.359286 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356404    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:36:00.359286 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356406    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:36:00.359286 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356409    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:36:00.359286 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356411    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:36:00.359286 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356414    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:36:00.359286 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356416    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:36:00.359286 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356419    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:36:00.359286 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356421    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:36:00.359286 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356424    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:36:00.359286 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356426    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:36:00.359286 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356429    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:36:00.359775 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356431    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:36:00.359775 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356434    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:36:00.359775 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356437    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:36:00.359775 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356439    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:36:00.359775 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356442    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:36:00.359775 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356445    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:36:00.359775 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356447    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:36:00.359775 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356453    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:36:00.359775 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356456    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:36:00.359775 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356459    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:36:00.359775 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356462    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:36:00.359775 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356465    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:36:00.359775 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356469    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:36:00.359775 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356471    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:36:00.359775 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356474    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:36:00.359775 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356477    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:36:00.359775 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356479    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:36:00.359775 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356482    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:36:00.359775 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356485    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:36:00.360252 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356488    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:36:00.360252 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356812    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:36:00.360252 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356817    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:36:00.360252 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356820    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:36:00.360252 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356822    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:36:00.360252 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356825    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:36:00.360252 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356828    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:36:00.360252 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356831    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:36:00.360252 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356834    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:36:00.360252 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356836    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:36:00.360252 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356839    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:36:00.360252 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356842    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:36:00.360252 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356844    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:36:00.360252 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356847    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:36:00.360252 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356850    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:36:00.360252 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356852    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:36:00.360252 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356855    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:36:00.360252 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356857    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:36:00.360252 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356860    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:36:00.360252 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356863    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:36:00.360739 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356865    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:36:00.360739 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356868    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:36:00.360739 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356870    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:36:00.360739 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356873    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:36:00.360739 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356875    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:36:00.360739 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356878    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:36:00.360739 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356881    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:36:00.360739 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356883    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:36:00.360739 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356886    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:36:00.360739 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356889    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:36:00.360739 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356891    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:36:00.360739 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356894    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:36:00.360739 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356897    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:36:00.360739 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356899    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:36:00.360739 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356902    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:36:00.360739 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356904    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:36:00.360739 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356906    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:36:00.360739 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356909    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:36:00.360739 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356911    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:36:00.360739 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356914    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:36:00.361229 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356916    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:36:00.361229 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356919    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:36:00.361229 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356921    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:36:00.361229 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356924    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:36:00.361229 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356926    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:36:00.361229 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356929    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:36:00.361229 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356932    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:36:00.361229 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356934    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:36:00.361229 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356937    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:36:00.361229 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356940    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:36:00.361229 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356943    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:36:00.361229 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356946    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:36:00.361229 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356948    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:36:00.361229 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356951    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:36:00.361229 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356954    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:36:00.361229 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356957    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:36:00.361229 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356959    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:36:00.361229 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356962    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:36:00.361229 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356965    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:36:00.361229 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356967    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:36:00.361731 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356970    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:36:00.361731 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356972    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:36:00.361731 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356975    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:36:00.361731 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356977    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:36:00.361731 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356980    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:36:00.361731 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356982    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:36:00.361731 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356985    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:36:00.361731 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356987    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:36:00.361731 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356990    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:36:00.361731 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356992    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:36:00.361731 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356995    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:36:00.361731 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.356997    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:36:00.361731 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357000    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:36:00.361731 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357002    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:36:00.361731 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357005    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:36:00.361731 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357007    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:36:00.361731 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357010    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:36:00.361731 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357013    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:36:00.361731 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357015    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:36:00.362198 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357018    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:36:00.362198 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357020    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:36:00.362198 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357023    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:36:00.362198 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357026    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:36:00.362198 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357028    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:36:00.362198 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357031    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:36:00.362198 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357054    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:36:00.362198 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357058    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:36:00.362198 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357063    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:36:00.362198 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357068    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:36:00.362198 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357071    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:36:00.362198 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357075    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:36:00.362198 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357079    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:36:00.362198 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357082    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:36:00.362198 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357085    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:36:00.362198 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357088    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:36:00.362198 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357091    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:36:00.362198 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357094    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:36:00.362198 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357097    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:36:00.362692 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357099    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:36:00.362692 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357102    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:36:00.362692 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357181    2564 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 20:36:00.362692 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357215    2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 20:36:00.362692 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357221    2564 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 20:36:00.362692 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357224    2564 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 20:36:00.362692 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357228    2564 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 20:36:00.362692 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357231    2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 20:36:00.362692 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357235    2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 20:36:00.362692 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357239    2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 20:36:00.362692 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357242    2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 20:36:00.362692 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357245    2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 20:36:00.362692 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357249    2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 20:36:00.362692 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357252    2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 20:36:00.362692 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357255    2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 20:36:00.362692 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357258    2564 flags.go:64] FLAG: --cgroup-root=""
Sep 10 20:36:00.362692 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357261    2564 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 20:36:00.362692 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357264    2564 flags.go:64] FLAG: --client-ca-file=""
Sep 10 20:36:00.362692 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357267    2564 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 20:36:00.362692 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357270    2564 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 20:36:00.362692 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357275    2564 flags.go:64] FLAG: --cluster-domain=""
Sep 10 20:36:00.362692 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357278    2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 20:36:00.362692 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357281    2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 20:36:00.362692 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357284    2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 20:36:00.363310 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357288    2564 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 20:36:00.363310 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357291    2564 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 20:36:00.363310 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357294    2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 20:36:00.363310 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357297    2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 20:36:00.363310 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357300    2564 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 20:36:00.363310 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357303    2564 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 20:36:00.363310 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357306    2564 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 20:36:00.363310 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357309    2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 20:36:00.363310 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357312    2564 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 20:36:00.363310 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357317    2564 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 20:36:00.363310 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357321    2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 20:36:00.363310 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357324    2564 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 20:36:00.363310 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357327    2564 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 20:36:00.363310 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357330    2564 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 20:36:00.363310 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357333    2564 flags.go:64] FLAG: --enable-server="true"
Sep 10 20:36:00.363310 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357336    2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 20:36:00.363310 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357340    2564 flags.go:64] FLAG: --event-burst="100"
Sep 10 20:36:00.363310 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357342    2564 flags.go:64] FLAG: --event-qps="50"
Sep 10 20:36:00.363310 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357345    2564 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 20:36:00.363310 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357348    2564 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 20:36:00.363310 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357351    2564 flags.go:64] FLAG: --eviction-hard=""
Sep 10 20:36:00.363310 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357354    2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 20:36:00.363310 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357357    2564 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 20:36:00.363310 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357360    2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 20:36:00.363310 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357363    2564 flags.go:64] FLAG: --eviction-soft=""
Sep 10 20:36:00.363947 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357366    2564 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 20:36:00.363947 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357369    2564 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 20:36:00.363947 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357372    2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 20:36:00.363947 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357374    2564 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 20:36:00.363947 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357377    2564 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 20:36:00.363947 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357380    2564 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 20:36:00.363947 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357382    2564 flags.go:64] FLAG: --feature-gates=""
Sep 10 20:36:00.363947 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357386    2564 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 20:36:00.363947 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357389    2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 20:36:00.363947 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357392    2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 20:36:00.363947 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357395    2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 20:36:00.363947 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357397    2564 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 20:36:00.363947 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357400    2564 flags.go:64] FLAG: --help="false"
Sep 10 20:36:00.363947 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357403    2564 flags.go:64] FLAG: --hostname-override="ip-10-0-141-185.ec2.internal"
Sep 10 20:36:00.363947 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357406    2564 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 20:36:00.363947 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357409    2564 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 20:36:00.363947 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357413    2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 20:36:00.363947 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357417    2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 20:36:00.363947 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357420    2564 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 20:36:00.363947 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357423    2564 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 20:36:00.363947 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357425    2564 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 20:36:00.363947 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357428    2564 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 20:36:00.363947 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357431    2564 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 20:36:00.364570 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357434    2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 20:36:00.364570 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357437    2564 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 20:36:00.364570 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357439    2564 flags.go:64] FLAG: --kube-reserved=""
Sep 10 20:36:00.364570 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357443    2564 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 20:36:00.364570 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357445    2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 20:36:00.364570 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357449    2564 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 20:36:00.364570 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357452    2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 20:36:00.364570 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357455    2564 flags.go:64] FLAG: --lock-file=""
Sep 10 20:36:00.364570 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357457    2564 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 20:36:00.364570 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357460    2564 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 20:36:00.364570 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357463    2564 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 20:36:00.364570 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357468    2564 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 20:36:00.364570 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357471    2564 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 20:36:00.364570 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357474    2564 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 20:36:00.364570 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357476    2564 flags.go:64] FLAG: --logging-format="text"
Sep 10 20:36:00.364570 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357479    2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 20:36:00.364570 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357482    2564 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 20:36:00.364570 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357485    2564 flags.go:64] FLAG: --manifest-url=""
Sep 10 20:36:00.364570 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357488    2564 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 20:36:00.364570 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357492    2564 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 20:36:00.364570 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357495    2564 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 20:36:00.364570 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357499    2564 flags.go:64] FLAG: --max-pods="110"
Sep 10 20:36:00.364570 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357501    2564 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 20:36:00.364570 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357505    2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 20:36:00.364570 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357508    2564 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 20:36:00.365265 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357513    2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 20:36:00.365265 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357517    2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 20:36:00.365265 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357520    2564 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 20:36:00.365265 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357522    2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 20:36:00.365265 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357529    2564 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 20:36:00.365265 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357532    2564 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 20:36:00.365265 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357535    2564 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 20:36:00.365265 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357538    2564 flags.go:64] FLAG: --pod-cidr=""
Sep 10 20:36:00.365265 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357541    2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 10 20:36:00.365265 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357546    2564 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 20:36:00.365265 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357551    2564 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 20:36:00.365265 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357554    2564 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 20:36:00.365265 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357557    2564 flags.go:64] FLAG: --port="10250"
Sep 10 20:36:00.365265 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357560    2564 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 20:36:00.365265 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357563    2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-006ec9c071085c5ac"
Sep 10 20:36:00.365265 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357566    2564 flags.go:64] FLAG: --qos-reserved=""
Sep 10 20:36:00.365265 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357569    2564 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 20:36:00.365265 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357572    2564 flags.go:64] FLAG: --register-node="true"
Sep 10 20:36:00.365265 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357575    2564 flags.go:64] FLAG: --register-with-taints=""
Sep 10 20:36:00.365265 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357578    2564 flags.go:64] FLAG: --registry-burst="10"
Sep 10 20:36:00.365265 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357581    2564 flags.go:64] FLAG: --registry-qps="5"
Sep 10 20:36:00.365265 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357584    2564 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 20:36:00.365265 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357587    2564 flags.go:64] FLAG: --reserved-memory=""
Sep 10 20:36:00.365265 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357590    2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 20:36:00.365265 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357593    2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 20:36:00.366235 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357596    2564 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 20:36:00.366235 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357599    2564 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 20:36:00.366235 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357602    2564 flags.go:64] FLAG: --runonce="false"
Sep 10 20:36:00.366235 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357605    2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 20:36:00.366235 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357608    2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 20:36:00.366235 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357610    2564 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 20:36:00.366235 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357613    2564 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 20:36:00.366235 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357616    2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 20:36:00.366235 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357621    2564 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 20:36:00.366235 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357624    2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 20:36:00.366235 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357627    2564 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 20:36:00.366235 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357630    2564 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 20:36:00.366235 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357632    2564 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 20:36:00.366235 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357635    2564 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 20:36:00.366235 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357638    2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 20:36:00.366235 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357641    2564 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 20:36:00.366235 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357644    2564 flags.go:64] FLAG: --system-cgroups=""
Sep 10 20:36:00.366235 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357648    2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 20:36:00.366235 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357653    2564 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 20:36:00.366235 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357655    2564 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 20:36:00.366235 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357658    2564 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 20:36:00.366235 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357662    2564 flags.go:64] FLAG: --tls-min-version=""
Sep 10 20:36:00.366235 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357665    2564 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 20:36:00.366235 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357668    2564 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 20:36:00.366235 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357671    2564 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 20:36:00.366850 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357674    2564 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 20:36:00.366850 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357677    2564 flags.go:64] FLAG: --v="2"
Sep 10 20:36:00.366850 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357680    2564 flags.go:64] FLAG: --version="false"
Sep 10 20:36:00.366850 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357684    2564 flags.go:64] FLAG: --vmodule=""
Sep 10 20:36:00.366850 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357687    2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 20:36:00.366850 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.357690    2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 20:36:00.366850 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357791    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:36:00.366850 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357796    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:36:00.366850 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357799    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:36:00.366850 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357803    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:36:00.366850 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357806    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:36:00.366850 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357809    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:36:00.366850 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357812    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:36:00.366850 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357815    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:36:00.366850 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357818    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:36:00.366850 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357821    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:36:00.366850 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357824    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:36:00.366850 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357827    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:36:00.366850 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357829    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:36:00.366850 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357832    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:36:00.366850 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357834    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:36:00.366850 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357837    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:36:00.367409 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357839    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:36:00.367409 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357842    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:36:00.367409 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357846    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:36:00.367409 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357849    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:36:00.367409 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357851    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:36:00.367409 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357853    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:36:00.367409 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357856    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:36:00.367409 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357858    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:36:00.367409 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357861    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:36:00.367409 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357864    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:36:00.367409 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357866    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:36:00.367409 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357869    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:36:00.367409 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357871    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:36:00.367409 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357874    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:36:00.367409 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357876    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:36:00.367409 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357879    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:36:00.367409 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357882    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:36:00.367409 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357884    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:36:00.367409 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357887    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:36:00.367409 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357889    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:36:00.367935 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357892    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:36:00.367935 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357894    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:36:00.367935 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357897    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:36:00.367935 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357899    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:36:00.367935 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357902    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:36:00.367935 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357905    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:36:00.367935 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357908    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:36:00.367935 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357912    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:36:00.367935 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357916    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:36:00.367935 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357919    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:36:00.367935 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357922    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:36:00.367935 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357925    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:36:00.367935 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357927    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:36:00.367935 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357930    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:36:00.367935 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357934    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:36:00.367935 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357936    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:36:00.367935 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357939    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:36:00.367935 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357942    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:36:00.367935 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357944    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:36:00.368432 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357947    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:36:00.368432 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357949    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:36:00.368432 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357952    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:36:00.368432 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357955    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:36:00.368432 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357957    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:36:00.368432 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357960    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:36:00.368432 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357962    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:36:00.368432 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357965    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:36:00.368432 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357968    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:36:00.368432 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357970    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:36:00.368432 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357973    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:36:00.368432 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357975    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:36:00.368432 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357977    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:36:00.368432 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357980    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:36:00.368432 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357982    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:36:00.368432 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357985    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:36:00.368432 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357988    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:36:00.368432 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357990    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:36:00.368432 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357993    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:36:00.368907 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357995    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:36:00.368907 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.357998    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:36:00.368907 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.358001    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:36:00.368907 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.358005    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:36:00.368907 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.358008    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:36:00.368907 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.358011    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:36:00.368907 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.358014    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:36:00.368907 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.358016    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:36:00.368907 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.358020    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:36:00.368907 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.358023    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:36:00.368907 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.358025    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:36:00.368907 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.358028    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:36:00.368907 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.358032    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:36:00.368907 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.358035    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:36:00.368907 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.358037    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:36:00.368907 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.358040    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:36:00.368907 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.358043    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:36:00.368907 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.358045    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:36:00.368907 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.358048    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:36:00.369389 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.358050    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:36:00.369389 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.358053    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:36:00.369389 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.358055    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:36:00.369389 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.358058    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:36:00.369389 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.358060    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:36:00.369389 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.358063    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:36:00.369389 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.358794    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 20:36:00.369389 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.366882    2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 20:36:00.369389 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.366979    2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 20:36:00.369389 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.367890    2564 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 20:36:00.369389 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.367905    2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 20:36:00.369389 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.367961    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:36:00.369389 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.367966    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:36:00.369389 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.367970    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:36:00.369389 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.367973    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:36:00.369765 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.367976    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:36:00.369765 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.367979    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:36:00.369765 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.367983    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:36:00.369765 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.367985    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:36:00.369765 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.367988    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:36:00.369765 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.367992    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:36:00.369765 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.367994    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:36:00.369765 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.367997    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:36:00.369765 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368001    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:36:00.369765 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368004    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:36:00.369765 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368007    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:36:00.369765 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368010    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:36:00.369765 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368012    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:36:00.369765 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368015    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:36:00.369765 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368018    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:36:00.369765 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368020    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:36:00.369765 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368024    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:36:00.369765 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368027    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:36:00.369765 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368029    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:36:00.369765 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368032    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:36:00.370356 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368035    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:36:00.370356 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368037    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:36:00.370356 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368040    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:36:00.370356 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368043    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:36:00.370356 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368045    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:36:00.370356 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368048    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:36:00.370356 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368051    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:36:00.370356 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368054    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:36:00.370356 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368056    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:36:00.370356 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368059    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:36:00.370356 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368061    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:36:00.370356 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368064    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:36:00.370356 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368067    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:36:00.370356 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368070    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:36:00.370356 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368072    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:36:00.370356 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368075    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:36:00.370356 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368078    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:36:00.370356 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368081    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:36:00.370356 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368083    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:36:00.370356 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368086    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:36:00.370854 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368089    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:36:00.370854 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368091    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:36:00.370854 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368093    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:36:00.370854 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368096    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:36:00.370854 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368099    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:36:00.370854 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368102    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:36:00.370854 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368104    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:36:00.370854 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368107    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:36:00.370854 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368109    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:36:00.370854 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368112    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:36:00.370854 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368114    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:36:00.370854 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368117    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:36:00.370854 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368120    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:36:00.370854 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368122    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:36:00.370854 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368125    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:36:00.370854 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368128    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:36:00.370854 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368131    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:36:00.370854 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368133    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:36:00.370854 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368136    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:36:00.370854 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368138    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:36:00.371384 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368141    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:36:00.371384 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368143    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:36:00.371384 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368146    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:36:00.371384 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368148    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:36:00.371384 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368151    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:36:00.371384 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368154    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:36:00.371384 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368156    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:36:00.371384 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368159    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:36:00.371384 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368162    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:36:00.371384 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368180    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:36:00.371384 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368183    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:36:00.371384 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368186    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:36:00.371384 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368188    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:36:00.371384 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368191    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:36:00.371384 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368193    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:36:00.371384 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368199    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:36:00.371384 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368203    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:36:00.371384 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368206    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:36:00.371384 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368208    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:36:00.371384 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368211    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:36:00.371861 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368214    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:36:00.371861 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368218    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:36:00.371861 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368221    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:36:00.371861 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368224    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:36:00.371861 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368227    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:36:00.371861 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368230    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:36:00.371861 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368233    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:36:00.371861 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368235    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:36:00.371861 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368239    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:36:00.371861 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368242    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:36:00.371861 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368245    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:36:00.371861 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368248    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:36:00.371861 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368251    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:36:00.371861 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368254    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:36:00.371861 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368257    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:36:00.372233 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.368262    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 20:36:00.372233 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368378    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:36:00.372233 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368384    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:36:00.372233 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368387    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:36:00.372233 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368390    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:36:00.372233 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368393    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:36:00.372233 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368397    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:36:00.372233 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368399    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:36:00.372233 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368402    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:36:00.372233 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368405    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:36:00.372233 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368407    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:36:00.372233 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368410    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:36:00.372233 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368412    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:36:00.372233 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368416    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:36:00.372233 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368418    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:36:00.372594 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368421    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:36:00.372594 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368424    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:36:00.372594 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368426    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:36:00.372594 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368429    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:36:00.372594 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368431    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:36:00.372594 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368434    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:36:00.372594 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368436    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:36:00.372594 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368439    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:36:00.372594 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368442    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:36:00.372594 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368445    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:36:00.372594 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368447    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:36:00.372594 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368450    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:36:00.372594 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368452    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:36:00.372594 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368455    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:36:00.372594 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368457    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:36:00.372594 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368460    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:36:00.372594 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368462    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:36:00.372594 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368465    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:36:00.372594 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368467    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:36:00.372594 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368470    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:36:00.373076 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368473    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:36:00.373076 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368475    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:36:00.373076 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368479    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:36:00.373076 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368481    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:36:00.373076 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368484    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:36:00.373076 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368486    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:36:00.373076 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368489    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:36:00.373076 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368491    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:36:00.373076 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368494    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:36:00.373076 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368496    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:36:00.373076 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368499    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:36:00.373076 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368502    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:36:00.373076 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368505    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:36:00.373076 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368507    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:36:00.373076 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368511    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:36:00.373076 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368514    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:36:00.373076 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368517    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:36:00.373076 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368520    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:36:00.373076 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368522    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:36:00.373547 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368525    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:36:00.373547 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368528    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:36:00.373547 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368530    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:36:00.373547 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368533    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:36:00.373547 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368536    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:36:00.373547 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368538    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:36:00.373547 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368541    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:36:00.373547 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368543    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:36:00.373547 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368546    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:36:00.373547 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368548    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:36:00.373547 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368551    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:36:00.373547 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368553    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:36:00.373547 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368556    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:36:00.373547 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368559    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:36:00.373547 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368562    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:36:00.373547 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368564    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:36:00.373547 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368567    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:36:00.373547 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368570    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:36:00.373547 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368572    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:36:00.373547 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368575    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:36:00.374024 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368577    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:36:00.374024 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368579    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:36:00.374024 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368582    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:36:00.374024 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368584    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:36:00.374024 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368588    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:36:00.374024 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368592    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:36:00.374024 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368594    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:36:00.374024 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368597    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:36:00.374024 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368599    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:36:00.374024 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368602    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:36:00.374024 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368604    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:36:00.374024 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368607    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:36:00.374024 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368609    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:36:00.374024 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368612    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:36:00.374024 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368614    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:36:00.374024 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368617    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:36:00.374024 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368619    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:36:00.374024 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368622    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:36:00.374024 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368624    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:36:00.374539 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368628    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:36:00.374539 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368631    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:36:00.374539 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368635    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:36:00.374539 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368638    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:36:00.374539 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368641    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:36:00.374539 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368644    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:36:00.374539 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:00.368647    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:36:00.374539 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.368653    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 20:36:00.374539 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.368739    2564 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 20:36:00.374539 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.372802    2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 20:36:00.374539 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.373832    2564 server.go:1019] "Starting client certificate rotation"
Sep 10 20:36:00.374539 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.373926    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 20:36:00.374539 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.373962    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 20:36:00.377492 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.377479    2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 20:36:00.378536 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.378514    2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 20:36:00.390463 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.390450    2564 log.go:25] "Validated CRI v1 runtime API"
Sep 10 20:36:00.393833 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.393818    2564 log.go:25] "Validated CRI v1 image API"
Sep 10 20:36:00.394629 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.394611    2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 20:36:00.398950 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.398933    2564 fs.go:135] Filesystem UUIDs: map[4b77988d-33b4-448a-b2c7-7ad85605950a:/dev/nvme0n1p3 6c074524-116e-4d3a-a64f-2382b9f5a4e6:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 10 20:36:00.399015 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.398950    2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 20:36:00.402781 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.402674    2564 manager.go:216] Machine: {Timestamp:2026-09-10 20:36:00.40223354 +0000 UTC m=+0.367301726 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3115246 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec289e8811049702bd0ea694a081ae53 SystemUUID:ec289e88-1104-9702-bd0e-a694a081ae53 BootID:ea3f2c13-099d-4b68-9753-f4eba6ed476e Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ab:64:3c:1e:c5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ab:64:3c:1e:c5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:a8:9b:2d:6a:2b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 20:36:00.402781 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.402771    2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 20:36:00.402877 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.402856    2564 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 20:36:00.404392 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.404330    2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 20:36:00.404656 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.404366    2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-185.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 20:36:00.404733 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.404635    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 20:36:00.406301 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.406285    2564 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 20:36:00.406358 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.406314    2564 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 20:36:00.406358 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.406328    2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 20:36:00.407195 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.407183    2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 20:36:00.408505 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.408493    2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 20:36:00.413222 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.413207    2564 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 20:36:00.413351 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.413339    2564 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 20:36:00.413406 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.413376    2564 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 20:36:00.413406 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.413390    2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 20:36:00.413494 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.413409    2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 20:36:00.413494 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.413422    2564 kubelet.go:398] "Adding apiserver pod source"
Sep 10 20:36:00.413494 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.413434    2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 20:36:00.413800 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.413783    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 20:36:00.413871 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.413806    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 20:36:00.416772 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.416756    2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 20:36:00.418925 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.418913    2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 20:36:00.418978 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.418942    2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 20:36:00.420987 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.420974    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 20:36:00.420987 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.420989    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 20:36:00.421104 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.420995    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 20:36:00.421104 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.421000    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 20:36:00.421104 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.421009    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 20:36:00.421104 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.421014    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 20:36:00.421104 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.421028    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 20:36:00.421104 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.421035    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 20:36:00.421104 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.421044    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 20:36:00.421104 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.421049    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 20:36:00.421104 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.421057    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 20:36:00.421104 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.421062    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 20:36:00.421104 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.421086    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 20:36:00.421104 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.421094    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 20:36:00.424784 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.424769    2564 server.go:1268] "Started kubelet"
Sep 10 20:36:00.424871 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.424817    2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 20:36:00.424929 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.424852    2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 20:36:00.424929 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.424916    2564 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 20:36:00.425374 ip-10-0-141-185 systemd[1]: Started Kubernetes Kubelet.
Sep 10 20:36:00.426203 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:00.426178    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-185.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 20:36:00.426276 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:00.426202    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 20:36:00.426322 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.426278    2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-185.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 20:36:00.427908 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.426954    2564 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 20:36:00.427908 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.427011    2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 20:36:00.430625 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:00.430606    2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 20:36:00.430794 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:00.429602    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-185.ec2.internal.18d41003c983e2fb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-185.ec2.internal,UID:ip-10-0-141-185.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-185.ec2.internal,},FirstTimestamp:2026-09-10 20:36:00.424739579 +0000 UTC m=+0.389807769,LastTimestamp:2026-09-10 20:36:00.424739579 +0000 UTC m=+0.389807769,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-185.ec2.internal,}"
Sep 10 20:36:00.434152 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.434139    2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 20:36:00.434236 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.434152    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 20:36:00.434236 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:00.434220    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-185.ec2.internal\" not found"
Sep 10 20:36:00.435086 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.435068    2564 factory.go:55] Registering systemd factory
Sep 10 20:36:00.435086 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.435087    2564 factory.go:223] Registration of the systemd container factory successfully
Sep 10 20:36:00.435257 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.435135    2564 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 20:36:00.435257 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.435148    2564 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 20:36:00.435355 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.435270    2564 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 20:36:00.435355 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.435277    2564 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 20:36:00.435355 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.435348    2564 factory.go:153] Registering CRI-O factory
Sep 10 20:36:00.435487 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.435360    2564 factory.go:223] Registration of the crio container factory successfully
Sep 10 20:36:00.435487 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.435408    2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 20:36:00.435487 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.435428    2564 factory.go:103] Registering Raw factory
Sep 10 20:36:00.435487 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.435443    2564 manager.go:1196] Started watching for new ooms in manager
Sep 10 20:36:00.435700 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.435676    2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 20:36:00.435867 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.435854    2564 manager.go:349] Starting recovery of all containers
Sep 10 20:36:00.439693 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:00.439666    2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-185.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 20:36:00.439778 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:00.439705    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 20:36:00.447720 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.447702    2564 manager.go:354] Recovery completed
Sep 10 20:36:00.452127 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.452114    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:36:00.453833 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.453810    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-185.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:36:00.453888 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.453845    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-185.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:36:00.453888 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.453859    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-185.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:36:00.454464 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.454451    2564 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 20:36:00.454464 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.454462    2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 20:36:00.454566 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.454475    2564 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 20:36:00.454566 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.454518    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:36:00.455179 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.455151    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-185.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:36:00.455259 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.455196    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-185.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:36:00.455259 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.455206    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-185.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:36:00.456185 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:00.456097    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-185.ec2.internal.18d41003cb3fcc8f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-185.ec2.internal,UID:ip-10-0-141-185.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-185.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-185.ec2.internal,},FirstTimestamp:2026-09-10 20:36:00.453831823 +0000 UTC m=+0.418900008,LastTimestamp:2026-09-10 20:36:00.453831823 +0000 UTC m=+0.418900008,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-185.ec2.internal,}"
Sep 10 20:36:00.456565 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.456551    2564 policy_none.go:49] "None policy: Start"
Sep 10 20:36:00.456627 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.456570    2564 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 20:36:00.456627 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.456583    2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 20:36:00.458620 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.458602    2564 policy_none.go:47] "Start"
Sep 10 20:36:00.464600 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.464582    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7dj8v"
Sep 10 20:36:00.466275 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:00.466215    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-185.ec2.internal.18d41003cb401eb0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-185.ec2.internal,UID:ip-10-0-141-185.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-185.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-185.ec2.internal,},FirstTimestamp:2026-09-10 20:36:00.453852848 +0000 UTC m=+0.418921034,LastTimestamp:2026-09-10 20:36:00.453852848 +0000 UTC m=+0.418921034,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-185.ec2.internal,}"
Sep 10 20:36:00.472571 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.472554    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7dj8v"
Sep 10 20:36:00.479930 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:00.479870    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-185.ec2.internal.18d41003cb404728  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-185.ec2.internal,UID:ip-10-0-141-185.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-185.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-185.ec2.internal,},FirstTimestamp:2026-09-10 20:36:00.453863208 +0000 UTC m=+0.418931393,LastTimestamp:2026-09-10 20:36:00.453863208 +0000 UTC m=+0.418931393,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-185.ec2.internal,}"
Sep 10 20:36:00.483776 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.483762    2564 manager.go:337] "Starting Device Plugin manager"
Sep 10 20:36:00.483855 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:00.483793    2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 20:36:00.483855 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.483802    2564 server.go:87] "Starting device plugin registration server"
Sep 10 20:36:00.484063 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.484050    2564 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 20:36:00.484120 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.484066    2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 20:36:00.484230 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.484216    2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 20:36:00.484355 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.484340    2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 20:36:00.484355 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.484351    2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 20:36:00.484777 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:00.484756    2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 20:36:00.484856 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:00.484810    2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-185.ec2.internal\" not found"
Sep 10 20:36:00.489868 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:00.489805    2564 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-141-185.ec2.internal.18d41003cb3fcc8f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-185.ec2.internal.18d41003cb3fcc8f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-185.ec2.internal,UID:ip-10-0-141-185.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-185.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-185.ec2.internal,},FirstTimestamp:2026-09-10 20:36:00.453831823 +0000 UTC m=+0.418900008,LastTimestamp:2026-09-10 20:36:00.455180946 +0000 UTC m=+0.420249135,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-185.ec2.internal,}"
Sep 10 20:36:00.601399 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:00.497613    2564 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-141-185.ec2.internal.18d41003cb401eb0\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-185.ec2.internal.18d41003cb401eb0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-185.ec2.internal,UID:ip-10-0-141-185.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-185.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-185.ec2.internal,},FirstTimestamp:2026-09-10 20:36:00.453852848 +0000 UTC m=+0.418921034,LastTimestamp:2026-09-10 20:36:00.455201348 +0000 UTC m=+0.420269538,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-185.ec2.internal,}"
Sep 10 20:36:00.601399 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:00.509745    2564 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-141-185.ec2.internal.18d41003cb404728\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-185.ec2.internal.18d41003cb404728  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-185.ec2.internal,UID:ip-10-0-141-185.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-185.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-185.ec2.internal,},FirstTimestamp:2026-09-10 20:36:00.453863208 +0000 UTC m=+0.418931393,LastTimestamp:2026-09-10 20:36:00.455210325 +0000 UTC m=+0.420278511,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-185.ec2.internal,}"
Sep 10 20:36:00.601399 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:00.521116    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-185.ec2.internal.18d41003cdbd3f31  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-185.ec2.internal,UID:ip-10-0-141-185.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-141-185.ec2.internal,},FirstTimestamp:2026-09-10 20:36:00.495607601 +0000 UTC m=+0.460675774,LastTimestamp:2026-09-10 20:36:00.495607601 +0000 UTC m=+0.460675774,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-185.ec2.internal,}"
Sep 10 20:36:00.601399 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.533035    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 20:36:00.601399 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.534138    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 20:36:00.601399 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.534193    2564 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 20:36:00.601399 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.534217    2564 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 20:36:00.601399 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:00.534250    2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 20:36:00.601399 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:00.543690    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 10 20:36:00.601399 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.584763    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:36:00.601399 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.585463    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-185.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:36:00.601399 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.585492    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-185.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:36:00.601399 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.585510    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-185.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:36:00.601399 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.585526    2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-185.ec2.internal"
Sep 10 20:36:00.601399 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:00.596758    2564 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-141-185.ec2.internal.18d41003cb3fcc8f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-185.ec2.internal.18d41003cb3fcc8f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-185.ec2.internal,UID:ip-10-0-141-185.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-185.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-185.ec2.internal,},FirstTimestamp:2026-09-10 20:36:00.453831823 +0000 UTC m=+0.418900008,LastTimestamp:2026-09-10 20:36:00.585475632 +0000 UTC m=+0.550543821,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-185.ec2.internal,}"
Sep 10 20:36:00.609873 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:00.609855    2564 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-141-185.ec2.internal\" not found" node="ip-10-0-141-185.ec2.internal"
Sep 10 20:36:00.635205 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.635185    2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-185.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-185.ec2.internal"]
Sep 10 20:36:00.635289 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.635245    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:36:00.636281 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.636268    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-185.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:36:00.636349 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.636292    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-185.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:36:00.636349 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.636305    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-185.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:36:00.638733 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.638720    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:36:00.638853 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.638841    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-185.ec2.internal"
Sep 10 20:36:00.638890 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.638866    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:36:00.639350 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.639334    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-185.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:36:00.639411 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.639357    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-185.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:36:00.639411 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.639368    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-185.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:36:00.639411 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.639378    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-185.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:36:00.639411 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.639395    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-185.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:36:00.639411 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.639404    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-185.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:36:00.641552 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.641535    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-185.ec2.internal"
Sep 10 20:36:00.641641 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.641563    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:36:00.642123 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.642110    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-185.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:36:00.642213 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.642135    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-185.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:36:00.642213 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.642148    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-185.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:36:00.645370 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:00.645356    2564 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-185.ec2.internal\" not found" node="ip-10-0-141-185.ec2.internal"
Sep 10 20:36:00.662684 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:00.662667    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-185.ec2.internal\" not found" node="ip-10-0-141-185.ec2.internal"
Sep 10 20:36:00.666706 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:00.666691    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-185.ec2.internal\" not found" node="ip-10-0-141-185.ec2.internal"
Sep 10 20:36:00.736478 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.736456    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7ae9783e5e544859626d5a53f1ccb92a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-185.ec2.internal\" (UID: \"7ae9783e5e544859626d5a53f1ccb92a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-185.ec2.internal"
Sep 10 20:36:00.736550 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.736485    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ae9783e5e544859626d5a53f1ccb92a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-185.ec2.internal\" (UID: \"7ae9783e5e544859626d5a53f1ccb92a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-185.ec2.internal"
Sep 10 20:36:00.810594 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.810568    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:36:00.811351 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.811337    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-185.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:36:00.811406 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.811363    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-185.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:36:00.811406 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.811378    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-185.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:36:00.811474 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.811408    2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-185.ec2.internal"
Sep 10 20:36:00.821523 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.821509    2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-185.ec2.internal"
Sep 10 20:36:00.821568 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:00.821528    2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-185.ec2.internal\": node \"ip-10-0-141-185.ec2.internal\" not found"
Sep 10 20:36:00.836816 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.836794    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7ae9783e5e544859626d5a53f1ccb92a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-185.ec2.internal\" (UID: \"7ae9783e5e544859626d5a53f1ccb92a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-185.ec2.internal"
Sep 10 20:36:00.836902 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.836820    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ae9783e5e544859626d5a53f1ccb92a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-185.ec2.internal\" (UID: \"7ae9783e5e544859626d5a53f1ccb92a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-185.ec2.internal"
Sep 10 20:36:00.836902 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.836838    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b75d22eafcad6f42168b54bcbeb68b91-config\") pod \"kube-apiserver-proxy-ip-10-0-141-185.ec2.internal\" (UID: \"b75d22eafcad6f42168b54bcbeb68b91\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-185.ec2.internal"
Sep 10 20:36:00.836902 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.836869    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ae9783e5e544859626d5a53f1ccb92a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-185.ec2.internal\" (UID: \"7ae9783e5e544859626d5a53f1ccb92a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-185.ec2.internal"
Sep 10 20:36:00.836902 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.836888    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7ae9783e5e544859626d5a53f1ccb92a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-185.ec2.internal\" (UID: \"7ae9783e5e544859626d5a53f1ccb92a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-185.ec2.internal"
Sep 10 20:36:00.864509 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:00.864467    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-185.ec2.internal\" not found"
Sep 10 20:36:00.936989 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.936966    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b75d22eafcad6f42168b54bcbeb68b91-config\") pod \"kube-apiserver-proxy-ip-10-0-141-185.ec2.internal\" (UID: \"b75d22eafcad6f42168b54bcbeb68b91\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-185.ec2.internal"
Sep 10 20:36:00.937040 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.937002    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b75d22eafcad6f42168b54bcbeb68b91-config\") pod \"kube-apiserver-proxy-ip-10-0-141-185.ec2.internal\" (UID: \"b75d22eafcad6f42168b54bcbeb68b91\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-185.ec2.internal"
Sep 10 20:36:00.964113 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.964092    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-185.ec2.internal"
Sep 10 20:36:00.965198 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:00.965182    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-185.ec2.internal\" not found"
Sep 10 20:36:00.969737 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:00.968912    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-185.ec2.internal"
Sep 10 20:36:01.066023 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:01.065997    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-185.ec2.internal\" not found"
Sep 10 20:36:01.166611 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:01.166555    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-185.ec2.internal\" not found"
Sep 10 20:36:01.268478 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:01.268442    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-185.ec2.internal\" not found"
Sep 10 20:36:01.369021 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:01.368995    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-185.ec2.internal\" not found"
Sep 10 20:36:01.373199 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:01.373155    2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 20:36:01.425525 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:01.425473    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:36:01.469434 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:01.469283    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-185.ec2.internal\" not found"
Sep 10 20:36:01.570142 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:01.570122    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-185.ec2.internal\" not found"
Sep 10 20:36:01.583364 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:01.583333    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae9783e5e544859626d5a53f1ccb92a.slice/crio-47749b358d6d1df9bbaca195db45921b05af7cc1868aa89ab3db74ec4c55c217 WatchSource:0}: Error finding container 47749b358d6d1df9bbaca195db45921b05af7cc1868aa89ab3db74ec4c55c217: Status 404 returned error can't find the container with id 47749b358d6d1df9bbaca195db45921b05af7cc1868aa89ab3db74ec4c55c217
Sep 10 20:36:01.587942 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:01.587926    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 20:36:01.588804 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:01.588771    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb75d22eafcad6f42168b54bcbeb68b91.slice/crio-e34e08e9721c8f7e597f35024789af586be83b9e44b7ee61042b9472dfb2e25f WatchSource:0}: Error finding container e34e08e9721c8f7e597f35024789af586be83b9e44b7ee61042b9472dfb2e25f: Status 404 returned error can't find the container with id e34e08e9721c8f7e597f35024789af586be83b9e44b7ee61042b9472dfb2e25f
Sep 10 20:36:01.604002 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:01.603962    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 20:31:00 +0000 UTC" deadline="2028-03-03 21:22:53.653560393 +0000 UTC"
Sep 10 20:36:01.604002 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:01.603996    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12960h46m52.049568156s"
Sep 10 20:36:01.656438 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:01.656417    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:36:01.657271 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:01.657252    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-185.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:36:01.657341 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:01.657281    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-185.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:36:01.657341 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:01.657292    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-185.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:36:01.671264 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:01.671244    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-185.ec2.internal\" not found"
Sep 10 20:36:01.735161 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:01.735118    2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:36:01.834483 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:01.834452    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-185.ec2.internal"
Sep 10 20:36:01.849125 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:01.849106    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 20:36:01.851367 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:01.851352    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-185.ec2.internal"
Sep 10 20:36:01.860965 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:01.860945    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 20:36:01.936799 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:01.936779    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:36:01.966075 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:01.966052    2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:36:02.414254 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.414229    2564 apiserver.go:52] "Watching apiserver"
Sep 10 20:36:02.420360 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.420339    2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 20:36:02.421819 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.421781    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-kn6fl","kube-system/konnectivity-agent-vv9cn","kube-system/kube-apiserver-proxy-ip-10-0-141-185.ec2.internal","openshift-dns/node-resolver-kqb7x","openshift-image-registry/node-ca-fxt2l","openshift-multus/multus-qc6bx","openshift-multus/network-metrics-daemon-vrrvq","openshift-network-diagnostics/network-check-target-wl95n","kube-system/global-pull-secret-syncer-v7c4f","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6425","openshift-cluster-node-tuning-operator/tuned-6sj2b","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-185.ec2.internal","openshift-multus/multus-additional-cni-plugins-hqsvv","openshift-network-operator/iptables-alerter-2mv5v"]
Sep 10 20:36:02.424520 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.424499    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrrvq"
Sep 10 20:36:02.424628 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:02.424592    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrrvq" podUID="e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b"
Sep 10 20:36:02.427022 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.426917    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hqsvv"
Sep 10 20:36:02.429867 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.429839    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2mv5v"
Sep 10 20:36:02.429977 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.429960    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 20:36:02.430613 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.430421    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-q78tm\""
Sep 10 20:36:02.430613 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.430523    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 20:36:02.431694 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.431673    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kqb7x"
Sep 10 20:36:02.432813 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.432791    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 20:36:02.432906 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.432889    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 20:36:02.433088 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.433068    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 20:36:02.433160 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.433113    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 20:36:02.433226 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.433068    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 20:36:02.433293 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.433277    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 20:36:02.433344 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.433318    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4qltg\""
Sep 10 20:36:02.434364 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.433964    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fxt2l"
Sep 10 20:36:02.434364 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.434292    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 20:36:02.436685 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.436420    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.438895 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.438753    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:36:02.438895 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:02.438832    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7c4f" podUID="e14c5919-af21-4e8f-9783-a2ee2b4885c0"
Sep 10 20:36:02.441422 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.441401    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.442086 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.442058    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 20:36:02.442187 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.442117    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 20:36:02.442370 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.442348    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 20:36:02.442458 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.442410    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 20:36:02.442521 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.442477    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 20:36:02.442521 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.442485    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 20:36:02.442622 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.442592    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gdq2c\""
Sep 10 20:36:02.442670 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.442619    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nn8rf\""
Sep 10 20:36:02.442809 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.442792    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qj45q\""
Sep 10 20:36:02.444005 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.443984    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vv9cn"
Sep 10 20:36:02.446851 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.445438    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1b63c96-edab-42b0-8af3-f9db7d20867a-cni-binary-copy\") pod \"multus-additional-cni-plugins-hqsvv\" (UID: \"b1b63c96-edab-42b0-8af3-f9db7d20867a\") " pod="openshift-multus/multus-additional-cni-plugins-hqsvv"
Sep 10 20:36:02.446851 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.445583    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xvb5\" (UniqueName: \"kubernetes.io/projected/a4bb15a1-d45e-4974-b648-8cffc50af95b-kube-api-access-8xvb5\") pod \"iptables-alerter-2mv5v\" (UID: \"a4bb15a1-d45e-4974-b648-8cffc50af95b\") " pod="openshift-network-operator/iptables-alerter-2mv5v"
Sep 10 20:36:02.446851 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.445616    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ad5d5af7-f551-415c-9a17-fb89dd6bbb0a-serviceca\") pod \"node-ca-fxt2l\" (UID: \"ad5d5af7-f551-415c-9a17-fb89dd6bbb0a\") " pod="openshift-image-registry/node-ca-fxt2l"
Sep 10 20:36:02.446851 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.445658    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-multus-conf-dir\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.446851 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.445687    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b-metrics-certs\") pod \"network-metrics-daemon-vrrvq\" (UID: \"e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b\") " pod="openshift-multus/network-metrics-daemon-vrrvq"
Sep 10 20:36:02.446851 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.445719    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjvmm\" (UniqueName: \"kubernetes.io/projected/b1b63c96-edab-42b0-8af3-f9db7d20867a-kube-api-access-mjvmm\") pod \"multus-additional-cni-plugins-hqsvv\" (UID: \"b1b63c96-edab-42b0-8af3-f9db7d20867a\") " pod="openshift-multus/multus-additional-cni-plugins-hqsvv"
Sep 10 20:36:02.446851 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.445841    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a4bb15a1-d45e-4974-b648-8cffc50af95b-host-slash\") pod \"iptables-alerter-2mv5v\" (UID: \"a4bb15a1-d45e-4974-b648-8cffc50af95b\") " pod="openshift-network-operator/iptables-alerter-2mv5v"
Sep 10 20:36:02.446851 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.445871    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-cnibin\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.446851 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.446000    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-system-cni-dir\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.446851 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.446025    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-host-var-lib-cni-bin\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.446851 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.446073    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-host-var-lib-cni-multus\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.446851 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.446217    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1b63c96-edab-42b0-8af3-f9db7d20867a-system-cni-dir\") pod \"multus-additional-cni-plugins-hqsvv\" (UID: \"b1b63c96-edab-42b0-8af3-f9db7d20867a\") " pod="openshift-multus/multus-additional-cni-plugins-hqsvv"
Sep 10 20:36:02.446851 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.446268    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-host-run-k8s-cni-cncf-io\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.446851 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.446299    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-etc-kubernetes\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.446851 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.446330    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gbwj\" (UniqueName: \"kubernetes.io/projected/0447592e-554b-4f55-a206-d5aa4a5e4013-kube-api-access-4gbwj\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.446851 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.446736    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 20:36:02.447908 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.447067    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 20:36:02.447908 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.447288    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kvbqz\""
Sep 10 20:36:02.447908 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.447486    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 20:36:02.447908 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.447540    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1b63c96-edab-42b0-8af3-f9db7d20867a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hqsvv\" (UID: \"b1b63c96-edab-42b0-8af3-f9db7d20867a\") " pod="openshift-multus/multus-additional-cni-plugins-hqsvv"
Sep 10 20:36:02.447908 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.447587    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-host-run-netns\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.447908 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.447609    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 20:36:02.447908 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.447624    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-hostroot\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.447908 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.447662    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-host-run-multus-certs\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.447908 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.447697    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1b63c96-edab-42b0-8af3-f9db7d20867a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hqsvv\" (UID: \"b1b63c96-edab-42b0-8af3-f9db7d20867a\") " pod="openshift-multus/multus-additional-cni-plugins-hqsvv"
Sep 10 20:36:02.447908 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.447731    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/af88567d-1c33-4670-acc8-484e2d8b11d4-hosts-file\") pod \"node-resolver-kqb7x\" (UID: \"af88567d-1c33-4670-acc8-484e2d8b11d4\") " pod="openshift-dns/node-resolver-kqb7x"
Sep 10 20:36:02.447908 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.447809    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 20:36:02.447908 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.447819    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfsjp\" (UniqueName: \"kubernetes.io/projected/ad5d5af7-f551-415c-9a17-fb89dd6bbb0a-kube-api-access-zfsjp\") pod \"node-ca-fxt2l\" (UID: \"ad5d5af7-f551-415c-9a17-fb89dd6bbb0a\") " pod="openshift-image-registry/node-ca-fxt2l"
Sep 10 20:36:02.447908 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.447854    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-multus-cni-dir\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.447908 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.447887    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-host-var-lib-kubelet\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.448580 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.448003    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfm27\" (UniqueName: \"kubernetes.io/projected/e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b-kube-api-access-nfm27\") pod \"network-metrics-daemon-vrrvq\" (UID: \"e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b\") " pod="openshift-multus/network-metrics-daemon-vrrvq"
Sep 10 20:36:02.448580 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.448033    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1b63c96-edab-42b0-8af3-f9db7d20867a-cnibin\") pod \"multus-additional-cni-plugins-hqsvv\" (UID: \"b1b63c96-edab-42b0-8af3-f9db7d20867a\") " pod="openshift-multus/multus-additional-cni-plugins-hqsvv"
Sep 10 20:36:02.448580 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.448068    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a4bb15a1-d45e-4974-b648-8cffc50af95b-iptables-alerter-script\") pod \"iptables-alerter-2mv5v\" (UID: \"a4bb15a1-d45e-4974-b648-8cffc50af95b\") " pod="openshift-network-operator/iptables-alerter-2mv5v"
Sep 10 20:36:02.448580 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.448035    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 20:36:02.448580 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.448449    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwk9g\" (UniqueName: \"kubernetes.io/projected/af88567d-1c33-4670-acc8-484e2d8b11d4-kube-api-access-xwk9g\") pod \"node-resolver-kqb7x\" (UID: \"af88567d-1c33-4670-acc8-484e2d8b11d4\") " pod="openshift-dns/node-resolver-kqb7x"
Sep 10 20:36:02.448580 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.448570    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0447592e-554b-4f55-a206-d5aa4a5e4013-cni-binary-copy\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.448940 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.448679    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-multus-socket-dir-parent\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.449119 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.449070    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6425"
Sep 10 20:36:02.449317 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.449243    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0447592e-554b-4f55-a206-d5aa4a5e4013-multus-daemon-config\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.449833 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.449292    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1b63c96-edab-42b0-8af3-f9db7d20867a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hqsvv\" (UID: \"b1b63c96-edab-42b0-8af3-f9db7d20867a\") " pod="openshift-multus/multus-additional-cni-plugins-hqsvv"
Sep 10 20:36:02.449833 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.449496    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af88567d-1c33-4670-acc8-484e2d8b11d4-tmp-dir\") pod \"node-resolver-kqb7x\" (UID: \"af88567d-1c33-4670-acc8-484e2d8b11d4\") " pod="openshift-dns/node-resolver-kqb7x"
Sep 10 20:36:02.449833 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.449524    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad5d5af7-f551-415c-9a17-fb89dd6bbb0a-host\") pod \"node-ca-fxt2l\" (UID: \"ad5d5af7-f551-415c-9a17-fb89dd6bbb0a\") " pod="openshift-image-registry/node-ca-fxt2l"
Sep 10 20:36:02.450452 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.450233    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 20:36:02.451739 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.451704    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.454261 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.454239    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl95n"
Sep 10 20:36:02.454355 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:02.454310    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wl95n" podUID="2e1ea2e3-fc76-40ad-a7a7-111080547273"
Sep 10 20:36:02.458093 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.458063    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4fw7w\""
Sep 10 20:36:02.458332 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.458310    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 20:36:02.459934 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.459916    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 20:36:02.462673 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.462652    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 20:36:02.462769 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.462672    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 20:36:02.462769 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.462717    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 20:36:02.462769 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.462751    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kt447\""
Sep 10 20:36:02.462969 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.462951    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 20:36:02.468849 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.468828    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 20:36:02.469102 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.469087    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-28xcw\""
Sep 10 20:36:02.537039 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.537019    2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 20:36:02.539965 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.539921    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-185.ec2.internal" event={"ID":"b75d22eafcad6f42168b54bcbeb68b91","Type":"ContainerStarted","Data":"e34e08e9721c8f7e597f35024789af586be83b9e44b7ee61042b9472dfb2e25f"}
Sep 10 20:36:02.541383 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.541357    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-185.ec2.internal" event={"ID":"7ae9783e5e544859626d5a53f1ccb92a","Type":"ContainerStarted","Data":"47749b358d6d1df9bbaca195db45921b05af7cc1868aa89ab3db74ec4c55c217"}
Sep 10 20:36:02.550103 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550081    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b112c171-a796-4f0b-b3ee-42a93807bbe1-lib-modules\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.550220 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550115    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b112c171-a796-4f0b-b3ee-42a93807bbe1-host\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.550220 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550146    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-host-var-lib-cni-bin\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.550220 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550173    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-host-var-lib-cni-multus\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.550400 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550219    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/04f51c2b-efe9-465f-bf46-f014f03cc131-socket-dir\") pod \"aws-ebs-csi-driver-node-r6425\" (UID: \"04f51c2b-efe9-465f-bf46-f014f03cc131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6425"
Sep 10 20:36:02.550400 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550241    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-host-var-lib-cni-bin\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.550400 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550255    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/04f51c2b-efe9-465f-bf46-f014f03cc131-registration-dir\") pod \"aws-ebs-csi-driver-node-r6425\" (UID: \"04f51c2b-efe9-465f-bf46-f014f03cc131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6425"
Sep 10 20:36:02.550400 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550276    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-host-var-lib-cni-multus\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.550400 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550286    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ebbed649-b086-441c-b19c-4fa20fa4f77f-ovnkube-script-lib\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.550400 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550317    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-host-run-k8s-cni-cncf-io\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.550400 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550344    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/821d4eb6-5a24-40b4-bee3-472994b9be36-konnectivity-ca\") pod \"konnectivity-agent-vv9cn\" (UID: \"821d4eb6-5a24-40b4-bee3-472994b9be36\") " pod="kube-system/konnectivity-agent-vv9cn"
Sep 10 20:36:02.550400 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550372    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1b63c96-edab-42b0-8af3-f9db7d20867a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hqsvv\" (UID: \"b1b63c96-edab-42b0-8af3-f9db7d20867a\") " pod="openshift-multus/multus-additional-cni-plugins-hqsvv"
Sep 10 20:36:02.550400 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550398    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-host-run-netns\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.550903 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550406    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-host-run-k8s-cni-cncf-io\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.550903 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550423    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-hostroot\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.550903 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550450    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-host-run-multus-certs\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.550903 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550492    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e14c5919-af21-4e8f-9783-a2ee2b4885c0-kubelet-config\") pod \"global-pull-secret-syncer-v7c4f\" (UID: \"e14c5919-af21-4e8f-9783-a2ee2b4885c0\") " pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:36:02.550903 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550518    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-systemd-units\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.550903 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550553    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ebbed649-b086-441c-b19c-4fa20fa4f77f-ovn-node-metrics-cert\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.550903 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550579    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b112c171-a796-4f0b-b3ee-42a93807bbe1-etc-modprobe-d\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.550903 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550604    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/af88567d-1c33-4670-acc8-484e2d8b11d4-hosts-file\") pod \"node-resolver-kqb7x\" (UID: \"af88567d-1c33-4670-acc8-484e2d8b11d4\") " pod="openshift-dns/node-resolver-kqb7x"
Sep 10 20:36:02.550903 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550631    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfsjp\" (UniqueName: \"kubernetes.io/projected/ad5d5af7-f551-415c-9a17-fb89dd6bbb0a-kube-api-access-zfsjp\") pod \"node-ca-fxt2l\" (UID: \"ad5d5af7-f551-415c-9a17-fb89dd6bbb0a\") " pod="openshift-image-registry/node-ca-fxt2l"
Sep 10 20:36:02.550903 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550655    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-host-var-lib-kubelet\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.550903 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550681    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/04f51c2b-efe9-465f-bf46-f014f03cc131-device-dir\") pod \"aws-ebs-csi-driver-node-r6425\" (UID: \"04f51c2b-efe9-465f-bf46-f014f03cc131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6425"
Sep 10 20:36:02.550903 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550705    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-host-cni-netd\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.550903 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550730    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b112c171-a796-4f0b-b3ee-42a93807bbe1-var-lib-kubelet\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.550903 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550752    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b112c171-a796-4f0b-b3ee-42a93807bbe1-etc-tuned\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.550903 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550789    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvr8v\" (UniqueName: \"kubernetes.io/projected/2e1ea2e3-fc76-40ad-a7a7-111080547273-kube-api-access-hvr8v\") pod \"network-check-target-wl95n\" (UID: \"2e1ea2e3-fc76-40ad-a7a7-111080547273\") " pod="openshift-network-diagnostics/network-check-target-wl95n"
Sep 10 20:36:02.550903 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550816    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpmvg\" (UniqueName: \"kubernetes.io/projected/04f51c2b-efe9-465f-bf46-f014f03cc131-kube-api-access-mpmvg\") pod \"aws-ebs-csi-driver-node-r6425\" (UID: \"04f51c2b-efe9-465f-bf46-f014f03cc131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6425"
Sep 10 20:36:02.550903 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550843    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1b63c96-edab-42b0-8af3-f9db7d20867a-cnibin\") pod \"multus-additional-cni-plugins-hqsvv\" (UID: \"b1b63c96-edab-42b0-8af3-f9db7d20867a\") " pod="openshift-multus/multus-additional-cni-plugins-hqsvv"
Sep 10 20:36:02.551820 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550868    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a4bb15a1-d45e-4974-b648-8cffc50af95b-iptables-alerter-script\") pod \"iptables-alerter-2mv5v\" (UID: \"a4bb15a1-d45e-4974-b648-8cffc50af95b\") " pod="openshift-network-operator/iptables-alerter-2mv5v"
Sep 10 20:36:02.551820 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550908    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwk9g\" (UniqueName: \"kubernetes.io/projected/af88567d-1c33-4670-acc8-484e2d8b11d4-kube-api-access-xwk9g\") pod \"node-resolver-kqb7x\" (UID: \"af88567d-1c33-4670-acc8-484e2d8b11d4\") " pod="openshift-dns/node-resolver-kqb7x"
Sep 10 20:36:02.551820 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550939    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/04f51c2b-efe9-465f-bf46-f014f03cc131-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r6425\" (UID: \"04f51c2b-efe9-465f-bf46-f014f03cc131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6425"
Sep 10 20:36:02.551820 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550969    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mbnn\" (UniqueName: \"kubernetes.io/projected/ebbed649-b086-441c-b19c-4fa20fa4f77f-kube-api-access-6mbnn\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.551820 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550998    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b112c171-a796-4f0b-b3ee-42a93807bbe1-run\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.551820 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.551027    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af88567d-1c33-4670-acc8-484e2d8b11d4-tmp-dir\") pod \"node-resolver-kqb7x\" (UID: \"af88567d-1c33-4670-acc8-484e2d8b11d4\") " pod="openshift-dns/node-resolver-kqb7x"
Sep 10 20:36:02.551820 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.551069    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-host-slash\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.551820 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.551104    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b112c171-a796-4f0b-b3ee-42a93807bbe1-tmp\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.551820 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.551125    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-host-run-multus-certs\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.551820 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.551274    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/af88567d-1c33-4670-acc8-484e2d8b11d4-hosts-file\") pod \"node-resolver-kqb7x\" (UID: \"af88567d-1c33-4670-acc8-484e2d8b11d4\") " pod="openshift-dns/node-resolver-kqb7x"
Sep 10 20:36:02.551820 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.551552    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-host-var-lib-kubelet\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.551820 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.551129    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjl6x\" (UniqueName: \"kubernetes.io/projected/b112c171-a796-4f0b-b3ee-42a93807bbe1-kube-api-access-tjl6x\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.551820 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.551628    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ad5d5af7-f551-415c-9a17-fb89dd6bbb0a-serviceca\") pod \"node-ca-fxt2l\" (UID: \"ad5d5af7-f551-415c-9a17-fb89dd6bbb0a\") " pod="openshift-image-registry/node-ca-fxt2l"
Sep 10 20:36:02.551820 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.551655    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-multus-conf-dir\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.551820 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.551682    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e14c5919-af21-4e8f-9783-a2ee2b4885c0-dbus\") pod \"global-pull-secret-syncer-v7c4f\" (UID: \"e14c5919-af21-4e8f-9783-a2ee2b4885c0\") " pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:36:02.551820 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.551709    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-host-kubelet\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.551820 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.551733    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-run-systemd\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.552713 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.551756    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-etc-openvswitch\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.552713 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.551770    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a4bb15a1-d45e-4974-b648-8cffc50af95b-iptables-alerter-script\") pod \"iptables-alerter-2mv5v\" (UID: \"a4bb15a1-d45e-4974-b648-8cffc50af95b\") " pod="openshift-network-operator/iptables-alerter-2mv5v"
Sep 10 20:36:02.552713 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.551785    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b-metrics-certs\") pod \"network-metrics-daemon-vrrvq\" (UID: \"e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b\") " pod="openshift-multus/network-metrics-daemon-vrrvq"
Sep 10 20:36:02.552713 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.551815    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-cnibin\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.552713 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.551853    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/04f51c2b-efe9-465f-bf46-f014f03cc131-sys-fs\") pod \"aws-ebs-csi-driver-node-r6425\" (UID: \"04f51c2b-efe9-465f-bf46-f014f03cc131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6425"
Sep 10 20:36:02.552713 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.551864    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1b63c96-edab-42b0-8af3-f9db7d20867a-cnibin\") pod \"multus-additional-cni-plugins-hqsvv\" (UID: \"b1b63c96-edab-42b0-8af3-f9db7d20867a\") " pod="openshift-multus/multus-additional-cni-plugins-hqsvv"
Sep 10 20:36:02.552713 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.550995    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1b63c96-edab-42b0-8af3-f9db7d20867a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hqsvv\" (UID: \"b1b63c96-edab-42b0-8af3-f9db7d20867a\") " pod="openshift-multus/multus-additional-cni-plugins-hqsvv"
Sep 10 20:36:02.552713 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.552314    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ad5d5af7-f551-415c-9a17-fb89dd6bbb0a-serviceca\") pod \"node-ca-fxt2l\" (UID: \"ad5d5af7-f551-415c-9a17-fb89dd6bbb0a\") " pod="openshift-image-registry/node-ca-fxt2l"
Sep 10 20:36:02.552713 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.552374    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-multus-conf-dir\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.552713 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.552439    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af88567d-1c33-4670-acc8-484e2d8b11d4-tmp-dir\") pod \"node-resolver-kqb7x\" (UID: \"af88567d-1c33-4670-acc8-484e2d8b11d4\") " pod="openshift-dns/node-resolver-kqb7x"
Sep 10 20:36:02.552713 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:02.552482    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:36:02.552713 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.551078    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-host-run-netns\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.552713 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:02.552552    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b-metrics-certs podName:e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b nodeName:}" failed. No retries permitted until 2026-09-10 20:36:03.052521158 +0000 UTC m=+3.017589339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b-metrics-certs") pod "network-metrics-daemon-vrrvq" (UID: "e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:36:02.552713 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.552550    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-hostroot\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.552713 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.552579    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e14c5919-af21-4e8f-9783-a2ee2b4885c0-original-pull-secret\") pod \"global-pull-secret-syncer-v7c4f\" (UID: \"e14c5919-af21-4e8f-9783-a2ee2b4885c0\") " pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:36:02.552713 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.552600    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-cnibin\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.552713 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.552640    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-system-cni-dir\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.553636 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.552669    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-var-lib-openvswitch\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.553636 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.552709    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-node-log\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.553636 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.552734    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-host-run-ovn-kubernetes\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.553636 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.552771    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ebbed649-b086-441c-b19c-4fa20fa4f77f-ovnkube-config\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.553636 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.552810    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1b63c96-edab-42b0-8af3-f9db7d20867a-system-cni-dir\") pod \"multus-additional-cni-plugins-hqsvv\" (UID: \"b1b63c96-edab-42b0-8af3-f9db7d20867a\") " pod="openshift-multus/multus-additional-cni-plugins-hqsvv"
Sep 10 20:36:02.553636 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.552837    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-etc-kubernetes\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.553636 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.552855    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-system-cni-dir\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.553636 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.552861    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gbwj\" (UniqueName: \"kubernetes.io/projected/0447592e-554b-4f55-a206-d5aa4a5e4013-kube-api-access-4gbwj\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.553636 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.552888    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-run-openvswitch\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.553636 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.552911    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ebbed649-b086-441c-b19c-4fa20fa4f77f-env-overrides\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.553636 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.552937    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b112c171-a796-4f0b-b3ee-42a93807bbe1-etc-sysctl-conf\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.553636 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.552964    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/04f51c2b-efe9-465f-bf46-f014f03cc131-etc-selinux\") pod \"aws-ebs-csi-driver-node-r6425\" (UID: \"04f51c2b-efe9-465f-bf46-f014f03cc131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6425"
Sep 10 20:36:02.553636 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.553016    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.553636 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.553044    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b112c171-a796-4f0b-b3ee-42a93807bbe1-etc-kubernetes\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.553636 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.553072    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1b63c96-edab-42b0-8af3-f9db7d20867a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hqsvv\" (UID: \"b1b63c96-edab-42b0-8af3-f9db7d20867a\") " pod="openshift-multus/multus-additional-cni-plugins-hqsvv"
Sep 10 20:36:02.553636 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.553094    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1b63c96-edab-42b0-8af3-f9db7d20867a-system-cni-dir\") pod \"multus-additional-cni-plugins-hqsvv\" (UID: \"b1b63c96-edab-42b0-8af3-f9db7d20867a\") " pod="openshift-multus/multus-additional-cni-plugins-hqsvv"
Sep 10 20:36:02.553636 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.553105    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-multus-cni-dir\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.554536 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.553135    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfm27\" (UniqueName: \"kubernetes.io/projected/e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b-kube-api-access-nfm27\") pod \"network-metrics-daemon-vrrvq\" (UID: \"e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b\") " pod="openshift-multus/network-metrics-daemon-vrrvq"
Sep 10 20:36:02.554536 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.553177    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-etc-kubernetes\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.554536 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.553287    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0447592e-554b-4f55-a206-d5aa4a5e4013-cni-binary-copy\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.554536 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.553309    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-multus-cni-dir\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.554536 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.553310    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1b63c96-edab-42b0-8af3-f9db7d20867a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hqsvv\" (UID: \"b1b63c96-edab-42b0-8af3-f9db7d20867a\") " pod="openshift-multus/multus-additional-cni-plugins-hqsvv"
Sep 10 20:36:02.554536 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.553333    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-multus-socket-dir-parent\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.554536 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.553382    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0447592e-554b-4f55-a206-d5aa4a5e4013-multus-daemon-config\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.554536 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.553386    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0447592e-554b-4f55-a206-d5aa4a5e4013-multus-socket-dir-parent\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.554536 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.553419    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-run-ovn\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.554536 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.553449    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1b63c96-edab-42b0-8af3-f9db7d20867a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hqsvv\" (UID: \"b1b63c96-edab-42b0-8af3-f9db7d20867a\") " pod="openshift-multus/multus-additional-cni-plugins-hqsvv"
Sep 10 20:36:02.554536 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.553479    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad5d5af7-f551-415c-9a17-fb89dd6bbb0a-host\") pod \"node-ca-fxt2l\" (UID: \"ad5d5af7-f551-415c-9a17-fb89dd6bbb0a\") " pod="openshift-image-registry/node-ca-fxt2l"
Sep 10 20:36:02.554536 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.553507    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b112c171-a796-4f0b-b3ee-42a93807bbe1-etc-systemd\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.554536 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.553532    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1b63c96-edab-42b0-8af3-f9db7d20867a-cni-binary-copy\") pod \"multus-additional-cni-plugins-hqsvv\" (UID: \"b1b63c96-edab-42b0-8af3-f9db7d20867a\") " pod="openshift-multus/multus-additional-cni-plugins-hqsvv"
Sep 10 20:36:02.554536 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.553558    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xvb5\" (UniqueName: \"kubernetes.io/projected/a4bb15a1-d45e-4974-b648-8cffc50af95b-kube-api-access-8xvb5\") pod \"iptables-alerter-2mv5v\" (UID: \"a4bb15a1-d45e-4974-b648-8cffc50af95b\") " pod="openshift-network-operator/iptables-alerter-2mv5v"
Sep 10 20:36:02.554536 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.553572    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad5d5af7-f551-415c-9a17-fb89dd6bbb0a-host\") pod \"node-ca-fxt2l\" (UID: \"ad5d5af7-f551-415c-9a17-fb89dd6bbb0a\") " pod="openshift-image-registry/node-ca-fxt2l"
Sep 10 20:36:02.554536 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.553583    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-host-cni-bin\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.554536 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.553611    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b112c171-a796-4f0b-b3ee-42a93807bbe1-etc-sysctl-d\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.555355 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.553634    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b112c171-a796-4f0b-b3ee-42a93807bbe1-sys\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.555355 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.553660    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjvmm\" (UniqueName: \"kubernetes.io/projected/b1b63c96-edab-42b0-8af3-f9db7d20867a-kube-api-access-mjvmm\") pod \"multus-additional-cni-plugins-hqsvv\" (UID: \"b1b63c96-edab-42b0-8af3-f9db7d20867a\") " pod="openshift-multus/multus-additional-cni-plugins-hqsvv"
Sep 10 20:36:02.555355 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.553686    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a4bb15a1-d45e-4974-b648-8cffc50af95b-host-slash\") pod \"iptables-alerter-2mv5v\" (UID: \"a4bb15a1-d45e-4974-b648-8cffc50af95b\") " pod="openshift-network-operator/iptables-alerter-2mv5v"
Sep 10 20:36:02.555355 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.553711    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/821d4eb6-5a24-40b4-bee3-472994b9be36-agent-certs\") pod \"konnectivity-agent-vv9cn\" (UID: \"821d4eb6-5a24-40b4-bee3-472994b9be36\") " pod="kube-system/konnectivity-agent-vv9cn"
Sep 10 20:36:02.555355 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.553736    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-host-run-netns\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.555355 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.553759    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-log-socket\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.555355 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.553783    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b112c171-a796-4f0b-b3ee-42a93807bbe1-etc-sysconfig\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.555355 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.553864    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0447592e-554b-4f55-a206-d5aa4a5e4013-cni-binary-copy\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.555355 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.553940    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a4bb15a1-d45e-4974-b648-8cffc50af95b-host-slash\") pod \"iptables-alerter-2mv5v\" (UID: \"a4bb15a1-d45e-4974-b648-8cffc50af95b\") " pod="openshift-network-operator/iptables-alerter-2mv5v"
Sep 10 20:36:02.555355 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.553954    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1b63c96-edab-42b0-8af3-f9db7d20867a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hqsvv\" (UID: \"b1b63c96-edab-42b0-8af3-f9db7d20867a\") " pod="openshift-multus/multus-additional-cni-plugins-hqsvv"
Sep 10 20:36:02.555355 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.554470    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0447592e-554b-4f55-a206-d5aa4a5e4013-multus-daemon-config\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.555355 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.554908    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1b63c96-edab-42b0-8af3-f9db7d20867a-cni-binary-copy\") pod \"multus-additional-cni-plugins-hqsvv\" (UID: \"b1b63c96-edab-42b0-8af3-f9db7d20867a\") " pod="openshift-multus/multus-additional-cni-plugins-hqsvv"
Sep 10 20:36:02.566149 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.566126    2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 20:36:02.570651 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.570627    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xvb5\" (UniqueName: \"kubernetes.io/projected/a4bb15a1-d45e-4974-b648-8cffc50af95b-kube-api-access-8xvb5\") pod \"iptables-alerter-2mv5v\" (UID: \"a4bb15a1-d45e-4974-b648-8cffc50af95b\") " pod="openshift-network-operator/iptables-alerter-2mv5v"
Sep 10 20:36:02.570787 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.570630    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwk9g\" (UniqueName: \"kubernetes.io/projected/af88567d-1c33-4670-acc8-484e2d8b11d4-kube-api-access-xwk9g\") pod \"node-resolver-kqb7x\" (UID: \"af88567d-1c33-4670-acc8-484e2d8b11d4\") " pod="openshift-dns/node-resolver-kqb7x"
Sep 10 20:36:02.571633 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.571587    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjvmm\" (UniqueName: \"kubernetes.io/projected/b1b63c96-edab-42b0-8af3-f9db7d20867a-kube-api-access-mjvmm\") pod \"multus-additional-cni-plugins-hqsvv\" (UID: \"b1b63c96-edab-42b0-8af3-f9db7d20867a\") " pod="openshift-multus/multus-additional-cni-plugins-hqsvv"
Sep 10 20:36:02.572770 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.572741    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfsjp\" (UniqueName: \"kubernetes.io/projected/ad5d5af7-f551-415c-9a17-fb89dd6bbb0a-kube-api-access-zfsjp\") pod \"node-ca-fxt2l\" (UID: \"ad5d5af7-f551-415c-9a17-fb89dd6bbb0a\") " pod="openshift-image-registry/node-ca-fxt2l"
Sep 10 20:36:02.573543 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.573515    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfm27\" (UniqueName: \"kubernetes.io/projected/e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b-kube-api-access-nfm27\") pod \"network-metrics-daemon-vrrvq\" (UID: \"e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b\") " pod="openshift-multus/network-metrics-daemon-vrrvq"
Sep 10 20:36:02.573679 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.573659    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gbwj\" (UniqueName: \"kubernetes.io/projected/0447592e-554b-4f55-a206-d5aa4a5e4013-kube-api-access-4gbwj\") pod \"multus-qc6bx\" (UID: \"0447592e-554b-4f55-a206-d5aa4a5e4013\") " pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.597372 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.597349    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qpslp"
Sep 10 20:36:02.607883 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.607859    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qpslp"
Sep 10 20:36:02.654223 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.654119    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/04f51c2b-efe9-465f-bf46-f014f03cc131-socket-dir\") pod \"aws-ebs-csi-driver-node-r6425\" (UID: \"04f51c2b-efe9-465f-bf46-f014f03cc131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6425"
Sep 10 20:36:02.654223 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.654158    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/04f51c2b-efe9-465f-bf46-f014f03cc131-registration-dir\") pod \"aws-ebs-csi-driver-node-r6425\" (UID: \"04f51c2b-efe9-465f-bf46-f014f03cc131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6425"
Sep 10 20:36:02.654223 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.654185    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ebbed649-b086-441c-b19c-4fa20fa4f77f-ovnkube-script-lib\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.654223 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.654205    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/821d4eb6-5a24-40b4-bee3-472994b9be36-konnectivity-ca\") pod \"konnectivity-agent-vv9cn\" (UID: \"821d4eb6-5a24-40b4-bee3-472994b9be36\") " pod="kube-system/konnectivity-agent-vv9cn"
Sep 10 20:36:02.654509 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.654239    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e14c5919-af21-4e8f-9783-a2ee2b4885c0-kubelet-config\") pod \"global-pull-secret-syncer-v7c4f\" (UID: \"e14c5919-af21-4e8f-9783-a2ee2b4885c0\") " pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:36:02.654509 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.654262    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-systemd-units\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.654509 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.654278    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ebbed649-b086-441c-b19c-4fa20fa4f77f-ovn-node-metrics-cert\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.654509 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.654294    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b112c171-a796-4f0b-b3ee-42a93807bbe1-etc-modprobe-d\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.654509 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.654302    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/04f51c2b-efe9-465f-bf46-f014f03cc131-socket-dir\") pod \"aws-ebs-csi-driver-node-r6425\" (UID: \"04f51c2b-efe9-465f-bf46-f014f03cc131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6425"
Sep 10 20:36:02.654509 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.654311    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/04f51c2b-efe9-465f-bf46-f014f03cc131-device-dir\") pod \"aws-ebs-csi-driver-node-r6425\" (UID: \"04f51c2b-efe9-465f-bf46-f014f03cc131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6425"
Sep 10 20:36:02.654509 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.654349    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/04f51c2b-efe9-465f-bf46-f014f03cc131-device-dir\") pod \"aws-ebs-csi-driver-node-r6425\" (UID: \"04f51c2b-efe9-465f-bf46-f014f03cc131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6425"
Sep 10 20:36:02.654509 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.654364    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-host-cni-netd\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.654509 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.654389    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/04f51c2b-efe9-465f-bf46-f014f03cc131-registration-dir\") pod \"aws-ebs-csi-driver-node-r6425\" (UID: \"04f51c2b-efe9-465f-bf46-f014f03cc131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6425"
Sep 10 20:36:02.654509 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.654394    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b112c171-a796-4f0b-b3ee-42a93807bbe1-var-lib-kubelet\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.654509 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.654422    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b112c171-a796-4f0b-b3ee-42a93807bbe1-etc-tuned\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.654509 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.654455    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvr8v\" (UniqueName: \"kubernetes.io/projected/2e1ea2e3-fc76-40ad-a7a7-111080547273-kube-api-access-hvr8v\") pod \"network-check-target-wl95n\" (UID: \"2e1ea2e3-fc76-40ad-a7a7-111080547273\") " pod="openshift-network-diagnostics/network-check-target-wl95n"
Sep 10 20:36:02.654509 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.654483    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpmvg\" (UniqueName: \"kubernetes.io/projected/04f51c2b-efe9-465f-bf46-f014f03cc131-kube-api-access-mpmvg\") pod \"aws-ebs-csi-driver-node-r6425\" (UID: \"04f51c2b-efe9-465f-bf46-f014f03cc131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6425"
Sep 10 20:36:02.655177 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.654561    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b112c171-a796-4f0b-b3ee-42a93807bbe1-var-lib-kubelet\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.655177 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.654752    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-host-cni-netd\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.655177 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.654796    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/04f51c2b-efe9-465f-bf46-f014f03cc131-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r6425\" (UID: \"04f51c2b-efe9-465f-bf46-f014f03cc131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6425"
Sep 10 20:36:02.655177 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.654836    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mbnn\" (UniqueName: \"kubernetes.io/projected/ebbed649-b086-441c-b19c-4fa20fa4f77f-kube-api-access-6mbnn\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.655177 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.654864    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b112c171-a796-4f0b-b3ee-42a93807bbe1-run\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.655177 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.654895    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-host-slash\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.655177 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.654907    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ebbed649-b086-441c-b19c-4fa20fa4f77f-ovnkube-script-lib\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.655177 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.654903    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/04f51c2b-efe9-465f-bf46-f014f03cc131-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r6425\" (UID: \"04f51c2b-efe9-465f-bf46-f014f03cc131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6425"
Sep 10 20:36:02.655177 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.654919    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b112c171-a796-4f0b-b3ee-42a93807bbe1-tmp\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.655177 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.654956    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjl6x\" (UniqueName: \"kubernetes.io/projected/b112c171-a796-4f0b-b3ee-42a93807bbe1-kube-api-access-tjl6x\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.655177 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.654989    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e14c5919-af21-4e8f-9783-a2ee2b4885c0-dbus\") pod \"global-pull-secret-syncer-v7c4f\" (UID: \"e14c5919-af21-4e8f-9783-a2ee2b4885c0\") " pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:36:02.655177 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.655009    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b112c171-a796-4f0b-b3ee-42a93807bbe1-etc-modprobe-d\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.655177 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.655015    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-host-kubelet\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.655177 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.655042    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-run-systemd\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.655177 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.655067    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-etc-openvswitch\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.655177 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.655110    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/04f51c2b-efe9-465f-bf46-f014f03cc131-sys-fs\") pod \"aws-ebs-csi-driver-node-r6425\" (UID: \"04f51c2b-efe9-465f-bf46-f014f03cc131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6425"
Sep 10 20:36:02.655177 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.655136    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e14c5919-af21-4e8f-9783-a2ee2b4885c0-original-pull-secret\") pod \"global-pull-secret-syncer-v7c4f\" (UID: \"e14c5919-af21-4e8f-9783-a2ee2b4885c0\") " pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:36:02.656086 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.655164    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-var-lib-openvswitch\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.656086 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.655190    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-node-log\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.656086 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.654805    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-systemd-units\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.656086 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.655218    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-host-run-ovn-kubernetes\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.656086 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.655268    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ebbed649-b086-441c-b19c-4fa20fa4f77f-ovnkube-config\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.656086 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.655294    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-etc-openvswitch\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.656086 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.655300    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-run-openvswitch\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.656086 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.655340    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ebbed649-b086-441c-b19c-4fa20fa4f77f-env-overrides\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.656086 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.655346    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/821d4eb6-5a24-40b4-bee3-472994b9be36-konnectivity-ca\") pod \"konnectivity-agent-vv9cn\" (UID: \"821d4eb6-5a24-40b4-bee3-472994b9be36\") " pod="kube-system/konnectivity-agent-vv9cn"
Sep 10 20:36:02.656086 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.655342    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-run-openvswitch\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.656086 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.655386    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-host-kubelet\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.656086 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.655390    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/04f51c2b-efe9-465f-bf46-f014f03cc131-sys-fs\") pod \"aws-ebs-csi-driver-node-r6425\" (UID: \"04f51c2b-efe9-465f-bf46-f014f03cc131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6425"
Sep 10 20:36:02.656086 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.655414    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e14c5919-af21-4e8f-9783-a2ee2b4885c0-kubelet-config\") pod \"global-pull-secret-syncer-v7c4f\" (UID: \"e14c5919-af21-4e8f-9783-a2ee2b4885c0\") " pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:36:02.656086 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:02.655480    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:36:02.656086 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:02.655535    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e14c5919-af21-4e8f-9783-a2ee2b4885c0-original-pull-secret podName:e14c5919-af21-4e8f-9783-a2ee2b4885c0 nodeName:}" failed. No retries permitted until 2026-09-10 20:36:03.15551608 +0000 UTC m=+3.120584271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e14c5919-af21-4e8f-9783-a2ee2b4885c0-original-pull-secret") pod "global-pull-secret-syncer-v7c4f" (UID: "e14c5919-af21-4e8f-9783-a2ee2b4885c0") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:36:02.656086 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.655693    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b112c171-a796-4f0b-b3ee-42a93807bbe1-etc-sysctl-conf\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.656086 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.655724    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/04f51c2b-efe9-465f-bf46-f014f03cc131-etc-selinux\") pod \"aws-ebs-csi-driver-node-r6425\" (UID: \"04f51c2b-efe9-465f-bf46-f014f03cc131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6425"
Sep 10 20:36:02.657016 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.655755    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.657016 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.655779    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ebbed649-b086-441c-b19c-4fa20fa4f77f-env-overrides\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.657016 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.655783    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b112c171-a796-4f0b-b3ee-42a93807bbe1-etc-kubernetes\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.657016 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.655833    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b112c171-a796-4f0b-b3ee-42a93807bbe1-etc-kubernetes\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.657016 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.655873    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-run-systemd\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.657016 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.655879    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-run-ovn\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.657016 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.655903    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ebbed649-b086-441c-b19c-4fa20fa4f77f-ovnkube-config\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.657016 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.655922    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-node-log\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.657016 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.655968    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-var-lib-openvswitch\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.657016 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.656012    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b112c171-a796-4f0b-b3ee-42a93807bbe1-run\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.657016 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.656044    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b112c171-a796-4f0b-b3ee-42a93807bbe1-etc-sysctl-conf\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.657016 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.656092    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-host-run-ovn-kubernetes\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.657016 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.655835    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-run-ovn\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.657016 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.656132    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b112c171-a796-4f0b-b3ee-42a93807bbe1-etc-systemd\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.657016 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.656158    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-host-cni-bin\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.657016 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.656176    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/04f51c2b-efe9-465f-bf46-f014f03cc131-etc-selinux\") pod \"aws-ebs-csi-driver-node-r6425\" (UID: \"04f51c2b-efe9-465f-bf46-f014f03cc131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6425"
Sep 10 20:36:02.657016 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.656183    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b112c171-a796-4f0b-b3ee-42a93807bbe1-etc-sysctl-d\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.657016 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.656194    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-host-slash\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.657937 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.656210    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b112c171-a796-4f0b-b3ee-42a93807bbe1-sys\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.657937 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.656227    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-host-cni-bin\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.657937 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.656238    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/821d4eb6-5a24-40b4-bee3-472994b9be36-agent-certs\") pod \"konnectivity-agent-vv9cn\" (UID: \"821d4eb6-5a24-40b4-bee3-472994b9be36\") " pod="kube-system/konnectivity-agent-vv9cn"
Sep 10 20:36:02.657937 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.656265    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b112c171-a796-4f0b-b3ee-42a93807bbe1-etc-systemd\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.657937 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.656299    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-host-run-netns\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.657937 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.656324    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-log-socket\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.657937 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.656349    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b112c171-a796-4f0b-b3ee-42a93807bbe1-etc-sysconfig\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.657937 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.656351    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b112c171-a796-4f0b-b3ee-42a93807bbe1-etc-sysctl-d\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.657937 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.656382    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b112c171-a796-4f0b-b3ee-42a93807bbe1-lib-modules\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.657937 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.656395    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b112c171-a796-4f0b-b3ee-42a93807bbe1-etc-sysconfig\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.657937 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.656160    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e14c5919-af21-4e8f-9783-a2ee2b4885c0-dbus\") pod \"global-pull-secret-syncer-v7c4f\" (UID: \"e14c5919-af21-4e8f-9783-a2ee2b4885c0\") " pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:36:02.657937 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.656452    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-log-socket\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.657937 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.656456    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b112c171-a796-4f0b-b3ee-42a93807bbe1-host\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.657937 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.656493    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-host-run-netns\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.657937 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.656406    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b112c171-a796-4f0b-b3ee-42a93807bbe1-host\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.657937 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.656092    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebbed649-b086-441c-b19c-4fa20fa4f77f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.657937 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.656580    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b112c171-a796-4f0b-b3ee-42a93807bbe1-sys\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.657937 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.656582    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b112c171-a796-4f0b-b3ee-42a93807bbe1-lib-modules\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.658888 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.657884    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ebbed649-b086-441c-b19c-4fa20fa4f77f-ovn-node-metrics-cert\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.658888 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.658710    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b112c171-a796-4f0b-b3ee-42a93807bbe1-tmp\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.660222 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.659869    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/821d4eb6-5a24-40b4-bee3-472994b9be36-agent-certs\") pod \"konnectivity-agent-vv9cn\" (UID: \"821d4eb6-5a24-40b4-bee3-472994b9be36\") " pod="kube-system/konnectivity-agent-vv9cn"
Sep 10 20:36:02.661613 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.661574    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b112c171-a796-4f0b-b3ee-42a93807bbe1-etc-tuned\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.669717 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:02.669657    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:36:02.669717 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:02.669685    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:36:02.669717 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:02.669700    2564 projected.go:196] Error preparing data for projected volume kube-api-access-hvr8v for pod openshift-network-diagnostics/network-check-target-wl95n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:36:02.669929 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:02.669765    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e1ea2e3-fc76-40ad-a7a7-111080547273-kube-api-access-hvr8v podName:2e1ea2e3-fc76-40ad-a7a7-111080547273 nodeName:}" failed. No retries permitted until 2026-09-10 20:36:03.169741184 +0000 UTC m=+3.134809373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hvr8v" (UniqueName: "kubernetes.io/projected/2e1ea2e3-fc76-40ad-a7a7-111080547273-kube-api-access-hvr8v") pod "network-check-target-wl95n" (UID: "2e1ea2e3-fc76-40ad-a7a7-111080547273") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:36:02.672099 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.672075    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpmvg\" (UniqueName: \"kubernetes.io/projected/04f51c2b-efe9-465f-bf46-f014f03cc131-kube-api-access-mpmvg\") pod \"aws-ebs-csi-driver-node-r6425\" (UID: \"04f51c2b-efe9-465f-bf46-f014f03cc131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6425"
Sep 10 20:36:02.672507 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.672476    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjl6x\" (UniqueName: \"kubernetes.io/projected/b112c171-a796-4f0b-b3ee-42a93807bbe1-kube-api-access-tjl6x\") pod \"tuned-6sj2b\" (UID: \"b112c171-a796-4f0b-b3ee-42a93807bbe1\") " pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:02.679469 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.679446    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mbnn\" (UniqueName: \"kubernetes.io/projected/ebbed649-b086-441c-b19c-4fa20fa4f77f-kube-api-access-6mbnn\") pod \"ovnkube-node-kn6fl\" (UID: \"ebbed649-b086-441c-b19c-4fa20fa4f77f\") " pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.740870 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.740846    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hqsvv"
Sep 10 20:36:02.754424 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.754400    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2mv5v"
Sep 10 20:36:02.764553 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.764525    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kqb7x"
Sep 10 20:36:02.769855 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.769832    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fxt2l"
Sep 10 20:36:02.777762 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.777742    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qc6bx"
Sep 10 20:36:02.786308 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.786290    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:02.796421 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.796399    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vv9cn"
Sep 10 20:36:02.803321 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.803300    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6425"
Sep 10 20:36:02.810159 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:02.810136    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6sj2b"
Sep 10 20:36:03.059889 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:03.059815    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b-metrics-certs\") pod \"network-metrics-daemon-vrrvq\" (UID: \"e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b\") " pod="openshift-multus/network-metrics-daemon-vrrvq"
Sep 10 20:36:03.060044 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:03.060007    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:36:03.060103 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:03.060089    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b-metrics-certs podName:e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b nodeName:}" failed. No retries permitted until 2026-09-10 20:36:04.060068492 +0000 UTC m=+4.025136682 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b-metrics-certs") pod "network-metrics-daemon-vrrvq" (UID: "e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:36:03.160522 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:03.160490    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e14c5919-af21-4e8f-9783-a2ee2b4885c0-original-pull-secret\") pod \"global-pull-secret-syncer-v7c4f\" (UID: \"e14c5919-af21-4e8f-9783-a2ee2b4885c0\") " pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:36:03.160679 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:03.160619    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:36:03.160740 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:03.160688    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e14c5919-af21-4e8f-9783-a2ee2b4885c0-original-pull-secret podName:e14c5919-af21-4e8f-9783-a2ee2b4885c0 nodeName:}" failed. No retries permitted until 2026-09-10 20:36:04.160669783 +0000 UTC m=+4.125737960 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e14c5919-af21-4e8f-9783-a2ee2b4885c0-original-pull-secret") pod "global-pull-secret-syncer-v7c4f" (UID: "e14c5919-af21-4e8f-9783-a2ee2b4885c0") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:36:03.261500 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:03.261471    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvr8v\" (UniqueName: \"kubernetes.io/projected/2e1ea2e3-fc76-40ad-a7a7-111080547273-kube-api-access-hvr8v\") pod \"network-check-target-wl95n\" (UID: \"2e1ea2e3-fc76-40ad-a7a7-111080547273\") " pod="openshift-network-diagnostics/network-check-target-wl95n"
Sep 10 20:36:03.261634 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:03.261609    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:36:03.261634 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:03.261626    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:36:03.261634 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:03.261634    2564 projected.go:196] Error preparing data for projected volume kube-api-access-hvr8v for pod openshift-network-diagnostics/network-check-target-wl95n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:36:03.261771 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:03.261681    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e1ea2e3-fc76-40ad-a7a7-111080547273-kube-api-access-hvr8v podName:2e1ea2e3-fc76-40ad-a7a7-111080547273 nodeName:}" failed. No retries permitted until 2026-09-10 20:36:04.261667811 +0000 UTC m=+4.226735985 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvr8v" (UniqueName: "kubernetes.io/projected/2e1ea2e3-fc76-40ad-a7a7-111080547273-kube-api-access-hvr8v") pod "network-check-target-wl95n" (UID: "2e1ea2e3-fc76-40ad-a7a7-111080547273") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:36:03.612561 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:03.612499    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 20:31:02 +0000 UTC" deadline="2028-04-27 05:53:34.279819009 +0000 UTC"
Sep 10 20:36:03.613095 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:03.612560    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14265h17m30.667264766s"
Sep 10 20:36:03.780544 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:03.780404    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebbed649_b086_441c_b19c_4fa20fa4f77f.slice/crio-15ef9445407a63e08ad61a469039cf2ad3e6d0cbf6c3886b5f53bf282ec7b1dc WatchSource:0}: Error finding container 15ef9445407a63e08ad61a469039cf2ad3e6d0cbf6c3886b5f53bf282ec7b1dc: Status 404 returned error can't find the container with id 15ef9445407a63e08ad61a469039cf2ad3e6d0cbf6c3886b5f53bf282ec7b1dc
Sep 10 20:36:03.785611 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:03.785582    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f51c2b_efe9_465f_bf46_f014f03cc131.slice/crio-c5df5784aa4cc05775fee0ec7ccffd6044498359578239ef6cd3d1aa9cef3107 WatchSource:0}: Error finding container c5df5784aa4cc05775fee0ec7ccffd6044498359578239ef6cd3d1aa9cef3107: Status 404 returned error can't find the container with id c5df5784aa4cc05775fee0ec7ccffd6044498359578239ef6cd3d1aa9cef3107
Sep 10 20:36:03.786863 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:03.786839    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b63c96_edab_42b0_8af3_f9db7d20867a.slice/crio-c2f334bbcfc4aeb655d9e3ed938b331ad94d0158d82257267d34f9d2b90a714c WatchSource:0}: Error finding container c2f334bbcfc4aeb655d9e3ed938b331ad94d0158d82257267d34f9d2b90a714c: Status 404 returned error can't find the container with id c2f334bbcfc4aeb655d9e3ed938b331ad94d0158d82257267d34f9d2b90a714c
Sep 10 20:36:04.068166 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:04.068137    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b-metrics-certs\") pod \"network-metrics-daemon-vrrvq\" (UID: \"e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b\") " pod="openshift-multus/network-metrics-daemon-vrrvq"
Sep 10 20:36:04.068303 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:04.068283    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:36:04.068367 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:04.068341    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b-metrics-certs podName:e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b nodeName:}" failed. No retries permitted until 2026-09-10 20:36:06.068326124 +0000 UTC m=+6.033394298 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b-metrics-certs") pod "network-metrics-daemon-vrrvq" (UID: "e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:36:04.169101 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:04.168874    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e14c5919-af21-4e8f-9783-a2ee2b4885c0-original-pull-secret\") pod \"global-pull-secret-syncer-v7c4f\" (UID: \"e14c5919-af21-4e8f-9783-a2ee2b4885c0\") " pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:36:04.169101 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:04.169063    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:36:04.169310 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:04.169134    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e14c5919-af21-4e8f-9783-a2ee2b4885c0-original-pull-secret podName:e14c5919-af21-4e8f-9783-a2ee2b4885c0 nodeName:}" failed. No retries permitted until 2026-09-10 20:36:06.169114649 +0000 UTC m=+6.134182826 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e14c5919-af21-4e8f-9783-a2ee2b4885c0-original-pull-secret") pod "global-pull-secret-syncer-v7c4f" (UID: "e14c5919-af21-4e8f-9783-a2ee2b4885c0") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:36:04.269352 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:04.269315    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvr8v\" (UniqueName: \"kubernetes.io/projected/2e1ea2e3-fc76-40ad-a7a7-111080547273-kube-api-access-hvr8v\") pod \"network-check-target-wl95n\" (UID: \"2e1ea2e3-fc76-40ad-a7a7-111080547273\") " pod="openshift-network-diagnostics/network-check-target-wl95n"
Sep 10 20:36:04.269514 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:04.269481    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:36:04.269514 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:04.269501    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:36:04.269514 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:04.269515    2564 projected.go:196] Error preparing data for projected volume kube-api-access-hvr8v for pod openshift-network-diagnostics/network-check-target-wl95n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:36:04.269686 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:04.269571    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e1ea2e3-fc76-40ad-a7a7-111080547273-kube-api-access-hvr8v podName:2e1ea2e3-fc76-40ad-a7a7-111080547273 nodeName:}" failed. No retries permitted until 2026-09-10 20:36:06.269551696 +0000 UTC m=+6.234619873 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvr8v" (UniqueName: "kubernetes.io/projected/2e1ea2e3-fc76-40ad-a7a7-111080547273-kube-api-access-hvr8v") pod "network-check-target-wl95n" (UID: "2e1ea2e3-fc76-40ad-a7a7-111080547273") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:36:04.539150 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:04.538317    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl95n"
Sep 10 20:36:04.539150 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:04.538443    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wl95n" podUID="2e1ea2e3-fc76-40ad-a7a7-111080547273"
Sep 10 20:36:04.539150 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:04.538848    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrrvq"
Sep 10 20:36:04.539150 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:04.538962    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrrvq" podUID="e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b"
Sep 10 20:36:04.539150 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:04.538990    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:36:04.539150 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:04.539061    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7c4f" podUID="e14c5919-af21-4e8f-9783-a2ee2b4885c0"
Sep 10 20:36:04.562720 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:04.562674    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-185.ec2.internal" event={"ID":"b75d22eafcad6f42168b54bcbeb68b91","Type":"ContainerStarted","Data":"0ae2e01edf42b5c5723f3fd7865f454cdfee789427db7ec1d727e2ebe4c78c0e"}
Sep 10 20:36:04.573506 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:04.573477    2564 generic.go:356] "Generic (PLEG): container finished" podID="7ae9783e5e544859626d5a53f1ccb92a" containerID="92fda9d8b26f9d2a2ebf2d1a022d686f6033b4591a82be22d013d1168f5a568f" exitCode=0
Sep 10 20:36:04.573615 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:04.573571    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-185.ec2.internal" event={"ID":"7ae9783e5e544859626d5a53f1ccb92a","Type":"ContainerDied","Data":"92fda9d8b26f9d2a2ebf2d1a022d686f6033b4591a82be22d013d1168f5a568f"}
Sep 10 20:36:04.594542 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:04.594485    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fxt2l" event={"ID":"ad5d5af7-f551-415c-9a17-fb89dd6bbb0a","Type":"ContainerStarted","Data":"d9833b7d2e7602a10a5744201fbaac79dd1fcf446b1ec59e2e2221a11fb69e19"}
Sep 10 20:36:04.613399 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:04.613309    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 20:31:02 +0000 UTC" deadline="2028-05-31 19:11:11.44059624 +0000 UTC"
Sep 10 20:36:04.613399 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:04.613361    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15094h35m6.827239356s"
Sep 10 20:36:04.618631 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:04.618572    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2mv5v" event={"ID":"a4bb15a1-d45e-4974-b648-8cffc50af95b","Type":"ContainerStarted","Data":"fa3f949edb92dd589c6862e38919116cb60a18a69b27514c70252e624b0fcd52"}
Sep 10 20:36:04.639413 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:04.639383    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vv9cn" event={"ID":"821d4eb6-5a24-40b4-bee3-472994b9be36","Type":"ContainerStarted","Data":"81dedef326bc9c7d3dd95be4ff3b0148ddced886328a32042632f4546982fd24"}
Sep 10 20:36:04.645234 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:04.645163    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6425" event={"ID":"04f51c2b-efe9-465f-bf46-f014f03cc131","Type":"ContainerStarted","Data":"c5df5784aa4cc05775fee0ec7ccffd6044498359578239ef6cd3d1aa9cef3107"}
Sep 10 20:36:04.655899 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:04.655809    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl" event={"ID":"ebbed649-b086-441c-b19c-4fa20fa4f77f","Type":"ContainerStarted","Data":"15ef9445407a63e08ad61a469039cf2ad3e6d0cbf6c3886b5f53bf282ec7b1dc"}
Sep 10 20:36:04.660211 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:04.660185    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6sj2b" event={"ID":"b112c171-a796-4f0b-b3ee-42a93807bbe1","Type":"ContainerStarted","Data":"f023311309f109f7c6672109a48efcfabb09db8cccdfe81b2701177b8198a125"}
Sep 10 20:36:04.665842 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:04.665806    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kqb7x" event={"ID":"af88567d-1c33-4670-acc8-484e2d8b11d4","Type":"ContainerStarted","Data":"e41356ed0e06cc7b234e53a95dd0d502fc64b1f8035994abc95f74cd415eab43"}
Sep 10 20:36:04.674764 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:04.674711    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-185.ec2.internal" podStartSLOduration=2.674694123 podStartE2EDuration="2.674694123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:36:04.637430509 +0000 UTC m=+4.602498709" watchObservedRunningTime="2026-09-10 20:36:04.674694123 +0000 UTC m=+4.639762326"
Sep 10 20:36:04.692313 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:04.692235    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qc6bx" event={"ID":"0447592e-554b-4f55-a206-d5aa4a5e4013","Type":"ContainerStarted","Data":"d08fa9b30850510c6198a8c38f6cc7bacc8ea4d06b05eef5ade6de923c18e412"}
Sep 10 20:36:04.695972 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:04.695947    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hqsvv" event={"ID":"b1b63c96-edab-42b0-8af3-f9db7d20867a","Type":"ContainerStarted","Data":"c2f334bbcfc4aeb655d9e3ed938b331ad94d0158d82257267d34f9d2b90a714c"}
Sep 10 20:36:05.712391 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:05.712312    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-185.ec2.internal" event={"ID":"7ae9783e5e544859626d5a53f1ccb92a","Type":"ContainerStarted","Data":"f0a14ab65f1fd037a51ba39c89c5082e0a3eb31fac973aa95d111c77dafee5e1"}
Sep 10 20:36:06.088007 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:06.087918    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b-metrics-certs\") pod \"network-metrics-daemon-vrrvq\" (UID: \"e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b\") " pod="openshift-multus/network-metrics-daemon-vrrvq"
Sep 10 20:36:06.088168 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:06.088070    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:36:06.088168 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:06.088129    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b-metrics-certs podName:e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b nodeName:}" failed. No retries permitted until 2026-09-10 20:36:10.08811159 +0000 UTC m=+10.053179772 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b-metrics-certs") pod "network-metrics-daemon-vrrvq" (UID: "e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:36:06.188698 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:06.188662    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e14c5919-af21-4e8f-9783-a2ee2b4885c0-original-pull-secret\") pod \"global-pull-secret-syncer-v7c4f\" (UID: \"e14c5919-af21-4e8f-9783-a2ee2b4885c0\") " pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:36:06.188863 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:06.188802    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:36:06.188863 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:06.188859    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e14c5919-af21-4e8f-9783-a2ee2b4885c0-original-pull-secret podName:e14c5919-af21-4e8f-9783-a2ee2b4885c0 nodeName:}" failed. No retries permitted until 2026-09-10 20:36:10.188842333 +0000 UTC m=+10.153910511 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e14c5919-af21-4e8f-9783-a2ee2b4885c0-original-pull-secret") pod "global-pull-secret-syncer-v7c4f" (UID: "e14c5919-af21-4e8f-9783-a2ee2b4885c0") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:36:06.289208 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:06.289162    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvr8v\" (UniqueName: \"kubernetes.io/projected/2e1ea2e3-fc76-40ad-a7a7-111080547273-kube-api-access-hvr8v\") pod \"network-check-target-wl95n\" (UID: \"2e1ea2e3-fc76-40ad-a7a7-111080547273\") " pod="openshift-network-diagnostics/network-check-target-wl95n"
Sep 10 20:36:06.289366 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:06.289324    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:36:06.289366 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:06.289345    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:36:06.289366 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:06.289357    2564 projected.go:196] Error preparing data for projected volume kube-api-access-hvr8v for pod openshift-network-diagnostics/network-check-target-wl95n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:36:06.289538 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:06.289413    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e1ea2e3-fc76-40ad-a7a7-111080547273-kube-api-access-hvr8v podName:2e1ea2e3-fc76-40ad-a7a7-111080547273 nodeName:}" failed. No retries permitted until 2026-09-10 20:36:10.289393445 +0000 UTC m=+10.254461622 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvr8v" (UniqueName: "kubernetes.io/projected/2e1ea2e3-fc76-40ad-a7a7-111080547273-kube-api-access-hvr8v") pod "network-check-target-wl95n" (UID: "2e1ea2e3-fc76-40ad-a7a7-111080547273") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:36:06.539764 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:06.535359    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:36:06.539764 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:06.535372    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl95n"
Sep 10 20:36:06.539764 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:06.535499    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7c4f" podUID="e14c5919-af21-4e8f-9783-a2ee2b4885c0"
Sep 10 20:36:06.539764 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:06.535602    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wl95n" podUID="2e1ea2e3-fc76-40ad-a7a7-111080547273"
Sep 10 20:36:06.539764 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:06.535660    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrrvq"
Sep 10 20:36:06.539764 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:06.535757    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrrvq" podUID="e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b"
Sep 10 20:36:08.536628 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:08.536583    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:36:08.537070 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:08.536708    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7c4f" podUID="e14c5919-af21-4e8f-9783-a2ee2b4885c0"
Sep 10 20:36:08.537153 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:08.537076    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl95n"
Sep 10 20:36:08.537270 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:08.537162    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wl95n" podUID="2e1ea2e3-fc76-40ad-a7a7-111080547273"
Sep 10 20:36:08.537270 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:08.537249    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrrvq"
Sep 10 20:36:08.537382 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:08.537338    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrrvq" podUID="e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b"
Sep 10 20:36:10.119044 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:10.119011    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b-metrics-certs\") pod \"network-metrics-daemon-vrrvq\" (UID: \"e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b\") " pod="openshift-multus/network-metrics-daemon-vrrvq"
Sep 10 20:36:10.119543 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:10.119258    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:36:10.119543 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:10.119330    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b-metrics-certs podName:e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b nodeName:}" failed. No retries permitted until 2026-09-10 20:36:18.119311559 +0000 UTC m=+18.084379749 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b-metrics-certs") pod "network-metrics-daemon-vrrvq" (UID: "e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:36:10.219984 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:10.219875    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e14c5919-af21-4e8f-9783-a2ee2b4885c0-original-pull-secret\") pod \"global-pull-secret-syncer-v7c4f\" (UID: \"e14c5919-af21-4e8f-9783-a2ee2b4885c0\") " pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:36:10.220154 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:10.220049    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:36:10.220154 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:10.220119    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e14c5919-af21-4e8f-9783-a2ee2b4885c0-original-pull-secret podName:e14c5919-af21-4e8f-9783-a2ee2b4885c0 nodeName:}" failed. No retries permitted until 2026-09-10 20:36:18.220099953 +0000 UTC m=+18.185168134 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e14c5919-af21-4e8f-9783-a2ee2b4885c0-original-pull-secret") pod "global-pull-secret-syncer-v7c4f" (UID: "e14c5919-af21-4e8f-9783-a2ee2b4885c0") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:36:10.321092 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:10.321050    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvr8v\" (UniqueName: \"kubernetes.io/projected/2e1ea2e3-fc76-40ad-a7a7-111080547273-kube-api-access-hvr8v\") pod \"network-check-target-wl95n\" (UID: \"2e1ea2e3-fc76-40ad-a7a7-111080547273\") " pod="openshift-network-diagnostics/network-check-target-wl95n"
Sep 10 20:36:10.321270 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:10.321243    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:36:10.321270 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:10.321267    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:36:10.321370 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:10.321282    2564 projected.go:196] Error preparing data for projected volume kube-api-access-hvr8v for pod openshift-network-diagnostics/network-check-target-wl95n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:36:10.321370 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:10.321344    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e1ea2e3-fc76-40ad-a7a7-111080547273-kube-api-access-hvr8v podName:2e1ea2e3-fc76-40ad-a7a7-111080547273 nodeName:}" failed. No retries permitted until 2026-09-10 20:36:18.32132563 +0000 UTC m=+18.286393806 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvr8v" (UniqueName: "kubernetes.io/projected/2e1ea2e3-fc76-40ad-a7a7-111080547273-kube-api-access-hvr8v") pod "network-check-target-wl95n" (UID: "2e1ea2e3-fc76-40ad-a7a7-111080547273") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:36:10.535379 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:10.535309    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrrvq"
Sep 10 20:36:10.535379 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:10.535331    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl95n"
Sep 10 20:36:10.535379 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:10.535312    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:36:10.535609 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:10.535421    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wl95n" podUID="2e1ea2e3-fc76-40ad-a7a7-111080547273"
Sep 10 20:36:10.535844 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:10.535812    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7c4f" podUID="e14c5919-af21-4e8f-9783-a2ee2b4885c0"
Sep 10 20:36:10.535933 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:10.535907    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrrvq" podUID="e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b"
Sep 10 20:36:12.538048 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:12.537373    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:36:12.538048 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:12.537496    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7c4f" podUID="e14c5919-af21-4e8f-9783-a2ee2b4885c0"
Sep 10 20:36:12.538048 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:12.537836    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl95n"
Sep 10 20:36:12.538048 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:12.537902    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wl95n" podUID="2e1ea2e3-fc76-40ad-a7a7-111080547273"
Sep 10 20:36:12.538048 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:12.537955    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrrvq"
Sep 10 20:36:12.538048 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:12.538004    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrrvq" podUID="e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b"
Sep 10 20:36:14.534795 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:14.534726    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl95n"
Sep 10 20:36:14.535284 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:14.534838    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrrvq"
Sep 10 20:36:14.535284 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:14.534847    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wl95n" podUID="2e1ea2e3-fc76-40ad-a7a7-111080547273"
Sep 10 20:36:14.535284 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:14.534923    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrrvq" podUID="e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b"
Sep 10 20:36:14.535284 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:14.534965    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:36:14.535284 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:14.535086    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7c4f" podUID="e14c5919-af21-4e8f-9783-a2ee2b4885c0"
Sep 10 20:36:16.534834 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:16.534801    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl95n"
Sep 10 20:36:16.535311 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:16.534801    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:36:16.535311 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:16.534922    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wl95n" podUID="2e1ea2e3-fc76-40ad-a7a7-111080547273"
Sep 10 20:36:16.535311 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:16.535024    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7c4f" podUID="e14c5919-af21-4e8f-9783-a2ee2b4885c0"
Sep 10 20:36:16.535311 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:16.534801    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrrvq"
Sep 10 20:36:16.535311 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:16.535154    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrrvq" podUID="e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b"
Sep 10 20:36:18.181448 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:18.181415    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b-metrics-certs\") pod \"network-metrics-daemon-vrrvq\" (UID: \"e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b\") " pod="openshift-multus/network-metrics-daemon-vrrvq"
Sep 10 20:36:18.181946 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:18.181523    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:36:18.181946 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:18.181574    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b-metrics-certs podName:e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b nodeName:}" failed. No retries permitted until 2026-09-10 20:36:34.181561039 +0000 UTC m=+34.146629212 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b-metrics-certs") pod "network-metrics-daemon-vrrvq" (UID: "e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:36:18.281784 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:18.281751    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e14c5919-af21-4e8f-9783-a2ee2b4885c0-original-pull-secret\") pod \"global-pull-secret-syncer-v7c4f\" (UID: \"e14c5919-af21-4e8f-9783-a2ee2b4885c0\") " pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:36:18.281948 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:18.281879    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:36:18.281948 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:18.281942    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e14c5919-af21-4e8f-9783-a2ee2b4885c0-original-pull-secret podName:e14c5919-af21-4e8f-9783-a2ee2b4885c0 nodeName:}" failed. No retries permitted until 2026-09-10 20:36:34.281924921 +0000 UTC m=+34.246993108 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e14c5919-af21-4e8f-9783-a2ee2b4885c0-original-pull-secret") pod "global-pull-secret-syncer-v7c4f" (UID: "e14c5919-af21-4e8f-9783-a2ee2b4885c0") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:36:18.382129 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:18.382099    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvr8v\" (UniqueName: \"kubernetes.io/projected/2e1ea2e3-fc76-40ad-a7a7-111080547273-kube-api-access-hvr8v\") pod \"network-check-target-wl95n\" (UID: \"2e1ea2e3-fc76-40ad-a7a7-111080547273\") " pod="openshift-network-diagnostics/network-check-target-wl95n"
Sep 10 20:36:18.382302 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:18.382234    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:36:18.382302 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:18.382258    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:36:18.382302 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:18.382271    2564 projected.go:196] Error preparing data for projected volume kube-api-access-hvr8v for pod openshift-network-diagnostics/network-check-target-wl95n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:36:18.382461 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:18.382333    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e1ea2e3-fc76-40ad-a7a7-111080547273-kube-api-access-hvr8v podName:2e1ea2e3-fc76-40ad-a7a7-111080547273 nodeName:}" failed. No retries permitted until 2026-09-10 20:36:34.382314837 +0000 UTC m=+34.347383024 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvr8v" (UniqueName: "kubernetes.io/projected/2e1ea2e3-fc76-40ad-a7a7-111080547273-kube-api-access-hvr8v") pod "network-check-target-wl95n" (UID: "2e1ea2e3-fc76-40ad-a7a7-111080547273") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:36:18.535374 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:18.535301    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrrvq"
Sep 10 20:36:18.535515 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:18.535301    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:36:18.535515 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:18.535429    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrrvq" podUID="e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b"
Sep 10 20:36:18.535515 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:18.535311    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl95n"
Sep 10 20:36:18.535515 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:18.535501    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7c4f" podUID="e14c5919-af21-4e8f-9783-a2ee2b4885c0"
Sep 10 20:36:18.535668 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:18.535559    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wl95n" podUID="2e1ea2e3-fc76-40ad-a7a7-111080547273"
Sep 10 20:36:20.536193 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:20.535971    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl95n"
Sep 10 20:36:20.536193 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:20.536056    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:36:20.536193 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:20.536101    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrrvq"
Sep 10 20:36:20.536193 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:20.536091    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wl95n" podUID="2e1ea2e3-fc76-40ad-a7a7-111080547273"
Sep 10 20:36:20.536779 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:20.536210    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrrvq" podUID="e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b"
Sep 10 20:36:20.536779 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:20.536298    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7c4f" podUID="e14c5919-af21-4e8f-9783-a2ee2b4885c0"
Sep 10 20:36:21.745210 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:21.745010    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fxt2l" event={"ID":"ad5d5af7-f551-415c-9a17-fb89dd6bbb0a","Type":"ContainerStarted","Data":"922b1d242d30cb4477072be175a176fb195f317453e54355d7ce0cf6ba1e47e4"}
Sep 10 20:36:21.747314 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:21.747285    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vv9cn" event={"ID":"821d4eb6-5a24-40b4-bee3-472994b9be36","Type":"ContainerStarted","Data":"77fde1a8a4c7c7596e743ba864effe4b20df37286b6398d436af5b2e29879524"}
Sep 10 20:36:21.750220 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:21.750197    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6425" event={"ID":"04f51c2b-efe9-465f-bf46-f014f03cc131","Type":"ContainerStarted","Data":"6b246d12b7e115bc5075a5829809dea8c7cd6b1856a3c0c76d35c421f701770f"}
Sep 10 20:36:21.752161 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:21.752132    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl" event={"ID":"ebbed649-b086-441c-b19c-4fa20fa4f77f","Type":"ContainerStarted","Data":"3db38c09a0e09717470df186ebe5c5b59bf6119e4f286bd8442024cbafd80eee"}
Sep 10 20:36:21.752534 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:21.752508    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl" event={"ID":"ebbed649-b086-441c-b19c-4fa20fa4f77f","Type":"ContainerStarted","Data":"6afc5a0c1127eca13d423f5801ab0213e96a5860b35554d540a5efefe874d18c"}
Sep 10 20:36:21.754089 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:21.754016    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6sj2b" event={"ID":"b112c171-a796-4f0b-b3ee-42a93807bbe1","Type":"ContainerStarted","Data":"080ae0a9dbd107c0290e96e73b346bb8b09f221ede22871f25855e195145dcf0"}
Sep 10 20:36:21.756014 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:21.755988    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kqb7x" event={"ID":"af88567d-1c33-4670-acc8-484e2d8b11d4","Type":"ContainerStarted","Data":"986200a57d955c96958b773c4df45e4f5af50f29dd490cf444be49d37d7a408e"}
Sep 10 20:36:21.757466 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:21.757433    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qc6bx" event={"ID":"0447592e-554b-4f55-a206-d5aa4a5e4013","Type":"ContainerStarted","Data":"13664ffb846ea34b0ee4f7305592a693f6b3e186d8edb56976a79a4196f663da"}
Sep 10 20:36:21.759013 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:21.758994    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hqsvv" event={"ID":"b1b63c96-edab-42b0-8af3-f9db7d20867a","Type":"ContainerStarted","Data":"afcfe5fb56db3203cf6e3ca516da0828a1627902d5512a5bafce842fe324b6c2"}
Sep 10 20:36:21.760888 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:21.760841    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-185.ec2.internal" podStartSLOduration=19.760824669 podStartE2EDuration="19.760824669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:36:05.741073932 +0000 UTC m=+5.706142131" watchObservedRunningTime="2026-09-10 20:36:21.760824669 +0000 UTC m=+21.725892872"
Sep 10 20:36:21.761474 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:21.761439    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fxt2l" podStartSLOduration=6.858509023 podStartE2EDuration="20.761428101s" totalImagesPullingTime="13.902919078s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:36:21.760460338 +0000 UTC m=+21.725528536" watchObservedRunningTime="2026-09-10 20:36:21.761428101 +0000 UTC m=+21.726496361"
Sep 10 20:36:21.775526 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:21.775493    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-kqb7x" podStartSLOduration=3.087505536 podStartE2EDuration="20.775476489s" totalImagesPullingTime="17.687970953s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:36:21.77541514 +0000 UTC m=+21.740483336" watchObservedRunningTime="2026-09-10 20:36:21.775476489 +0000 UTC m=+21.740544684"
Sep 10 20:36:21.796261 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:21.796228    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6sj2b" podStartSLOduration=3.115242537 podStartE2EDuration="20.796218654s" totalImagesPullingTime="17.680976117s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:36:21.796089134 +0000 UTC m=+21.761157329" watchObservedRunningTime="2026-09-10 20:36:21.796218654 +0000 UTC m=+21.761286827"
Sep 10 20:36:21.851270 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:21.851231    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qc6bx" podStartSLOduration=3.101867052 podStartE2EDuration="20.851216546s" totalImagesPullingTime="17.749349494s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:36:21.850697295 +0000 UTC m=+21.815765491" watchObservedRunningTime="2026-09-10 20:36:21.851216546 +0000 UTC m=+21.816284741"
Sep 10 20:36:21.867823 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:21.867770    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vv9cn" podStartSLOduration=6.958336842 podStartE2EDuration="20.867750335s" totalImagesPullingTime="13.909413493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:36:21.866749188 +0000 UTC m=+21.831817397" watchObservedRunningTime="2026-09-10 20:36:21.867750335 +0000 UTC m=+21.832818533"
Sep 10 20:36:22.534732 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:22.534702    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl95n"
Sep 10 20:36:22.534869 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:22.534806    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrrvq"
Sep 10 20:36:22.534869 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:22.534818    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:36:22.534869 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:22.534820    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wl95n" podUID="2e1ea2e3-fc76-40ad-a7a7-111080547273"
Sep 10 20:36:22.535016 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:22.534917    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrrvq" podUID="e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b"
Sep 10 20:36:22.535016 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:22.534996    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7c4f" podUID="e14c5919-af21-4e8f-9783-a2ee2b4885c0"
Sep 10 20:36:22.762125 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:22.762061    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2mv5v" event={"ID":"a4bb15a1-d45e-4974-b648-8cffc50af95b","Type":"ContainerStarted","Data":"851161a5ce8047ee2b08068c436e3c7e6b929ef68270c4ac6898327a47930cdd"}
Sep 10 20:36:22.764943 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:22.764922    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl" event={"ID":"ebbed649-b086-441c-b19c-4fa20fa4f77f","Type":"ContainerStarted","Data":"824cb2858c7665617783a81e2ad89c45a73349afd10b0936e00ede48b0b6aad7"}
Sep 10 20:36:22.765060 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:22.764990    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl" event={"ID":"ebbed649-b086-441c-b19c-4fa20fa4f77f","Type":"ContainerStarted","Data":"45ddd769360e0d534517bbfed8d5f0dc51333c6545663af8d465094a6e3bf18d"}
Sep 10 20:36:22.765060 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:22.765039    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl" event={"ID":"ebbed649-b086-441c-b19c-4fa20fa4f77f","Type":"ContainerStarted","Data":"d461e43eb056e1728c3441b989a51710db686556399cf0db20e6d215c5342f91"}
Sep 10 20:36:22.765157 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:22.765084    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl" event={"ID":"ebbed649-b086-441c-b19c-4fa20fa4f77f","Type":"ContainerStarted","Data":"14e9eec3f9a7542391b76360ffc0461ea6d17779f9f53a613345791befcace8f"}
Sep 10 20:36:22.766376 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:22.766358    2564 generic.go:356] "Generic (PLEG): container finished" podID="b1b63c96-edab-42b0-8af3-f9db7d20867a" containerID="afcfe5fb56db3203cf6e3ca516da0828a1627902d5512a5bafce842fe324b6c2" exitCode=0
Sep 10 20:36:22.766476 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:22.766452    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hqsvv" event={"ID":"b1b63c96-edab-42b0-8af3-f9db7d20867a","Type":"ContainerDied","Data":"afcfe5fb56db3203cf6e3ca516da0828a1627902d5512a5bafce842fe324b6c2"}
Sep 10 20:36:22.779793 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:22.779758    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2mv5v" podStartSLOduration=4.08916756 podStartE2EDuration="21.779746115s" totalImagesPullingTime="17.690578555s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:36:22.779557031 +0000 UTC m=+22.744625236" watchObservedRunningTime="2026-09-10 20:36:22.779746115 +0000 UTC m=+22.744814310"
Sep 10 20:36:23.181903 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:23.181884    2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 20:36:23.497251 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:23.497140    2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T20:36:23.181899525Z","UUID":"6e15f661-bc45-4b02-86f8-2e15fd1258fc","Handler":null,"Name":"","Endpoint":""}
Sep 10 20:36:23.498496 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:23.498480    2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 20:36:23.498595 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:23.498506    2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 20:36:23.618388 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:23.618362    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vv9cn"
Sep 10 20:36:23.618824 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:23.618807    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vv9cn"
Sep 10 20:36:23.769609 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:23.769557    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6425" event={"ID":"04f51c2b-efe9-465f-bf46-f014f03cc131","Type":"ContainerStarted","Data":"684c05e4f579e2563dd9d099cc66e4e70a2ee24d34f3574fb797a6c15bcbf084"}
Sep 10 20:36:23.769915 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:23.769872    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vv9cn"
Sep 10 20:36:23.770262 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:23.770245    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vv9cn"
Sep 10 20:36:24.534895 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:24.534824    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:36:24.535050 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:24.534832    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrrvq"
Sep 10 20:36:24.535050 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:24.534955    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7c4f" podUID="e14c5919-af21-4e8f-9783-a2ee2b4885c0"
Sep 10 20:36:24.535050 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:24.534830    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl95n"
Sep 10 20:36:24.535207 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:24.535049    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrrvq" podUID="e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b"
Sep 10 20:36:24.535207 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:24.535105    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wl95n" podUID="2e1ea2e3-fc76-40ad-a7a7-111080547273"
Sep 10 20:36:24.775434 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:24.775398    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6425" event={"ID":"04f51c2b-efe9-465f-bf46-f014f03cc131","Type":"ContainerStarted","Data":"6395488a4e441b3c05829d18a4b2e422aa642c75939cb875f00d09445dd89375"}
Sep 10 20:36:24.779685 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:24.779647    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl" event={"ID":"ebbed649-b086-441c-b19c-4fa20fa4f77f","Type":"ContainerStarted","Data":"fceefdfd799662661d6cec9185179c5ba835d17427d9718430be7cdb3b4eea5c"}
Sep 10 20:36:24.794083 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:24.794000    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6425" podStartSLOduration=3.705644045 podStartE2EDuration="23.793984159s" totalImagesPullingTime="20.088340114s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:36:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:36:24.793255572 +0000 UTC m=+24.758323779" watchObservedRunningTime="2026-09-10 20:36:24.793984159 +0000 UTC m=+24.759052356"
Sep 10 20:36:26.535435 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:26.535207    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrrvq"
Sep 10 20:36:26.535873 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:26.535212    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:36:26.535873 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:26.535544    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrrvq" podUID="e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b"
Sep 10 20:36:26.535873 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:26.535262    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl95n"
Sep 10 20:36:26.535873 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:26.535626    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7c4f" podUID="e14c5919-af21-4e8f-9783-a2ee2b4885c0"
Sep 10 20:36:26.535873 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:26.535693    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wl95n" podUID="2e1ea2e3-fc76-40ad-a7a7-111080547273"
Sep 10 20:36:28.534507 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:28.534439    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:36:28.535243 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:28.534544    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7c4f" podUID="e14c5919-af21-4e8f-9783-a2ee2b4885c0"
Sep 10 20:36:28.535243 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:28.534440    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl95n"
Sep 10 20:36:28.535243 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:28.534678    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wl95n" podUID="2e1ea2e3-fc76-40ad-a7a7-111080547273"
Sep 10 20:36:28.535243 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:28.534686    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrrvq"
Sep 10 20:36:28.535243 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:28.534750    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrrvq" podUID="e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b"
Sep 10 20:36:28.796872 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:28.796797    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl" event={"ID":"ebbed649-b086-441c-b19c-4fa20fa4f77f","Type":"ContainerStarted","Data":"ea936974e14549acca5f9ba90400f0f230246afb54f10d71d80ed02138248235"}
Sep 10 20:36:28.797181 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:28.797135    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:28.797318 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:28.797225    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:28.799257 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:28.799230    2564 generic.go:356] "Generic (PLEG): container finished" podID="b1b63c96-edab-42b0-8af3-f9db7d20867a" containerID="d13e29cd6d3b43973ccd5be19f41b5f4df0355849a473f5b2c5118a8c4e8322a" exitCode=0
Sep 10 20:36:28.799364 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:28.799281    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hqsvv" event={"ID":"b1b63c96-edab-42b0-8af3-f9db7d20867a","Type":"ContainerDied","Data":"d13e29cd6d3b43973ccd5be19f41b5f4df0355849a473f5b2c5118a8c4e8322a"}
Sep 10 20:36:28.814254 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:28.814234    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:28.814444 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:28.814429    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:28.835598 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:28.835531    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl" podStartSLOduration=10.079039317 podStartE2EDuration="27.835517801s" totalImagesPullingTime="17.756478484s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:36:28.835072604 +0000 UTC m=+28.800140799" watchObservedRunningTime="2026-09-10 20:36:28.835517801 +0000 UTC m=+28.800585995"
Sep 10 20:36:29.801365 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:29.801335    2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 10 20:36:29.940106 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:29.940073    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v7c4f"]
Sep 10 20:36:29.940256 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:29.940239    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:36:29.940437 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:29.940339    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7c4f" podUID="e14c5919-af21-4e8f-9783-a2ee2b4885c0"
Sep 10 20:36:29.944414 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:29.944391    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wl95n"]
Sep 10 20:36:29.944511 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:29.944486    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl95n"
Sep 10 20:36:29.944595 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:29.944575    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wl95n" podUID="2e1ea2e3-fc76-40ad-a7a7-111080547273"
Sep 10 20:36:29.959065 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:29.959044    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vrrvq"]
Sep 10 20:36:29.959180 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:29.959130    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrrvq"
Sep 10 20:36:29.959242 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:29.959227    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrrvq" podUID="e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b"
Sep 10 20:36:30.426756 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:30.426726    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:36:30.805938 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:30.805867    2564 generic.go:356] "Generic (PLEG): container finished" podID="b1b63c96-edab-42b0-8af3-f9db7d20867a" containerID="77a42126ae2daff49eaa092d63cde688b66129524d316761832ec3ca37157585" exitCode=0
Sep 10 20:36:30.806270 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:30.805944    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hqsvv" event={"ID":"b1b63c96-edab-42b0-8af3-f9db7d20867a","Type":"ContainerDied","Data":"77a42126ae2daff49eaa092d63cde688b66129524d316761832ec3ca37157585"}
Sep 10 20:36:31.535045 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:31.534980    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl95n"
Sep 10 20:36:31.535196 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:31.534984    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:36:31.535196 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:31.535066    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wl95n" podUID="2e1ea2e3-fc76-40ad-a7a7-111080547273"
Sep 10 20:36:31.535196 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:31.535094    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrrvq"
Sep 10 20:36:31.535196 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:31.535162    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7c4f" podUID="e14c5919-af21-4e8f-9783-a2ee2b4885c0"
Sep 10 20:36:31.535346 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:31.535229    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrrvq" podUID="e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b"
Sep 10 20:36:32.813265 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:32.813190    2564 generic.go:356] "Generic (PLEG): container finished" podID="b1b63c96-edab-42b0-8af3-f9db7d20867a" containerID="87392481dac1918a0834cffd7f2b6303e4cb07e8c26b73b2e7f999ac0c009c73" exitCode=0
Sep 10 20:36:32.813265 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:32.813240    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hqsvv" event={"ID":"b1b63c96-edab-42b0-8af3-f9db7d20867a","Type":"ContainerDied","Data":"87392481dac1918a0834cffd7f2b6303e4cb07e8c26b73b2e7f999ac0c009c73"}
Sep 10 20:36:33.534795 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:33.534571    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:36:33.534938 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:33.534615    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl95n"
Sep 10 20:36:33.534938 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:33.534887    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7c4f" podUID="e14c5919-af21-4e8f-9783-a2ee2b4885c0"
Sep 10 20:36:33.535039 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:33.534626    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrrvq"
Sep 10 20:36:33.535039 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:33.534944    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wl95n" podUID="2e1ea2e3-fc76-40ad-a7a7-111080547273"
Sep 10 20:36:33.535118 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:33.535078    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrrvq" podUID="e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b"
Sep 10 20:36:34.201313 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.201287    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b-metrics-certs\") pod \"network-metrics-daemon-vrrvq\" (UID: \"e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b\") " pod="openshift-multus/network-metrics-daemon-vrrvq"
Sep 10 20:36:34.201628 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:34.201396    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:36:34.201628 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:34.201443    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b-metrics-certs podName:e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b nodeName:}" failed. No retries permitted until 2026-09-10 20:37:06.20143034 +0000 UTC m=+66.166498512 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b-metrics-certs") pod "network-metrics-daemon-vrrvq" (UID: "e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:36:34.302483 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.302414    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e14c5919-af21-4e8f-9783-a2ee2b4885c0-original-pull-secret\") pod \"global-pull-secret-syncer-v7c4f\" (UID: \"e14c5919-af21-4e8f-9783-a2ee2b4885c0\") " pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:36:34.302594 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:34.302528    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:36:34.302594 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:34.302575    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e14c5919-af21-4e8f-9783-a2ee2b4885c0-original-pull-secret podName:e14c5919-af21-4e8f-9783-a2ee2b4885c0 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:06.302561477 +0000 UTC m=+66.267629650 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e14c5919-af21-4e8f-9783-a2ee2b4885c0-original-pull-secret") pod "global-pull-secret-syncer-v7c4f" (UID: "e14c5919-af21-4e8f-9783-a2ee2b4885c0") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:36:34.314778 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.314759    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-185.ec2.internal" event="NodeReady"
Sep 10 20:36:34.314876 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.314846    2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 20:36:34.352791 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.352767    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-67bfdbc8bd-nprqr"]
Sep 10 20:36:34.378441 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.378421    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-f6hc2"]
Sep 10 20:36:34.378560 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.378543    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:36:34.381162 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.381140    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 20:36:34.381279 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.381140    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 20:36:34.381456 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.381443    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-drws7\""
Sep 10 20:36:34.381500 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.381456    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 20:36:34.387096 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.387079    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 20:36:34.398343 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.398325    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67bfdbc8bd-nprqr"]
Sep 10 20:36:34.398426 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.398354    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x9fdk"]
Sep 10 20:36:34.398468 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.398428    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-f6hc2"
Sep 10 20:36:34.400449 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.400434    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 20:36:34.400639 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.400626    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 20:36:34.400700 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.400646    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-89pjq\""
Sep 10 20:36:34.402766 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.402742    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvr8v\" (UniqueName: \"kubernetes.io/projected/2e1ea2e3-fc76-40ad-a7a7-111080547273-kube-api-access-hvr8v\") pod \"network-check-target-wl95n\" (UID: \"2e1ea2e3-fc76-40ad-a7a7-111080547273\") " pod="openshift-network-diagnostics/network-check-target-wl95n"
Sep 10 20:36:34.402910 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:34.402887    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:36:34.402910 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:34.402912    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:36:34.403025 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:34.402921    2564 projected.go:196] Error preparing data for projected volume kube-api-access-hvr8v for pod openshift-network-diagnostics/network-check-target-wl95n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:36:34.403025 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:34.402966    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e1ea2e3-fc76-40ad-a7a7-111080547273-kube-api-access-hvr8v podName:2e1ea2e3-fc76-40ad-a7a7-111080547273 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:06.402953142 +0000 UTC m=+66.368021315 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvr8v" (UniqueName: "kubernetes.io/projected/2e1ea2e3-fc76-40ad-a7a7-111080547273-kube-api-access-hvr8v") pod "network-check-target-wl95n" (UID: "2e1ea2e3-fc76-40ad-a7a7-111080547273") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:36:34.424160 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.424140    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-f6hc2"]
Sep 10 20:36:34.424253 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.424186    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x9fdk"]
Sep 10 20:36:34.424300 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.424250    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x9fdk"
Sep 10 20:36:34.426370 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.426351    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2vtqg\""
Sep 10 20:36:34.426486 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.426376    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 20:36:34.426486 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.426442    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 20:36:34.426486 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.426482    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 20:36:34.504041 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.504018    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d345a7a-fae7-4330-baf7-8be851843ba9-tmp-dir\") pod \"dns-default-f6hc2\" (UID: \"7d345a7a-fae7-4330-baf7-8be851843ba9\") " pod="openshift-dns/dns-default-f6hc2"
Sep 10 20:36:34.504134 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.504046    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1d5ba49b-5e81-426b-b8b0-ba260f754c14-registry-certificates\") pod \"image-registry-67bfdbc8bd-nprqr\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") " pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:36:34.504134 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.504064    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1d5ba49b-5e81-426b-b8b0-ba260f754c14-installation-pull-secrets\") pod \"image-registry-67bfdbc8bd-nprqr\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") " pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:36:34.504134 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.504095    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2j5x\" (UniqueName: \"kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-kube-api-access-j2j5x\") pod \"image-registry-67bfdbc8bd-nprqr\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") " pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:36:34.504134 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.504108    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88836648-21c4-45c8-9cd9-1e30c6fb4529-cert\") pod \"ingress-canary-x9fdk\" (UID: \"88836648-21c4-45c8-9cd9-1e30c6fb4529\") " pod="openshift-ingress-canary/ingress-canary-x9fdk"
Sep 10 20:36:34.504297 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.504182    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-registry-tls\") pod \"image-registry-67bfdbc8bd-nprqr\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") " pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:36:34.504297 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.504213    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-bound-sa-token\") pod \"image-registry-67bfdbc8bd-nprqr\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") " pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:36:34.504297 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.504233    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2pfh\" (UniqueName: \"kubernetes.io/projected/88836648-21c4-45c8-9cd9-1e30c6fb4529-kube-api-access-x2pfh\") pod \"ingress-canary-x9fdk\" (UID: \"88836648-21c4-45c8-9cd9-1e30c6fb4529\") " pod="openshift-ingress-canary/ingress-canary-x9fdk"
Sep 10 20:36:34.504297 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.504268    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1d5ba49b-5e81-426b-b8b0-ba260f754c14-ca-trust-extracted\") pod \"image-registry-67bfdbc8bd-nprqr\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") " pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:36:34.504297 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.504286    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d5ba49b-5e81-426b-b8b0-ba260f754c14-trusted-ca\") pod \"image-registry-67bfdbc8bd-nprqr\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") " pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:36:34.504436 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.504302    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r62p4\" (UniqueName: \"kubernetes.io/projected/7d345a7a-fae7-4330-baf7-8be851843ba9-kube-api-access-r62p4\") pod \"dns-default-f6hc2\" (UID: \"7d345a7a-fae7-4330-baf7-8be851843ba9\") " pod="openshift-dns/dns-default-f6hc2"
Sep 10 20:36:34.504436 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.504335    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1d5ba49b-5e81-426b-b8b0-ba260f754c14-image-registry-private-configuration\") pod \"image-registry-67bfdbc8bd-nprqr\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") " pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:36:34.504436 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.504354    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d345a7a-fae7-4330-baf7-8be851843ba9-metrics-tls\") pod \"dns-default-f6hc2\" (UID: \"7d345a7a-fae7-4330-baf7-8be851843ba9\") " pod="openshift-dns/dns-default-f6hc2"
Sep 10 20:36:34.504436 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.504392    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7d345a7a-fae7-4330-baf7-8be851843ba9-config-volume\") pod \"dns-default-f6hc2\" (UID: \"7d345a7a-fae7-4330-baf7-8be851843ba9\") " pod="openshift-dns/dns-default-f6hc2"
Sep 10 20:36:34.605342 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.605313    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2pfh\" (UniqueName: \"kubernetes.io/projected/88836648-21c4-45c8-9cd9-1e30c6fb4529-kube-api-access-x2pfh\") pod \"ingress-canary-x9fdk\" (UID: \"88836648-21c4-45c8-9cd9-1e30c6fb4529\") " pod="openshift-ingress-canary/ingress-canary-x9fdk"
Sep 10 20:36:34.605459 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.605354    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1d5ba49b-5e81-426b-b8b0-ba260f754c14-ca-trust-extracted\") pod \"image-registry-67bfdbc8bd-nprqr\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") " pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:36:34.605459 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.605380    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d5ba49b-5e81-426b-b8b0-ba260f754c14-trusted-ca\") pod \"image-registry-67bfdbc8bd-nprqr\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") " pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:36:34.605459 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.605408    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r62p4\" (UniqueName: \"kubernetes.io/projected/7d345a7a-fae7-4330-baf7-8be851843ba9-kube-api-access-r62p4\") pod \"dns-default-f6hc2\" (UID: \"7d345a7a-fae7-4330-baf7-8be851843ba9\") " pod="openshift-dns/dns-default-f6hc2"
Sep 10 20:36:34.605459 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.605440    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1d5ba49b-5e81-426b-b8b0-ba260f754c14-image-registry-private-configuration\") pod \"image-registry-67bfdbc8bd-nprqr\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") " pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:36:34.605638 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.605463    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d345a7a-fae7-4330-baf7-8be851843ba9-metrics-tls\") pod \"dns-default-f6hc2\" (UID: \"7d345a7a-fae7-4330-baf7-8be851843ba9\") " pod="openshift-dns/dns-default-f6hc2"
Sep 10 20:36:34.605638 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.605485    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7d345a7a-fae7-4330-baf7-8be851843ba9-config-volume\") pod \"dns-default-f6hc2\" (UID: \"7d345a7a-fae7-4330-baf7-8be851843ba9\") " pod="openshift-dns/dns-default-f6hc2"
Sep 10 20:36:34.605638 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.605532    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d345a7a-fae7-4330-baf7-8be851843ba9-tmp-dir\") pod \"dns-default-f6hc2\" (UID: \"7d345a7a-fae7-4330-baf7-8be851843ba9\") " pod="openshift-dns/dns-default-f6hc2"
Sep 10 20:36:34.605638 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.605561    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1d5ba49b-5e81-426b-b8b0-ba260f754c14-registry-certificates\") pod \"image-registry-67bfdbc8bd-nprqr\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") " pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:36:34.605638 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.605582    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1d5ba49b-5e81-426b-b8b0-ba260f754c14-installation-pull-secrets\") pod \"image-registry-67bfdbc8bd-nprqr\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") " pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:36:34.605638 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.605632    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2j5x\" (UniqueName: \"kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-kube-api-access-j2j5x\") pod \"image-registry-67bfdbc8bd-nprqr\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") " pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:36:34.605909 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.605656    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88836648-21c4-45c8-9cd9-1e30c6fb4529-cert\") pod \"ingress-canary-x9fdk\" (UID: \"88836648-21c4-45c8-9cd9-1e30c6fb4529\") " pod="openshift-ingress-canary/ingress-canary-x9fdk"
Sep 10 20:36:34.605909 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.605796    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1d5ba49b-5e81-426b-b8b0-ba260f754c14-ca-trust-extracted\") pod \"image-registry-67bfdbc8bd-nprqr\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") " pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:36:34.605909 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:34.605900    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:36:34.606042 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.605940    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d345a7a-fae7-4330-baf7-8be851843ba9-tmp-dir\") pod \"dns-default-f6hc2\" (UID: \"7d345a7a-fae7-4330-baf7-8be851843ba9\") " pod="openshift-dns/dns-default-f6hc2"
Sep 10 20:36:34.606042 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:34.605973    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d345a7a-fae7-4330-baf7-8be851843ba9-metrics-tls podName:7d345a7a-fae7-4330-baf7-8be851843ba9 nodeName:}" failed. No retries permitted until 2026-09-10 20:36:35.105951524 +0000 UTC m=+35.071019702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7d345a7a-fae7-4330-baf7-8be851843ba9-metrics-tls") pod "dns-default-f6hc2" (UID: "7d345a7a-fae7-4330-baf7-8be851843ba9") : secret "dns-default-metrics-tls" not found
Sep 10 20:36:34.606042 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.606023    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-registry-tls\") pod \"image-registry-67bfdbc8bd-nprqr\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") " pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:36:34.606215 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.606052    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-bound-sa-token\") pod \"image-registry-67bfdbc8bd-nprqr\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") " pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:36:34.606215 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.606100    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7d345a7a-fae7-4330-baf7-8be851843ba9-config-volume\") pod \"dns-default-f6hc2\" (UID: \"7d345a7a-fae7-4330-baf7-8be851843ba9\") " pod="openshift-dns/dns-default-f6hc2"
Sep 10 20:36:34.606319 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:34.606214    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:36:34.606319 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:34.606228    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67bfdbc8bd-nprqr: secret "image-registry-tls" not found
Sep 10 20:36:34.606421 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:34.606330    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-registry-tls podName:1d5ba49b-5e81-426b-b8b0-ba260f754c14 nodeName:}" failed. No retries permitted until 2026-09-10 20:36:35.106314231 +0000 UTC m=+35.071382405 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-registry-tls") pod "image-registry-67bfdbc8bd-nprqr" (UID: "1d5ba49b-5e81-426b-b8b0-ba260f754c14") : secret "image-registry-tls" not found
Sep 10 20:36:34.606681 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.606656    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1d5ba49b-5e81-426b-b8b0-ba260f754c14-registry-certificates\") pod \"image-registry-67bfdbc8bd-nprqr\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") " pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:36:34.606891 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.606867    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d5ba49b-5e81-426b-b8b0-ba260f754c14-trusted-ca\") pod \"image-registry-67bfdbc8bd-nprqr\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") " pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:36:34.609990 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.609972    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88836648-21c4-45c8-9cd9-1e30c6fb4529-cert\") pod \"ingress-canary-x9fdk\" (UID: \"88836648-21c4-45c8-9cd9-1e30c6fb4529\") " pod="openshift-ingress-canary/ingress-canary-x9fdk"
Sep 10 20:36:34.609990 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.609982    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1d5ba49b-5e81-426b-b8b0-ba260f754c14-installation-pull-secrets\") pod \"image-registry-67bfdbc8bd-nprqr\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") " pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:36:34.610147 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.609971    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1d5ba49b-5e81-426b-b8b0-ba260f754c14-image-registry-private-configuration\") pod \"image-registry-67bfdbc8bd-nprqr\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") " pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:36:34.616340 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.616315    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r62p4\" (UniqueName: \"kubernetes.io/projected/7d345a7a-fae7-4330-baf7-8be851843ba9-kube-api-access-r62p4\") pod \"dns-default-f6hc2\" (UID: \"7d345a7a-fae7-4330-baf7-8be851843ba9\") " pod="openshift-dns/dns-default-f6hc2"
Sep 10 20:36:34.617027 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.616895    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-bound-sa-token\") pod \"image-registry-67bfdbc8bd-nprqr\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") " pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:36:34.617785 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.617762    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2pfh\" (UniqueName: \"kubernetes.io/projected/88836648-21c4-45c8-9cd9-1e30c6fb4529-kube-api-access-x2pfh\") pod \"ingress-canary-x9fdk\" (UID: \"88836648-21c4-45c8-9cd9-1e30c6fb4529\") " pod="openshift-ingress-canary/ingress-canary-x9fdk"
Sep 10 20:36:34.618146 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.618126    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2j5x\" (UniqueName: \"kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-kube-api-access-j2j5x\") pod \"image-registry-67bfdbc8bd-nprqr\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") " pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:36:34.732911 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:34.732888    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x9fdk"
Sep 10 20:36:35.052552 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:35.052487    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x9fdk"]
Sep 10 20:36:35.055998 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:36:35.055970    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88836648_21c4_45c8_9cd9_1e30c6fb4529.slice/crio-614a9274b26944e9023d7d95f32b074bbdc78f26e95fd913023198b740a82d71 WatchSource:0}: Error finding container 614a9274b26944e9023d7d95f32b074bbdc78f26e95fd913023198b740a82d71: Status 404 returned error can't find the container with id 614a9274b26944e9023d7d95f32b074bbdc78f26e95fd913023198b740a82d71
Sep 10 20:36:35.109457 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:35.109434    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d345a7a-fae7-4330-baf7-8be851843ba9-metrics-tls\") pod \"dns-default-f6hc2\" (UID: \"7d345a7a-fae7-4330-baf7-8be851843ba9\") " pod="openshift-dns/dns-default-f6hc2"
Sep 10 20:36:35.109574 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:35.109532    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-registry-tls\") pod \"image-registry-67bfdbc8bd-nprqr\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") " pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:36:35.109680 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:35.109590    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:36:35.109680 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:35.109612    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:36:35.109680 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:35.109623    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67bfdbc8bd-nprqr: secret "image-registry-tls" not found
Sep 10 20:36:35.109680 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:35.109648    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d345a7a-fae7-4330-baf7-8be851843ba9-metrics-tls podName:7d345a7a-fae7-4330-baf7-8be851843ba9 nodeName:}" failed. No retries permitted until 2026-09-10 20:36:36.109629671 +0000 UTC m=+36.074697862 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7d345a7a-fae7-4330-baf7-8be851843ba9-metrics-tls") pod "dns-default-f6hc2" (UID: "7d345a7a-fae7-4330-baf7-8be851843ba9") : secret "dns-default-metrics-tls" not found
Sep 10 20:36:35.109680 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:35.109664    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-registry-tls podName:1d5ba49b-5e81-426b-b8b0-ba260f754c14 nodeName:}" failed. No retries permitted until 2026-09-10 20:36:36.109656213 +0000 UTC m=+36.074724392 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-registry-tls") pod "image-registry-67bfdbc8bd-nprqr" (UID: "1d5ba49b-5e81-426b-b8b0-ba260f754c14") : secret "image-registry-tls" not found
Sep 10 20:36:35.535155 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:35.535127    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrrvq"
Sep 10 20:36:35.536057 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:35.535163    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl95n"
Sep 10 20:36:35.536057 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:35.535163    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:36:35.538666 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:35.538428    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 20:36:35.538666 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:35.538460    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 20:36:35.538666 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:35.538481    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ks6z6\""
Sep 10 20:36:35.538666 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:35.538494    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d24xx\""
Sep 10 20:36:35.538666 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:35.538460    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 20:36:35.538666 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:35.538432    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 20:36:35.823452 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:35.823422    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x9fdk" event={"ID":"88836648-21c4-45c8-9cd9-1e30c6fb4529","Type":"ContainerStarted","Data":"614a9274b26944e9023d7d95f32b074bbdc78f26e95fd913023198b740a82d71"}
Sep 10 20:36:36.115972 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:36.115899    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-registry-tls\") pod \"image-registry-67bfdbc8bd-nprqr\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") " pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:36:36.115972 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:36.115961    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d345a7a-fae7-4330-baf7-8be851843ba9-metrics-tls\") pod \"dns-default-f6hc2\" (UID: \"7d345a7a-fae7-4330-baf7-8be851843ba9\") " pod="openshift-dns/dns-default-f6hc2"
Sep 10 20:36:36.116187 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:36.116076    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:36:36.116187 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:36.116084    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:36:36.116187 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:36.116104    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67bfdbc8bd-nprqr: secret "image-registry-tls" not found
Sep 10 20:36:36.116187 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:36.116139    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d345a7a-fae7-4330-baf7-8be851843ba9-metrics-tls podName:7d345a7a-fae7-4330-baf7-8be851843ba9 nodeName:}" failed. No retries permitted until 2026-09-10 20:36:38.11612093 +0000 UTC m=+38.081189109 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7d345a7a-fae7-4330-baf7-8be851843ba9-metrics-tls") pod "dns-default-f6hc2" (UID: "7d345a7a-fae7-4330-baf7-8be851843ba9") : secret "dns-default-metrics-tls" not found
Sep 10 20:36:36.116187 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:36.116156    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-registry-tls podName:1d5ba49b-5e81-426b-b8b0-ba260f754c14 nodeName:}" failed. No retries permitted until 2026-09-10 20:36:38.11614732 +0000 UTC m=+38.081215498 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-registry-tls") pod "image-registry-67bfdbc8bd-nprqr" (UID: "1d5ba49b-5e81-426b-b8b0-ba260f754c14") : secret "image-registry-tls" not found
Sep 10 20:36:38.130392 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:38.130362    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-registry-tls\") pod \"image-registry-67bfdbc8bd-nprqr\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") " pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:36:38.130836 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:38.130411    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d345a7a-fae7-4330-baf7-8be851843ba9-metrics-tls\") pod \"dns-default-f6hc2\" (UID: \"7d345a7a-fae7-4330-baf7-8be851843ba9\") " pod="openshift-dns/dns-default-f6hc2"
Sep 10 20:36:38.130836 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:38.130531    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:36:38.130836 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:38.130549    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67bfdbc8bd-nprqr: secret "image-registry-tls" not found
Sep 10 20:36:38.130836 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:38.130593    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:36:38.130836 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:38.130601    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-registry-tls podName:1d5ba49b-5e81-426b-b8b0-ba260f754c14 nodeName:}" failed. No retries permitted until 2026-09-10 20:36:42.130587197 +0000 UTC m=+42.095655392 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-registry-tls") pod "image-registry-67bfdbc8bd-nprqr" (UID: "1d5ba49b-5e81-426b-b8b0-ba260f754c14") : secret "image-registry-tls" not found
Sep 10 20:36:38.130836 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:38.130640    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d345a7a-fae7-4330-baf7-8be851843ba9-metrics-tls podName:7d345a7a-fae7-4330-baf7-8be851843ba9 nodeName:}" failed. No retries permitted until 2026-09-10 20:36:42.130624765 +0000 UTC m=+42.095692961 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7d345a7a-fae7-4330-baf7-8be851843ba9-metrics-tls") pod "dns-default-f6hc2" (UID: "7d345a7a-fae7-4330-baf7-8be851843ba9") : secret "dns-default-metrics-tls" not found
Sep 10 20:36:39.835018 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:39.834984    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x9fdk" event={"ID":"88836648-21c4-45c8-9cd9-1e30c6fb4529","Type":"ContainerStarted","Data":"20ebfa2f2a69753fae04ad5463c85ac4a3743e40cc1a4bbeafc6ac4a6bce60e1"}
Sep 10 20:36:39.838272 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:39.838235    2564 generic.go:356] "Generic (PLEG): container finished" podID="b1b63c96-edab-42b0-8af3-f9db7d20867a" containerID="e85f921923dc442cfc8f8de6ae0d63d8cb909e0c2c43c2914e09575e6d725380" exitCode=0
Sep 10 20:36:39.838386 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:39.838288    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hqsvv" event={"ID":"b1b63c96-edab-42b0-8af3-f9db7d20867a","Type":"ContainerDied","Data":"e85f921923dc442cfc8f8de6ae0d63d8cb909e0c2c43c2914e09575e6d725380"}
Sep 10 20:36:39.850497 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:39.850457    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x9fdk" podStartSLOduration=1.395308976 podStartE2EDuration="5.850444537s" totalImagesPullingTime="4.455135561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:36:39.849622916 +0000 UTC m=+39.814691111" watchObservedRunningTime="2026-09-10 20:36:39.850444537 +0000 UTC m=+39.815512731"
Sep 10 20:36:40.846081 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:40.846012    2564 generic.go:356] "Generic (PLEG): container finished" podID="b1b63c96-edab-42b0-8af3-f9db7d20867a" containerID="4abb1597a4b01ae4b419b5d62661d94c049181a10a10c51fb8344ec44d8eadc5" exitCode=0
Sep 10 20:36:40.846552 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:40.846108    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hqsvv" event={"ID":"b1b63c96-edab-42b0-8af3-f9db7d20867a","Type":"ContainerDied","Data":"4abb1597a4b01ae4b419b5d62661d94c049181a10a10c51fb8344ec44d8eadc5"}
Sep 10 20:36:41.852613 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:41.852575    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hqsvv" event={"ID":"b1b63c96-edab-42b0-8af3-f9db7d20867a","Type":"ContainerStarted","Data":"eccc82e1e7d481401f2702a3e2521c7f96d918259838fc5a169fac4cb9ebcfdd"}
Sep 10 20:36:41.879341 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:41.879293    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hqsvv" podStartSLOduration=8.513414504 podStartE2EDuration="40.879278434s" totalImagesPullingTime="32.36586393s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:36:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:36:41.878132389 +0000 UTC m=+41.843200586" watchObservedRunningTime="2026-09-10 20:36:41.879278434 +0000 UTC m=+41.844346629"
Sep 10 20:36:42.157881 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:42.157812    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-registry-tls\") pod \"image-registry-67bfdbc8bd-nprqr\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") " pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:36:42.157881 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:42.157859    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d345a7a-fae7-4330-baf7-8be851843ba9-metrics-tls\") pod \"dns-default-f6hc2\" (UID: \"7d345a7a-fae7-4330-baf7-8be851843ba9\") " pod="openshift-dns/dns-default-f6hc2"
Sep 10 20:36:42.158074 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:42.157944    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:36:42.158074 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:42.157969    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:36:42.158074 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:42.157991    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67bfdbc8bd-nprqr: secret "image-registry-tls" not found
Sep 10 20:36:42.158074 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:42.158006    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d345a7a-fae7-4330-baf7-8be851843ba9-metrics-tls podName:7d345a7a-fae7-4330-baf7-8be851843ba9 nodeName:}" failed. No retries permitted until 2026-09-10 20:36:50.157992768 +0000 UTC m=+50.123060941 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7d345a7a-fae7-4330-baf7-8be851843ba9-metrics-tls") pod "dns-default-f6hc2" (UID: "7d345a7a-fae7-4330-baf7-8be851843ba9") : secret "dns-default-metrics-tls" not found
Sep 10 20:36:42.158074 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:42.158062    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-registry-tls podName:1d5ba49b-5e81-426b-b8b0-ba260f754c14 nodeName:}" failed. No retries permitted until 2026-09-10 20:36:50.158041756 +0000 UTC m=+50.123109944 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-registry-tls") pod "image-registry-67bfdbc8bd-nprqr" (UID: "1d5ba49b-5e81-426b-b8b0-ba260f754c14") : secret "image-registry-tls" not found
Sep 10 20:36:50.214016 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:50.213971    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d345a7a-fae7-4330-baf7-8be851843ba9-metrics-tls\") pod \"dns-default-f6hc2\" (UID: \"7d345a7a-fae7-4330-baf7-8be851843ba9\") " pod="openshift-dns/dns-default-f6hc2"
Sep 10 20:36:50.214471 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:36:50.214044    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-registry-tls\") pod \"image-registry-67bfdbc8bd-nprqr\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") " pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:36:50.214471 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:50.214132    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:36:50.214471 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:50.214137    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:36:50.214471 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:50.214233    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d345a7a-fae7-4330-baf7-8be851843ba9-metrics-tls podName:7d345a7a-fae7-4330-baf7-8be851843ba9 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:06.214215953 +0000 UTC m=+66.179284131 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7d345a7a-fae7-4330-baf7-8be851843ba9-metrics-tls") pod "dns-default-f6hc2" (UID: "7d345a7a-fae7-4330-baf7-8be851843ba9") : secret "dns-default-metrics-tls" not found
Sep 10 20:36:50.214471 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:50.214238    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67bfdbc8bd-nprqr: secret "image-registry-tls" not found
Sep 10 20:36:50.214471 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:36:50.214295    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-registry-tls podName:1d5ba49b-5e81-426b-b8b0-ba260f754c14 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:06.214282282 +0000 UTC m=+66.179350455 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-registry-tls") pod "image-registry-67bfdbc8bd-nprqr" (UID: "1d5ba49b-5e81-426b-b8b0-ba260f754c14") : secret "image-registry-tls" not found
Sep 10 20:37:01.823320 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:01.823290    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kn6fl"
Sep 10 20:37:04.651096 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:04.651056    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ttfr9"]
Sep 10 20:37:04.654141 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:04.654120    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ttfr9"
Sep 10 20:37:04.656743 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:04.656719    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 10 20:37:04.656867 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:04.656846    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 10 20:37:04.657573 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:04.657548    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 10 20:37:04.657805 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:04.657776    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nbx49\""
Sep 10 20:37:04.666888 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:04.666862    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ttfr9"]
Sep 10 20:37:04.708080 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:04.708056    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3173cc02-0594-4f6a-b991-297572a874d2-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-ttfr9\" (UID: \"3173cc02-0594-4f6a-b991-297572a874d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ttfr9"
Sep 10 20:37:04.708222 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:04.708123    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpbfz\" (UniqueName: \"kubernetes.io/projected/3173cc02-0594-4f6a-b991-297572a874d2-kube-api-access-hpbfz\") pod \"cluster-samples-operator-86499c89c4-ttfr9\" (UID: \"3173cc02-0594-4f6a-b991-297572a874d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ttfr9"
Sep 10 20:37:04.809370 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:04.809348    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpbfz\" (UniqueName: \"kubernetes.io/projected/3173cc02-0594-4f6a-b991-297572a874d2-kube-api-access-hpbfz\") pod \"cluster-samples-operator-86499c89c4-ttfr9\" (UID: \"3173cc02-0594-4f6a-b991-297572a874d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ttfr9"
Sep 10 20:37:04.809483 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:04.809401    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3173cc02-0594-4f6a-b991-297572a874d2-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-ttfr9\" (UID: \"3173cc02-0594-4f6a-b991-297572a874d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ttfr9"
Sep 10 20:37:04.809547 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:04.809493    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 20:37:04.809547 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:04.809543    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3173cc02-0594-4f6a-b991-297572a874d2-samples-operator-tls podName:3173cc02-0594-4f6a-b991-297572a874d2 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:05.309530078 +0000 UTC m=+65.274598251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3173cc02-0594-4f6a-b991-297572a874d2-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-ttfr9" (UID: "3173cc02-0594-4f6a-b991-297572a874d2") : secret "samples-operator-tls" not found
Sep 10 20:37:04.821069 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:04.821038    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpbfz\" (UniqueName: \"kubernetes.io/projected/3173cc02-0594-4f6a-b991-297572a874d2-kube-api-access-hpbfz\") pod \"cluster-samples-operator-86499c89c4-ttfr9\" (UID: \"3173cc02-0594-4f6a-b991-297572a874d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ttfr9"
Sep 10 20:37:05.313058 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:05.313027    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3173cc02-0594-4f6a-b991-297572a874d2-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-ttfr9\" (UID: \"3173cc02-0594-4f6a-b991-297572a874d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ttfr9"
Sep 10 20:37:05.313202 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:05.313156    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 20:37:05.313255 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:05.313242    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3173cc02-0594-4f6a-b991-297572a874d2-samples-operator-tls podName:3173cc02-0594-4f6a-b991-297572a874d2 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:06.313224801 +0000 UTC m=+66.278292975 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3173cc02-0594-4f6a-b991-297572a874d2-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-ttfr9" (UID: "3173cc02-0594-4f6a-b991-297572a874d2") : secret "samples-operator-tls" not found
Sep 10 20:37:06.219068 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:06.219032    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d345a7a-fae7-4330-baf7-8be851843ba9-metrics-tls\") pod \"dns-default-f6hc2\" (UID: \"7d345a7a-fae7-4330-baf7-8be851843ba9\") " pod="openshift-dns/dns-default-f6hc2"
Sep 10 20:37:06.219533 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:06.219106    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b-metrics-certs\") pod \"network-metrics-daemon-vrrvq\" (UID: \"e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b\") " pod="openshift-multus/network-metrics-daemon-vrrvq"
Sep 10 20:37:06.219533 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:06.219152    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-registry-tls\") pod \"image-registry-67bfdbc8bd-nprqr\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") " pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:37:06.219533 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:06.219206    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:37:06.219533 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:06.219279    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d345a7a-fae7-4330-baf7-8be851843ba9-metrics-tls podName:7d345a7a-fae7-4330-baf7-8be851843ba9 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:38.219260727 +0000 UTC m=+98.184328955 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7d345a7a-fae7-4330-baf7-8be851843ba9-metrics-tls") pod "dns-default-f6hc2" (UID: "7d345a7a-fae7-4330-baf7-8be851843ba9") : secret "dns-default-metrics-tls" not found
Sep 10 20:37:06.219533 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:06.219302    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:37:06.219533 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:06.219320    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67bfdbc8bd-nprqr: secret "image-registry-tls" not found
Sep 10 20:37:06.219533 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:06.219401    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-registry-tls podName:1d5ba49b-5e81-426b-b8b0-ba260f754c14 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:38.219382342 +0000 UTC m=+98.184450584 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-registry-tls") pod "image-registry-67bfdbc8bd-nprqr" (UID: "1d5ba49b-5e81-426b-b8b0-ba260f754c14") : secret "image-registry-tls" not found
Sep 10 20:37:06.221772 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:06.221748    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 20:37:06.230074 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:06.230050    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 10 20:37:06.230137 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:06.230113    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b-metrics-certs podName:e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b nodeName:}" failed. No retries permitted until 2026-09-10 20:38:10.230094041 +0000 UTC m=+130.195162263 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b-metrics-certs") pod "network-metrics-daemon-vrrvq" (UID: "e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b") : secret "metrics-daemon-secret" not found
Sep 10 20:37:06.319895 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:06.319867    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e14c5919-af21-4e8f-9783-a2ee2b4885c0-original-pull-secret\") pod \"global-pull-secret-syncer-v7c4f\" (UID: \"e14c5919-af21-4e8f-9783-a2ee2b4885c0\") " pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:37:06.320000 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:06.319925    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3173cc02-0594-4f6a-b991-297572a874d2-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-ttfr9\" (UID: \"3173cc02-0594-4f6a-b991-297572a874d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ttfr9"
Sep 10 20:37:06.320048 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:06.320020    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 20:37:06.320090 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:06.320067    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3173cc02-0594-4f6a-b991-297572a874d2-samples-operator-tls podName:3173cc02-0594-4f6a-b991-297572a874d2 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:08.320055967 +0000 UTC m=+68.285124141 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3173cc02-0594-4f6a-b991-297572a874d2-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-ttfr9" (UID: "3173cc02-0594-4f6a-b991-297572a874d2") : secret "samples-operator-tls" not found
Sep 10 20:37:06.322504 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:06.322486    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 20:37:06.332522 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:06.332494    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e14c5919-af21-4e8f-9783-a2ee2b4885c0-original-pull-secret\") pod \"global-pull-secret-syncer-v7c4f\" (UID: \"e14c5919-af21-4e8f-9783-a2ee2b4885c0\") " pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:37:06.420993 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:06.420956    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvr8v\" (UniqueName: \"kubernetes.io/projected/2e1ea2e3-fc76-40ad-a7a7-111080547273-kube-api-access-hvr8v\") pod \"network-check-target-wl95n\" (UID: \"2e1ea2e3-fc76-40ad-a7a7-111080547273\") " pod="openshift-network-diagnostics/network-check-target-wl95n"
Sep 10 20:37:06.423307 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:06.423287    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 20:37:06.434504 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:06.434482    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 20:37:06.444694 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:06.444664    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvr8v\" (UniqueName: \"kubernetes.io/projected/2e1ea2e3-fc76-40ad-a7a7-111080547273-kube-api-access-hvr8v\") pod \"network-check-target-wl95n\" (UID: \"2e1ea2e3-fc76-40ad-a7a7-111080547273\") " pod="openshift-network-diagnostics/network-check-target-wl95n"
Sep 10 20:37:06.457115 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:06.457090    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d24xx\""
Sep 10 20:37:06.459084 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:06.459065    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7c4f"
Sep 10 20:37:06.465664 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:06.465634    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl95n"
Sep 10 20:37:06.736067 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:06.735997    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v7c4f"]
Sep 10 20:37:06.740850 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:06.740820    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wl95n"]
Sep 10 20:37:06.740850 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:37:06.740838    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode14c5919_af21_4e8f_9783_a2ee2b4885c0.slice/crio-7e86a02e7f4d272ef1eb91785ff0798fa3d1123fc8d456573c07de4ce97e969b WatchSource:0}: Error finding container 7e86a02e7f4d272ef1eb91785ff0798fa3d1123fc8d456573c07de4ce97e969b: Status 404 returned error can't find the container with id 7e86a02e7f4d272ef1eb91785ff0798fa3d1123fc8d456573c07de4ce97e969b
Sep 10 20:37:06.745130 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:37:06.745100    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e1ea2e3_fc76_40ad_a7a7_111080547273.slice/crio-62b85ba74595de5be6f1d0f73935cd65aa2bb5b95bcdb3f92576ec50ed82d7b0 WatchSource:0}: Error finding container 62b85ba74595de5be6f1d0f73935cd65aa2bb5b95bcdb3f92576ec50ed82d7b0: Status 404 returned error can't find the container with id 62b85ba74595de5be6f1d0f73935cd65aa2bb5b95bcdb3f92576ec50ed82d7b0
Sep 10 20:37:06.911709 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:06.911676    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wl95n" event={"ID":"2e1ea2e3-fc76-40ad-a7a7-111080547273","Type":"ContainerStarted","Data":"62b85ba74595de5be6f1d0f73935cd65aa2bb5b95bcdb3f92576ec50ed82d7b0"}
Sep 10 20:37:06.912784 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:06.912762    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v7c4f" event={"ID":"e14c5919-af21-4e8f-9783-a2ee2b4885c0","Type":"ContainerStarted","Data":"7e86a02e7f4d272ef1eb91785ff0798fa3d1123fc8d456573c07de4ce97e969b"}
Sep 10 20:37:07.452634 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:07.452564    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-nn88n"]
Sep 10 20:37:07.456000 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:07.455973    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-nn88n"
Sep 10 20:37:07.459329 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:07.459303    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-dbbgd\""
Sep 10 20:37:07.476599 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:07.476550    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-nn88n"]
Sep 10 20:37:07.529109 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:07.529081    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxskk\" (UniqueName: \"kubernetes.io/projected/e7dcdc2d-00a7-4573-a1a0-20197bdc1ed9-kube-api-access-pxskk\") pod \"network-check-source-6f8b768455-nn88n\" (UID: \"e7dcdc2d-00a7-4573-a1a0-20197bdc1ed9\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-nn88n"
Sep 10 20:37:07.629513 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:07.629485    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxskk\" (UniqueName: \"kubernetes.io/projected/e7dcdc2d-00a7-4573-a1a0-20197bdc1ed9-kube-api-access-pxskk\") pod \"network-check-source-6f8b768455-nn88n\" (UID: \"e7dcdc2d-00a7-4573-a1a0-20197bdc1ed9\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-nn88n"
Sep 10 20:37:07.643792 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:07.643767    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxskk\" (UniqueName: \"kubernetes.io/projected/e7dcdc2d-00a7-4573-a1a0-20197bdc1ed9-kube-api-access-pxskk\") pod \"network-check-source-6f8b768455-nn88n\" (UID: \"e7dcdc2d-00a7-4573-a1a0-20197bdc1ed9\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-nn88n"
Sep 10 20:37:07.768404 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:07.768343    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-nn88n"
Sep 10 20:37:08.069573 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:08.069539    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-nn88n"]
Sep 10 20:37:08.073644 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:37:08.073616    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7dcdc2d_00a7_4573_a1a0_20197bdc1ed9.slice/crio-0e52df5bc9316f47d77977143d203bcfd32b92a7d33cf2ea35bd3910e11bbf02 WatchSource:0}: Error finding container 0e52df5bc9316f47d77977143d203bcfd32b92a7d33cf2ea35bd3910e11bbf02: Status 404 returned error can't find the container with id 0e52df5bc9316f47d77977143d203bcfd32b92a7d33cf2ea35bd3910e11bbf02
Sep 10 20:37:08.333974 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:08.333904    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3173cc02-0594-4f6a-b991-297572a874d2-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-ttfr9\" (UID: \"3173cc02-0594-4f6a-b991-297572a874d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ttfr9"
Sep 10 20:37:08.334096 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:08.334059    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 20:37:08.334134 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:08.334129    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3173cc02-0594-4f6a-b991-297572a874d2-samples-operator-tls podName:3173cc02-0594-4f6a-b991-297572a874d2 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:12.334114187 +0000 UTC m=+72.299182360 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3173cc02-0594-4f6a-b991-297572a874d2-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-ttfr9" (UID: "3173cc02-0594-4f6a-b991-297572a874d2") : secret "samples-operator-tls" not found
Sep 10 20:37:08.919232 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:08.919197    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-nn88n" event={"ID":"e7dcdc2d-00a7-4573-a1a0-20197bdc1ed9","Type":"ContainerStarted","Data":"0e52df5bc9316f47d77977143d203bcfd32b92a7d33cf2ea35bd3910e11bbf02"}
Sep 10 20:37:09.742039 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:09.742004    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zvhvz"]
Sep 10 20:37:09.763010 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:09.762402    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zvhvz"
Sep 10 20:37:09.763010 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:09.762491    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zvhvz"]
Sep 10 20:37:09.764456 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:09.764432    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 20:37:09.765141 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:09.764849    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-djk6c\""
Sep 10 20:37:09.765454 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:09.765410    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 20:37:09.765749 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:09.765416    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 20:37:09.765749 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:09.765435    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 20:37:09.844608 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:09.844576    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d024393a-e68c-4001-a13f-9a046bc2a924-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zvhvz\" (UID: \"d024393a-e68c-4001-a13f-9a046bc2a924\") " pod="openshift-insights/insights-runtime-extractor-zvhvz"
Sep 10 20:37:09.844769 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:09.844637    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d024393a-e68c-4001-a13f-9a046bc2a924-crio-socket\") pod \"insights-runtime-extractor-zvhvz\" (UID: \"d024393a-e68c-4001-a13f-9a046bc2a924\") " pod="openshift-insights/insights-runtime-extractor-zvhvz"
Sep 10 20:37:09.844769 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:09.844665    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d024393a-e68c-4001-a13f-9a046bc2a924-data-volume\") pod \"insights-runtime-extractor-zvhvz\" (UID: \"d024393a-e68c-4001-a13f-9a046bc2a924\") " pod="openshift-insights/insights-runtime-extractor-zvhvz"
Sep 10 20:37:09.844769 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:09.844689    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d024393a-e68c-4001-a13f-9a046bc2a924-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zvhvz\" (UID: \"d024393a-e68c-4001-a13f-9a046bc2a924\") " pod="openshift-insights/insights-runtime-extractor-zvhvz"
Sep 10 20:37:09.844769 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:09.844735    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdlwx\" (UniqueName: \"kubernetes.io/projected/d024393a-e68c-4001-a13f-9a046bc2a924-kube-api-access-cdlwx\") pod \"insights-runtime-extractor-zvhvz\" (UID: \"d024393a-e68c-4001-a13f-9a046bc2a924\") " pod="openshift-insights/insights-runtime-extractor-zvhvz"
Sep 10 20:37:09.945773 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:09.945733    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d024393a-e68c-4001-a13f-9a046bc2a924-crio-socket\") pod \"insights-runtime-extractor-zvhvz\" (UID: \"d024393a-e68c-4001-a13f-9a046bc2a924\") " pod="openshift-insights/insights-runtime-extractor-zvhvz"
Sep 10 20:37:09.946203 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:09.945793    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d024393a-e68c-4001-a13f-9a046bc2a924-data-volume\") pod \"insights-runtime-extractor-zvhvz\" (UID: \"d024393a-e68c-4001-a13f-9a046bc2a924\") " pod="openshift-insights/insights-runtime-extractor-zvhvz"
Sep 10 20:37:09.946203 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:09.945814    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d024393a-e68c-4001-a13f-9a046bc2a924-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zvhvz\" (UID: \"d024393a-e68c-4001-a13f-9a046bc2a924\") " pod="openshift-insights/insights-runtime-extractor-zvhvz"
Sep 10 20:37:09.946203 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:09.945817    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d024393a-e68c-4001-a13f-9a046bc2a924-crio-socket\") pod \"insights-runtime-extractor-zvhvz\" (UID: \"d024393a-e68c-4001-a13f-9a046bc2a924\") " pod="openshift-insights/insights-runtime-extractor-zvhvz"
Sep 10 20:37:09.946203 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:09.945879    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdlwx\" (UniqueName: \"kubernetes.io/projected/d024393a-e68c-4001-a13f-9a046bc2a924-kube-api-access-cdlwx\") pod \"insights-runtime-extractor-zvhvz\" (UID: \"d024393a-e68c-4001-a13f-9a046bc2a924\") " pod="openshift-insights/insights-runtime-extractor-zvhvz"
Sep 10 20:37:09.946203 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:09.945907    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:37:09.946203 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:09.945926    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d024393a-e68c-4001-a13f-9a046bc2a924-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zvhvz\" (UID: \"d024393a-e68c-4001-a13f-9a046bc2a924\") " pod="openshift-insights/insights-runtime-extractor-zvhvz"
Sep 10 20:37:09.946203 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:09.945987    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d024393a-e68c-4001-a13f-9a046bc2a924-kube-rbac-proxy-cm podName:d024393a-e68c-4001-a13f-9a046bc2a924 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:10.445966914 +0000 UTC m=+70.411035104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d024393a-e68c-4001-a13f-9a046bc2a924-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zvhvz" (UID: "d024393a-e68c-4001-a13f-9a046bc2a924") : configmap "kube-rbac-proxy" not found
Sep 10 20:37:09.946203 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:09.946030    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:37:09.946203 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:09.946077    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d024393a-e68c-4001-a13f-9a046bc2a924-insights-runtime-extractor-tls podName:d024393a-e68c-4001-a13f-9a046bc2a924 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:10.446061729 +0000 UTC m=+70.411129904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d024393a-e68c-4001-a13f-9a046bc2a924-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zvhvz" (UID: "d024393a-e68c-4001-a13f-9a046bc2a924") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:37:09.946672 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:09.946556    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d024393a-e68c-4001-a13f-9a046bc2a924-data-volume\") pod \"insights-runtime-extractor-zvhvz\" (UID: \"d024393a-e68c-4001-a13f-9a046bc2a924\") " pod="openshift-insights/insights-runtime-extractor-zvhvz"
Sep 10 20:37:09.955439 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:09.955415    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdlwx\" (UniqueName: \"kubernetes.io/projected/d024393a-e68c-4001-a13f-9a046bc2a924-kube-api-access-cdlwx\") pod \"insights-runtime-extractor-zvhvz\" (UID: \"d024393a-e68c-4001-a13f-9a046bc2a924\") " pod="openshift-insights/insights-runtime-extractor-zvhvz"
Sep 10 20:37:10.448327 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:10.448284    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d024393a-e68c-4001-a13f-9a046bc2a924-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zvhvz\" (UID: \"d024393a-e68c-4001-a13f-9a046bc2a924\") " pod="openshift-insights/insights-runtime-extractor-zvhvz"
Sep 10 20:37:10.448528 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:10.448430    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:37:10.448528 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:10.448464    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d024393a-e68c-4001-a13f-9a046bc2a924-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zvhvz\" (UID: \"d024393a-e68c-4001-a13f-9a046bc2a924\") " pod="openshift-insights/insights-runtime-extractor-zvhvz"
Sep 10 20:37:10.448528 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:10.448499    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d024393a-e68c-4001-a13f-9a046bc2a924-kube-rbac-proxy-cm podName:d024393a-e68c-4001-a13f-9a046bc2a924 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:11.448480108 +0000 UTC m=+71.413548295 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d024393a-e68c-4001-a13f-9a046bc2a924-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zvhvz" (UID: "d024393a-e68c-4001-a13f-9a046bc2a924") : configmap "kube-rbac-proxy" not found
Sep 10 20:37:10.448713 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:10.448569    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:37:10.448713 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:10.448621    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d024393a-e68c-4001-a13f-9a046bc2a924-insights-runtime-extractor-tls podName:d024393a-e68c-4001-a13f-9a046bc2a924 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:11.448606228 +0000 UTC m=+71.413674407 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d024393a-e68c-4001-a13f-9a046bc2a924-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zvhvz" (UID: "d024393a-e68c-4001-a13f-9a046bc2a924") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:37:11.456628 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:11.456593    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d024393a-e68c-4001-a13f-9a046bc2a924-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zvhvz\" (UID: \"d024393a-e68c-4001-a13f-9a046bc2a924\") " pod="openshift-insights/insights-runtime-extractor-zvhvz"
Sep 10 20:37:11.457193 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:11.456686    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d024393a-e68c-4001-a13f-9a046bc2a924-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zvhvz\" (UID: \"d024393a-e68c-4001-a13f-9a046bc2a924\") " pod="openshift-insights/insights-runtime-extractor-zvhvz"
Sep 10 20:37:11.457193 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:11.456767    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:37:11.457193 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:11.456791    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:37:11.457193 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:11.456861    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d024393a-e68c-4001-a13f-9a046bc2a924-insights-runtime-extractor-tls podName:d024393a-e68c-4001-a13f-9a046bc2a924 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:13.456823303 +0000 UTC m=+73.421891495 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d024393a-e68c-4001-a13f-9a046bc2a924-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zvhvz" (UID: "d024393a-e68c-4001-a13f-9a046bc2a924") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:37:11.457193 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:11.456878    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d024393a-e68c-4001-a13f-9a046bc2a924-kube-rbac-proxy-cm podName:d024393a-e68c-4001-a13f-9a046bc2a924 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:13.456871833 +0000 UTC m=+73.421940005 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d024393a-e68c-4001-a13f-9a046bc2a924-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zvhvz" (UID: "d024393a-e68c-4001-a13f-9a046bc2a924") : configmap "kube-rbac-proxy" not found
Sep 10 20:37:12.305528 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:12.305498    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kqb7x_af88567d-1c33-4670-acc8-484e2d8b11d4/dns-node-resolver/0.log"
Sep 10 20:37:12.364104 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:12.364072    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3173cc02-0594-4f6a-b991-297572a874d2-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-ttfr9\" (UID: \"3173cc02-0594-4f6a-b991-297572a874d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ttfr9"
Sep 10 20:37:12.364277 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:12.364227    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 20:37:12.364336 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:12.364301    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3173cc02-0594-4f6a-b991-297572a874d2-samples-operator-tls podName:3173cc02-0594-4f6a-b991-297572a874d2 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:20.364283153 +0000 UTC m=+80.329351344 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3173cc02-0594-4f6a-b991-297572a874d2-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-ttfr9" (UID: "3173cc02-0594-4f6a-b991-297572a874d2") : secret "samples-operator-tls" not found
Sep 10 20:37:12.931301 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:12.931265    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wl95n" event={"ID":"2e1ea2e3-fc76-40ad-a7a7-111080547273","Type":"ContainerStarted","Data":"6b35779d1a93789f05b694ceaff376a29dae58bb8ae0c7267ac0cb176e431aa3"}
Sep 10 20:37:12.931730 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:12.931359    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wl95n"
Sep 10 20:37:12.932625 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:12.932595    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v7c4f" event={"ID":"e14c5919-af21-4e8f-9783-a2ee2b4885c0","Type":"ContainerStarted","Data":"5e6b84200e96424d56fb2e7b1006cb9788852cccaead964c0ba7e28722432163"}
Sep 10 20:37:12.934036 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:12.934008    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-nn88n" event={"ID":"e7dcdc2d-00a7-4573-a1a0-20197bdc1ed9","Type":"ContainerStarted","Data":"fdd2be1257985f8e4d5db63897927f51a89d912c45293675a92adfdbdf4e5c68"}
Sep 10 20:37:12.965606 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:12.965562    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wl95n" podStartSLOduration=66.575549306 podStartE2EDuration="1m11.965548488s" totalImagesPullingTime="5.389999182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:37:12.964299689 +0000 UTC m=+72.929367883" watchObservedRunningTime="2026-09-10 20:37:12.965548488 +0000 UTC m=+72.930616674"
Sep 10 20:37:12.994244 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:12.994204    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-v7c4f" podStartSLOduration=66.602063079 podStartE2EDuration="1m11.994191635s" totalImagesPullingTime="5.392128556s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:37:12.993605612 +0000 UTC m=+72.958673808" watchObservedRunningTime="2026-09-10 20:37:12.994191635 +0000 UTC m=+72.959259824"
Sep 10 20:37:13.025448 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:13.025411    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-nn88n" podStartSLOduration=1.953479314 podStartE2EDuration="6.025400587s" totalImagesPullingTime="4.071921273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:37:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:37:13.024710094 +0000 UTC m=+72.989778292" watchObservedRunningTime="2026-09-10 20:37:13.025400587 +0000 UTC m=+72.990468972"
Sep 10 20:37:13.304844 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:13.304778    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fxt2l_ad5d5af7-f551-415c-9a17-fb89dd6bbb0a/node-ca/0.log"
Sep 10 20:37:13.471362 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:13.471327    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d024393a-e68c-4001-a13f-9a046bc2a924-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zvhvz\" (UID: \"d024393a-e68c-4001-a13f-9a046bc2a924\") " pod="openshift-insights/insights-runtime-extractor-zvhvz"
Sep 10 20:37:13.471466 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:13.471397    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d024393a-e68c-4001-a13f-9a046bc2a924-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zvhvz\" (UID: \"d024393a-e68c-4001-a13f-9a046bc2a924\") " pod="openshift-insights/insights-runtime-extractor-zvhvz"
Sep 10 20:37:13.471466 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:13.471462    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:37:13.471540 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:13.471508    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d024393a-e68c-4001-a13f-9a046bc2a924-kube-rbac-proxy-cm podName:d024393a-e68c-4001-a13f-9a046bc2a924 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:17.471495557 +0000 UTC m=+77.436563730 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d024393a-e68c-4001-a13f-9a046bc2a924-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zvhvz" (UID: "d024393a-e68c-4001-a13f-9a046bc2a924") : configmap "kube-rbac-proxy" not found
Sep 10 20:37:13.471582 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:13.471463    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:37:13.471617 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:13.471599    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d024393a-e68c-4001-a13f-9a046bc2a924-insights-runtime-extractor-tls podName:d024393a-e68c-4001-a13f-9a046bc2a924 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:17.471583579 +0000 UTC m=+77.436651757 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d024393a-e68c-4001-a13f-9a046bc2a924-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zvhvz" (UID: "d024393a-e68c-4001-a13f-9a046bc2a924") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:37:14.305006 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:14.304974    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x9fdk_88836648-21c4-45c8-9cd9-1e30c6fb4529/serve-healthcheck-canary/0.log"
Sep 10 20:37:14.455223 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:14.455188    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-8tmw9"]
Sep 10 20:37:14.458301 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:14.458279    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-8tmw9"
Sep 10 20:37:14.462013 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:14.461984    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 10 20:37:14.462143 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:14.462073    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-nmv4h\""
Sep 10 20:37:14.462775 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:14.462745    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 10 20:37:14.462775 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:14.462744    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 10 20:37:14.462995 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:14.462870    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 10 20:37:14.470082 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:14.470043    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-8tmw9"]
Sep 10 20:37:14.479407 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:14.479380    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/703c98fe-8048-4747-8cd3-1bd7a0fd0007-signing-key\") pod \"service-ca-568869fdc5-8tmw9\" (UID: \"703c98fe-8048-4747-8cd3-1bd7a0fd0007\") " pod="openshift-service-ca/service-ca-568869fdc5-8tmw9"
Sep 10 20:37:14.479504 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:14.479420    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grh94\" (UniqueName: \"kubernetes.io/projected/703c98fe-8048-4747-8cd3-1bd7a0fd0007-kube-api-access-grh94\") pod \"service-ca-568869fdc5-8tmw9\" (UID: \"703c98fe-8048-4747-8cd3-1bd7a0fd0007\") " pod="openshift-service-ca/service-ca-568869fdc5-8tmw9"
Sep 10 20:37:14.479504 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:14.479485    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/703c98fe-8048-4747-8cd3-1bd7a0fd0007-signing-cabundle\") pod \"service-ca-568869fdc5-8tmw9\" (UID: \"703c98fe-8048-4747-8cd3-1bd7a0fd0007\") " pod="openshift-service-ca/service-ca-568869fdc5-8tmw9"
Sep 10 20:37:14.579877 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:14.579852    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/703c98fe-8048-4747-8cd3-1bd7a0fd0007-signing-key\") pod \"service-ca-568869fdc5-8tmw9\" (UID: \"703c98fe-8048-4747-8cd3-1bd7a0fd0007\") " pod="openshift-service-ca/service-ca-568869fdc5-8tmw9"
Sep 10 20:37:14.579988 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:14.579899    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grh94\" (UniqueName: \"kubernetes.io/projected/703c98fe-8048-4747-8cd3-1bd7a0fd0007-kube-api-access-grh94\") pod \"service-ca-568869fdc5-8tmw9\" (UID: \"703c98fe-8048-4747-8cd3-1bd7a0fd0007\") " pod="openshift-service-ca/service-ca-568869fdc5-8tmw9"
Sep 10 20:37:14.579988 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:14.579965    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/703c98fe-8048-4747-8cd3-1bd7a0fd0007-signing-cabundle\") pod \"service-ca-568869fdc5-8tmw9\" (UID: \"703c98fe-8048-4747-8cd3-1bd7a0fd0007\") " pod="openshift-service-ca/service-ca-568869fdc5-8tmw9"
Sep 10 20:37:14.580581 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:14.580559    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/703c98fe-8048-4747-8cd3-1bd7a0fd0007-signing-cabundle\") pod \"service-ca-568869fdc5-8tmw9\" (UID: \"703c98fe-8048-4747-8cd3-1bd7a0fd0007\") " pod="openshift-service-ca/service-ca-568869fdc5-8tmw9"
Sep 10 20:37:14.582375 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:14.582357    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/703c98fe-8048-4747-8cd3-1bd7a0fd0007-signing-key\") pod \"service-ca-568869fdc5-8tmw9\" (UID: \"703c98fe-8048-4747-8cd3-1bd7a0fd0007\") " pod="openshift-service-ca/service-ca-568869fdc5-8tmw9"
Sep 10 20:37:14.589843 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:14.589815    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grh94\" (UniqueName: \"kubernetes.io/projected/703c98fe-8048-4747-8cd3-1bd7a0fd0007-kube-api-access-grh94\") pod \"service-ca-568869fdc5-8tmw9\" (UID: \"703c98fe-8048-4747-8cd3-1bd7a0fd0007\") " pod="openshift-service-ca/service-ca-568869fdc5-8tmw9"
Sep 10 20:37:14.768003 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:14.767976    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-8tmw9"
Sep 10 20:37:15.039195 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:15.039116    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-8tmw9"]
Sep 10 20:37:15.042268 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:37:15.042238    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod703c98fe_8048_4747_8cd3_1bd7a0fd0007.slice/crio-116ba5736f1854c02aadfb6eb583654adefb32f42d51447249f0c8edbf968eee WatchSource:0}: Error finding container 116ba5736f1854c02aadfb6eb583654adefb32f42d51447249f0c8edbf968eee: Status 404 returned error can't find the container with id 116ba5736f1854c02aadfb6eb583654adefb32f42d51447249f0c8edbf968eee
Sep 10 20:37:15.942207 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:15.942155    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-8tmw9" event={"ID":"703c98fe-8048-4747-8cd3-1bd7a0fd0007","Type":"ContainerStarted","Data":"116ba5736f1854c02aadfb6eb583654adefb32f42d51447249f0c8edbf968eee"}
Sep 10 20:37:16.946900 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:16.946867    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-8tmw9" event={"ID":"703c98fe-8048-4747-8cd3-1bd7a0fd0007","Type":"ContainerStarted","Data":"bf64a448da55c07c9083f43df8556647e49e564a42ad2eb43058d376ce875742"}
Sep 10 20:37:16.968752 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:16.968698    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-8tmw9" podStartSLOduration=1.159617486 podStartE2EDuration="2.968678745s" totalImagesPullingTime="1.809061259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:37:16.967303511 +0000 UTC m=+76.932371718" watchObservedRunningTime="2026-09-10 20:37:16.968678745 +0000 UTC m=+76.933746944"
Sep 10 20:37:17.499611 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:17.499580    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d024393a-e68c-4001-a13f-9a046bc2a924-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zvhvz\" (UID: \"d024393a-e68c-4001-a13f-9a046bc2a924\") " pod="openshift-insights/insights-runtime-extractor-zvhvz"
Sep 10 20:37:17.499742 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:17.499660    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d024393a-e68c-4001-a13f-9a046bc2a924-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zvhvz\" (UID: \"d024393a-e68c-4001-a13f-9a046bc2a924\") " pod="openshift-insights/insights-runtime-extractor-zvhvz"
Sep 10 20:37:17.499742 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:17.499689    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:37:17.499820 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:17.499767    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d024393a-e68c-4001-a13f-9a046bc2a924-kube-rbac-proxy-cm podName:d024393a-e68c-4001-a13f-9a046bc2a924 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:25.499753716 +0000 UTC m=+85.464821888 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d024393a-e68c-4001-a13f-9a046bc2a924-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zvhvz" (UID: "d024393a-e68c-4001-a13f-9a046bc2a924") : configmap "kube-rbac-proxy" not found
Sep 10 20:37:17.499820 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:17.499810    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:37:17.499905 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:17.499873    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d024393a-e68c-4001-a13f-9a046bc2a924-insights-runtime-extractor-tls podName:d024393a-e68c-4001-a13f-9a046bc2a924 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:25.499856219 +0000 UTC m=+85.464924406 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d024393a-e68c-4001-a13f-9a046bc2a924-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zvhvz" (UID: "d024393a-e68c-4001-a13f-9a046bc2a924") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:37:20.421021 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:20.420987    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3173cc02-0594-4f6a-b991-297572a874d2-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-ttfr9\" (UID: \"3173cc02-0594-4f6a-b991-297572a874d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ttfr9"
Sep 10 20:37:20.423268 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:20.423250    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3173cc02-0594-4f6a-b991-297572a874d2-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-ttfr9\" (UID: \"3173cc02-0594-4f6a-b991-297572a874d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ttfr9"
Sep 10 20:37:20.563462 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:20.563439    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ttfr9"
Sep 10 20:37:20.833580 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:20.833553    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ttfr9"]
Sep 10 20:37:20.958860 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:20.958830    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ttfr9" event={"ID":"3173cc02-0594-4f6a-b991-297572a874d2","Type":"ContainerStarted","Data":"8760b0755bbf012306370aee6581292f7c2d7265a337ae081ef895c114e2e9ee"}
Sep 10 20:37:22.966427 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:22.966303    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ttfr9" event={"ID":"3173cc02-0594-4f6a-b991-297572a874d2","Type":"ContainerStarted","Data":"9275140b12feb013ef2c4d6b5952af815322155a01990f54f507d24d86864933"}
Sep 10 20:37:22.966427 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:22.966364    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ttfr9" event={"ID":"3173cc02-0594-4f6a-b991-297572a874d2","Type":"ContainerStarted","Data":"ad2f0f851046f7f11536275122b76088bb8c73ff86c78f3d53ad5aec03649315"}
Sep 10 20:37:22.984559 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:22.984515    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-ttfr9" podStartSLOduration=17.21909466 podStartE2EDuration="18.984499408s" totalImagesPullingTime="1.765404748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:37:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:37:22.98387011 +0000 UTC m=+82.948938316" watchObservedRunningTime="2026-09-10 20:37:22.984499408 +0000 UTC m=+82.949567608"
Sep 10 20:37:25.560836 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:25.560802    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d024393a-e68c-4001-a13f-9a046bc2a924-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zvhvz\" (UID: \"d024393a-e68c-4001-a13f-9a046bc2a924\") " pod="openshift-insights/insights-runtime-extractor-zvhvz"
Sep 10 20:37:25.561219 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:25.560894    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d024393a-e68c-4001-a13f-9a046bc2a924-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zvhvz\" (UID: \"d024393a-e68c-4001-a13f-9a046bc2a924\") " pod="openshift-insights/insights-runtime-extractor-zvhvz"
Sep 10 20:37:25.565099 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:25.561339    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:37:25.565099 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:37:25.561420    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d024393a-e68c-4001-a13f-9a046bc2a924-insights-runtime-extractor-tls podName:d024393a-e68c-4001-a13f-9a046bc2a924 nodeName:}" failed. No retries permitted until 2026-09-10 20:37:41.561401449 +0000 UTC m=+101.526469638 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d024393a-e68c-4001-a13f-9a046bc2a924-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zvhvz" (UID: "d024393a-e68c-4001-a13f-9a046bc2a924") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:37:25.565099 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:25.561798    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d024393a-e68c-4001-a13f-9a046bc2a924-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zvhvz\" (UID: \"d024393a-e68c-4001-a13f-9a046bc2a924\") " pod="openshift-insights/insights-runtime-extractor-zvhvz"
Sep 10 20:37:35.501858 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:35.501816    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78c4556494-wmqdt"]
Sep 10 20:37:35.506349 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:35.506330    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78c4556494-wmqdt"
Sep 10 20:37:35.521627 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:35.521605    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 20:37:35.524465 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:35.524444    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 20:37:35.525536 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:35.525493    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 10 20:37:35.525536 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:35.525523    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 20:37:35.525536 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:35.525527    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 10 20:37:35.525776 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:35.525593    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 10 20:37:35.525776 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:35.525502    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 10 20:37:35.554883 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:35.554863    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78c4556494-wmqdt"]
Sep 10 20:37:35.625989 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:35.625961    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cc6799f7-db89-42a7-8878-686ebc2ace3f-hub\") pod \"cluster-proxy-proxy-agent-78c4556494-wmqdt\" (UID: \"cc6799f7-db89-42a7-8878-686ebc2ace3f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78c4556494-wmqdt"
Sep 10 20:37:35.626096 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:35.625999    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cc6799f7-db89-42a7-8878-686ebc2ace3f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-78c4556494-wmqdt\" (UID: \"cc6799f7-db89-42a7-8878-686ebc2ace3f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78c4556494-wmqdt"
Sep 10 20:37:35.626096 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:35.626071    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6799f7-db89-42a7-8878-686ebc2ace3f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-78c4556494-wmqdt\" (UID: \"cc6799f7-db89-42a7-8878-686ebc2ace3f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78c4556494-wmqdt"
Sep 10 20:37:35.626245 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:35.626107    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cc6799f7-db89-42a7-8878-686ebc2ace3f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-78c4556494-wmqdt\" (UID: \"cc6799f7-db89-42a7-8878-686ebc2ace3f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78c4556494-wmqdt"
Sep 10 20:37:35.626245 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:35.626206    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cc6799f7-db89-42a7-8878-686ebc2ace3f-ca\") pod \"cluster-proxy-proxy-agent-78c4556494-wmqdt\" (UID: \"cc6799f7-db89-42a7-8878-686ebc2ace3f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78c4556494-wmqdt"
Sep 10 20:37:35.626245 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:35.626236    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dbvf\" (UniqueName: \"kubernetes.io/projected/cc6799f7-db89-42a7-8878-686ebc2ace3f-kube-api-access-5dbvf\") pod \"cluster-proxy-proxy-agent-78c4556494-wmqdt\" (UID: \"cc6799f7-db89-42a7-8878-686ebc2ace3f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78c4556494-wmqdt"
Sep 10 20:37:35.726450 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:35.726424    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6799f7-db89-42a7-8878-686ebc2ace3f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-78c4556494-wmqdt\" (UID: \"cc6799f7-db89-42a7-8878-686ebc2ace3f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78c4556494-wmqdt"
Sep 10 20:37:35.726546 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:35.726453    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cc6799f7-db89-42a7-8878-686ebc2ace3f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-78c4556494-wmqdt\" (UID: \"cc6799f7-db89-42a7-8878-686ebc2ace3f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78c4556494-wmqdt"
Sep 10 20:37:35.726546 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:35.726505    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cc6799f7-db89-42a7-8878-686ebc2ace3f-ca\") pod \"cluster-proxy-proxy-agent-78c4556494-wmqdt\" (UID: \"cc6799f7-db89-42a7-8878-686ebc2ace3f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78c4556494-wmqdt"
Sep 10 20:37:35.726685 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:35.726660    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dbvf\" (UniqueName: \"kubernetes.io/projected/cc6799f7-db89-42a7-8878-686ebc2ace3f-kube-api-access-5dbvf\") pod \"cluster-proxy-proxy-agent-78c4556494-wmqdt\" (UID: \"cc6799f7-db89-42a7-8878-686ebc2ace3f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78c4556494-wmqdt"
Sep 10 20:37:35.726739 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:35.726721    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cc6799f7-db89-42a7-8878-686ebc2ace3f-hub\") pod \"cluster-proxy-proxy-agent-78c4556494-wmqdt\" (UID: \"cc6799f7-db89-42a7-8878-686ebc2ace3f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78c4556494-wmqdt"
Sep 10 20:37:35.726792 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:35.726770    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cc6799f7-db89-42a7-8878-686ebc2ace3f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-78c4556494-wmqdt\" (UID: \"cc6799f7-db89-42a7-8878-686ebc2ace3f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78c4556494-wmqdt"
Sep 10 20:37:35.727500 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:35.727475    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6799f7-db89-42a7-8878-686ebc2ace3f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-78c4556494-wmqdt\" (UID: \"cc6799f7-db89-42a7-8878-686ebc2ace3f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78c4556494-wmqdt"
Sep 10 20:37:35.729249 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:35.729225    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cc6799f7-db89-42a7-8878-686ebc2ace3f-ca\") pod \"cluster-proxy-proxy-agent-78c4556494-wmqdt\" (UID: \"cc6799f7-db89-42a7-8878-686ebc2ace3f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78c4556494-wmqdt"
Sep 10 20:37:35.729558 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:35.729534    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cc6799f7-db89-42a7-8878-686ebc2ace3f-hub\") pod \"cluster-proxy-proxy-agent-78c4556494-wmqdt\" (UID: \"cc6799f7-db89-42a7-8878-686ebc2ace3f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78c4556494-wmqdt"
Sep 10 20:37:35.729748 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:35.729726    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cc6799f7-db89-42a7-8878-686ebc2ace3f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-78c4556494-wmqdt\" (UID: \"cc6799f7-db89-42a7-8878-686ebc2ace3f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78c4556494-wmqdt"
Sep 10 20:37:35.729958 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:35.729943    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cc6799f7-db89-42a7-8878-686ebc2ace3f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-78c4556494-wmqdt\" (UID: \"cc6799f7-db89-42a7-8878-686ebc2ace3f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78c4556494-wmqdt"
Sep 10 20:37:35.740187 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:35.740142    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dbvf\" (UniqueName: \"kubernetes.io/projected/cc6799f7-db89-42a7-8878-686ebc2ace3f-kube-api-access-5dbvf\") pod \"cluster-proxy-proxy-agent-78c4556494-wmqdt\" (UID: \"cc6799f7-db89-42a7-8878-686ebc2ace3f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78c4556494-wmqdt"
Sep 10 20:37:35.830818 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:35.830789    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78c4556494-wmqdt"
Sep 10 20:37:36.105402 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:36.105346    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78c4556494-wmqdt"]
Sep 10 20:37:36.108336 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:37:36.108302    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc6799f7_db89_42a7_8878_686ebc2ace3f.slice/crio-8078b9c7bf5ae6691d98bc3a74ecdf806109522650ea2c3ca1b91756d60d1a89 WatchSource:0}: Error finding container 8078b9c7bf5ae6691d98bc3a74ecdf806109522650ea2c3ca1b91756d60d1a89: Status 404 returned error can't find the container with id 8078b9c7bf5ae6691d98bc3a74ecdf806109522650ea2c3ca1b91756d60d1a89
Sep 10 20:37:37.014868 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:37.014836    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78c4556494-wmqdt" event={"ID":"cc6799f7-db89-42a7-8878-686ebc2ace3f","Type":"ContainerStarted","Data":"8078b9c7bf5ae6691d98bc3a74ecdf806109522650ea2c3ca1b91756d60d1a89"}
Sep 10 20:37:38.245263 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:38.245222    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d345a7a-fae7-4330-baf7-8be851843ba9-metrics-tls\") pod \"dns-default-f6hc2\" (UID: \"7d345a7a-fae7-4330-baf7-8be851843ba9\") " pod="openshift-dns/dns-default-f6hc2"
Sep 10 20:37:38.245735 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:38.245304    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-registry-tls\") pod \"image-registry-67bfdbc8bd-nprqr\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") " pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:37:38.248005 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:38.247952    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d345a7a-fae7-4330-baf7-8be851843ba9-metrics-tls\") pod \"dns-default-f6hc2\" (UID: \"7d345a7a-fae7-4330-baf7-8be851843ba9\") " pod="openshift-dns/dns-default-f6hc2"
Sep 10 20:37:38.248122 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:38.248102    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-registry-tls\") pod \"image-registry-67bfdbc8bd-nprqr\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") " pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:37:38.290624 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:38.290598    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-drws7\""
Sep 10 20:37:38.298803 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:38.298697    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:37:38.309356 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:38.309332    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-89pjq\""
Sep 10 20:37:38.317398 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:38.317375    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-f6hc2"
Sep 10 20:37:38.621698 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:38.621669    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-f6hc2"]
Sep 10 20:37:38.628206 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:38.628097    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67bfdbc8bd-nprqr"]
Sep 10 20:37:39.545388 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:37:39.545356    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d345a7a_fae7_4330_baf7_8be851843ba9.slice/crio-42befa268553fd1de2c582b5179a47098428a09e15eb3fd62f8e49c95311f7b5 WatchSource:0}: Error finding container 42befa268553fd1de2c582b5179a47098428a09e15eb3fd62f8e49c95311f7b5: Status 404 returned error can't find the container with id 42befa268553fd1de2c582b5179a47098428a09e15eb3fd62f8e49c95311f7b5
Sep 10 20:37:39.546149 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:37:39.546048    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d5ba49b_5e81_426b_b8b0_ba260f754c14.slice/crio-337dee02d5aa287c3c7bf18255409d312ed0c00a4e11695c3840f8f4c4c7c59a WatchSource:0}: Error finding container 337dee02d5aa287c3c7bf18255409d312ed0c00a4e11695c3840f8f4c4c7c59a: Status 404 returned error can't find the container with id 337dee02d5aa287c3c7bf18255409d312ed0c00a4e11695c3840f8f4c4c7c59a
Sep 10 20:37:40.026768 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:40.026677    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr" event={"ID":"1d5ba49b-5e81-426b-b8b0-ba260f754c14","Type":"ContainerStarted","Data":"43dd900a020a1dc3a73351f3346db55d43b6441a45845e29dfbc0d1856e23a42"}
Sep 10 20:37:40.026768 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:40.026739    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr" event={"ID":"1d5ba49b-5e81-426b-b8b0-ba260f754c14","Type":"ContainerStarted","Data":"337dee02d5aa287c3c7bf18255409d312ed0c00a4e11695c3840f8f4c4c7c59a"}
Sep 10 20:37:40.026971 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:40.026794    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:37:40.027827 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:40.027801    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f6hc2" event={"ID":"7d345a7a-fae7-4330-baf7-8be851843ba9","Type":"ContainerStarted","Data":"42befa268553fd1de2c582b5179a47098428a09e15eb3fd62f8e49c95311f7b5"}
Sep 10 20:37:40.029858 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:40.029837    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78c4556494-wmqdt" event={"ID":"cc6799f7-db89-42a7-8878-686ebc2ace3f","Type":"ContainerStarted","Data":"1d84c3b234a7a1b397ec93fd6bc0b1d8d54fcc218d53173c5f9afdcd5021333e"}
Sep 10 20:37:40.029943 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:40.029866    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78c4556494-wmqdt" event={"ID":"cc6799f7-db89-42a7-8878-686ebc2ace3f","Type":"ContainerStarted","Data":"6c4a7fdbc963923105cc8d07d9a2184d6a81fd02cd0b8d86280785c963559fa4"}
Sep 10 20:37:40.029943 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:40.029878    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78c4556494-wmqdt" event={"ID":"cc6799f7-db89-42a7-8878-686ebc2ace3f","Type":"ContainerStarted","Data":"78e2f4689e714e523f857df168c4a1f3116519533776820d6e1dd554d14a9a9e"}
Sep 10 20:37:40.077358 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:40.077310    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr" podStartSLOduration=99.077294853 podStartE2EDuration="1m39.077294853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:37:40.053739848 +0000 UTC m=+100.018808042" watchObservedRunningTime="2026-09-10 20:37:40.077294853 +0000 UTC m=+100.042363049"
Sep 10 20:37:40.077448 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:40.077430    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78c4556494-wmqdt" podStartSLOduration=1.5752971059999998 podStartE2EDuration="5.077425459s" totalImagesPullingTime="3.502128353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:37:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:37:40.077097534 +0000 UTC m=+100.042165730" watchObservedRunningTime="2026-09-10 20:37:40.077425459 +0000 UTC m=+100.042493653"
Sep 10 20:37:41.574203 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:41.574144    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d024393a-e68c-4001-a13f-9a046bc2a924-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zvhvz\" (UID: \"d024393a-e68c-4001-a13f-9a046bc2a924\") " pod="openshift-insights/insights-runtime-extractor-zvhvz"
Sep 10 20:37:41.576832 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:41.576805    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d024393a-e68c-4001-a13f-9a046bc2a924-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zvhvz\" (UID: \"d024393a-e68c-4001-a13f-9a046bc2a924\") " pod="openshift-insights/insights-runtime-extractor-zvhvz"
Sep 10 20:37:41.874391 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:41.874361    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zvhvz"
Sep 10 20:37:42.037577 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:42.037542    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f6hc2" event={"ID":"7d345a7a-fae7-4330-baf7-8be851843ba9","Type":"ContainerStarted","Data":"d0f5b6bfaba1360fe4f04e7667d34b0f3e0998d5af946cc14a4228179b9a758d"}
Sep 10 20:37:42.037679 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:42.037590    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f6hc2" event={"ID":"7d345a7a-fae7-4330-baf7-8be851843ba9","Type":"ContainerStarted","Data":"1537f730abe5f9455918e62b406149ac5019a38b0e0ef26835b548c4b66ed203"}
Sep 10 20:37:42.037725 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:42.037670    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-f6hc2"
Sep 10 20:37:42.063025 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:42.062977    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-f6hc2" podStartSLOduration=66.066614805 podStartE2EDuration="1m8.062965903s" totalImagesPullingTime="1.996351098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:37:42.061289807 +0000 UTC m=+102.026358001" watchObservedRunningTime="2026-09-10 20:37:42.062965903 +0000 UTC m=+102.028034139"
Sep 10 20:37:42.146460 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:42.146408    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zvhvz"]
Sep 10 20:37:42.149433 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:37:42.149409    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd024393a_e68c_4001_a13f_9a046bc2a924.slice/crio-9639e5784144cf2c6fda65202bb7272a11cb79ee4ba6c67da1e3756b3282a868 WatchSource:0}: Error finding container 9639e5784144cf2c6fda65202bb7272a11cb79ee4ba6c67da1e3756b3282a868: Status 404 returned error can't find the container with id 9639e5784144cf2c6fda65202bb7272a11cb79ee4ba6c67da1e3756b3282a868
Sep 10 20:37:43.042367 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:43.042332    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zvhvz" event={"ID":"d024393a-e68c-4001-a13f-9a046bc2a924","Type":"ContainerStarted","Data":"6d61801e939b53f3574ed5fbf99a2485aa91411a90df9c412499b4416ecfd0a7"}
Sep 10 20:37:43.042746 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:43.042388    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zvhvz" event={"ID":"d024393a-e68c-4001-a13f-9a046bc2a924","Type":"ContainerStarted","Data":"9639e5784144cf2c6fda65202bb7272a11cb79ee4ba6c67da1e3756b3282a868"}
Sep 10 20:37:43.939515 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:43.939482    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wl95n"
Sep 10 20:37:44.048040 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:44.047996    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zvhvz" event={"ID":"d024393a-e68c-4001-a13f-9a046bc2a924","Type":"ContainerStarted","Data":"6806a75c37a146e43d08d28c1dbd999bc4ceac825cede00be27a67e22cd705f7"}
Sep 10 20:37:46.057576 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:46.057533    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zvhvz" event={"ID":"d024393a-e68c-4001-a13f-9a046bc2a924","Type":"ContainerStarted","Data":"61d6a773a1075f6ae57aa0a256ca9e3ee255283be41c91695ee4b25d6c6cc696"}
Sep 10 20:37:52.044599 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:52.044562    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-f6hc2"
Sep 10 20:37:52.070006 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:52.069954    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zvhvz" podStartSLOduration=40.252678472 podStartE2EDuration="43.06993689s" totalImagesPullingTime="2.817258418s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:37:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:37:46.085285565 +0000 UTC m=+106.050353761" watchObservedRunningTime="2026-09-10 20:37:52.06993689 +0000 UTC m=+112.035005084"
Sep 10 20:37:57.916660 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:57.916622    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67bfdbc8bd-nprqr"]
Sep 10 20:37:57.920995 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:57.920949    2564 patch_prober.go:28] interesting pod/image-registry-67bfdbc8bd-nprqr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 10 20:37:57.921157 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:57.921009    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr" podUID="1d5ba49b-5e81-426b-b8b0-ba260f754c14" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 20:37:59.415251 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.415218    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jltvz"]
Sep 10 20:37:59.422192 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.422156    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jltvz"
Sep 10 20:37:59.426101 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.426080    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 20:37:59.426535 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.426515    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 20:37:59.426671 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.426649    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pfdlc\""
Sep 10 20:37:59.428232 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.428210    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 20:37:59.428350 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.428233    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 20:37:59.428690 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.428615    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 20:37:59.428963 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.428937    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 20:37:59.489298 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.489274    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bfaa24d2-46c0-4a59-972b-97f3b965a05d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jltvz\" (UID: \"bfaa24d2-46c0-4a59-972b-97f3b965a05d\") " pod="openshift-monitoring/node-exporter-jltvz"
Sep 10 20:37:59.489398 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.489303    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bfaa24d2-46c0-4a59-972b-97f3b965a05d-node-exporter-wtmp\") pod \"node-exporter-jltvz\" (UID: \"bfaa24d2-46c0-4a59-972b-97f3b965a05d\") " pod="openshift-monitoring/node-exporter-jltvz"
Sep 10 20:37:59.489398 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.489320    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bfaa24d2-46c0-4a59-972b-97f3b965a05d-node-exporter-tls\") pod \"node-exporter-jltvz\" (UID: \"bfaa24d2-46c0-4a59-972b-97f3b965a05d\") " pod="openshift-monitoring/node-exporter-jltvz"
Sep 10 20:37:59.489398 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.489338    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bfaa24d2-46c0-4a59-972b-97f3b965a05d-root\") pod \"node-exporter-jltvz\" (UID: \"bfaa24d2-46c0-4a59-972b-97f3b965a05d\") " pod="openshift-monitoring/node-exporter-jltvz"
Sep 10 20:37:59.489398 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.489352    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bfaa24d2-46c0-4a59-972b-97f3b965a05d-node-exporter-textfile\") pod \"node-exporter-jltvz\" (UID: \"bfaa24d2-46c0-4a59-972b-97f3b965a05d\") " pod="openshift-monitoring/node-exporter-jltvz"
Sep 10 20:37:59.489551 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.489418    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfaa24d2-46c0-4a59-972b-97f3b965a05d-sys\") pod \"node-exporter-jltvz\" (UID: \"bfaa24d2-46c0-4a59-972b-97f3b965a05d\") " pod="openshift-monitoring/node-exporter-jltvz"
Sep 10 20:37:59.489551 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.489435    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skkzz\" (UniqueName: \"kubernetes.io/projected/bfaa24d2-46c0-4a59-972b-97f3b965a05d-kube-api-access-skkzz\") pod \"node-exporter-jltvz\" (UID: \"bfaa24d2-46c0-4a59-972b-97f3b965a05d\") " pod="openshift-monitoring/node-exporter-jltvz"
Sep 10 20:37:59.489551 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.489451    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfaa24d2-46c0-4a59-972b-97f3b965a05d-metrics-client-ca\") pod \"node-exporter-jltvz\" (UID: \"bfaa24d2-46c0-4a59-972b-97f3b965a05d\") " pod="openshift-monitoring/node-exporter-jltvz"
Sep 10 20:37:59.489551 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.489474    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bfaa24d2-46c0-4a59-972b-97f3b965a05d-node-exporter-accelerators-collector-config\") pod \"node-exporter-jltvz\" (UID: \"bfaa24d2-46c0-4a59-972b-97f3b965a05d\") " pod="openshift-monitoring/node-exporter-jltvz"
Sep 10 20:37:59.590391 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.590372    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bfaa24d2-46c0-4a59-972b-97f3b965a05d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jltvz\" (UID: \"bfaa24d2-46c0-4a59-972b-97f3b965a05d\") " pod="openshift-monitoring/node-exporter-jltvz"
Sep 10 20:37:59.590495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.590398    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bfaa24d2-46c0-4a59-972b-97f3b965a05d-node-exporter-wtmp\") pod \"node-exporter-jltvz\" (UID: \"bfaa24d2-46c0-4a59-972b-97f3b965a05d\") " pod="openshift-monitoring/node-exporter-jltvz"
Sep 10 20:37:59.590495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.590416    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bfaa24d2-46c0-4a59-972b-97f3b965a05d-node-exporter-tls\") pod \"node-exporter-jltvz\" (UID: \"bfaa24d2-46c0-4a59-972b-97f3b965a05d\") " pod="openshift-monitoring/node-exporter-jltvz"
Sep 10 20:37:59.590495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.590434    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bfaa24d2-46c0-4a59-972b-97f3b965a05d-root\") pod \"node-exporter-jltvz\" (UID: \"bfaa24d2-46c0-4a59-972b-97f3b965a05d\") " pod="openshift-monitoring/node-exporter-jltvz"
Sep 10 20:37:59.590495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.590449    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bfaa24d2-46c0-4a59-972b-97f3b965a05d-node-exporter-textfile\") pod \"node-exporter-jltvz\" (UID: \"bfaa24d2-46c0-4a59-972b-97f3b965a05d\") " pod="openshift-monitoring/node-exporter-jltvz"
Sep 10 20:37:59.590495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.590488    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfaa24d2-46c0-4a59-972b-97f3b965a05d-sys\") pod \"node-exporter-jltvz\" (UID: \"bfaa24d2-46c0-4a59-972b-97f3b965a05d\") " pod="openshift-monitoring/node-exporter-jltvz"
Sep 10 20:37:59.590792 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.590515    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skkzz\" (UniqueName: \"kubernetes.io/projected/bfaa24d2-46c0-4a59-972b-97f3b965a05d-kube-api-access-skkzz\") pod \"node-exporter-jltvz\" (UID: \"bfaa24d2-46c0-4a59-972b-97f3b965a05d\") " pod="openshift-monitoring/node-exporter-jltvz"
Sep 10 20:37:59.590792 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.590528    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bfaa24d2-46c0-4a59-972b-97f3b965a05d-root\") pod \"node-exporter-jltvz\" (UID: \"bfaa24d2-46c0-4a59-972b-97f3b965a05d\") " pod="openshift-monitoring/node-exporter-jltvz"
Sep 10 20:37:59.590792 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.590540    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfaa24d2-46c0-4a59-972b-97f3b965a05d-metrics-client-ca\") pod \"node-exporter-jltvz\" (UID: \"bfaa24d2-46c0-4a59-972b-97f3b965a05d\") " pod="openshift-monitoring/node-exporter-jltvz"
Sep 10 20:37:59.590792 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.590553    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bfaa24d2-46c0-4a59-972b-97f3b965a05d-node-exporter-wtmp\") pod \"node-exporter-jltvz\" (UID: \"bfaa24d2-46c0-4a59-972b-97f3b965a05d\") " pod="openshift-monitoring/node-exporter-jltvz"
Sep 10 20:37:59.590792 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.590565    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bfaa24d2-46c0-4a59-972b-97f3b965a05d-node-exporter-accelerators-collector-config\") pod \"node-exporter-jltvz\" (UID: \"bfaa24d2-46c0-4a59-972b-97f3b965a05d\") " pod="openshift-monitoring/node-exporter-jltvz"
Sep 10 20:37:59.590792 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.590579    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfaa24d2-46c0-4a59-972b-97f3b965a05d-sys\") pod \"node-exporter-jltvz\" (UID: \"bfaa24d2-46c0-4a59-972b-97f3b965a05d\") " pod="openshift-monitoring/node-exporter-jltvz"
Sep 10 20:37:59.591085 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.591019    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bfaa24d2-46c0-4a59-972b-97f3b965a05d-node-exporter-textfile\") pod \"node-exporter-jltvz\" (UID: \"bfaa24d2-46c0-4a59-972b-97f3b965a05d\") " pod="openshift-monitoring/node-exporter-jltvz"
Sep 10 20:37:59.591085 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.591076    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bfaa24d2-46c0-4a59-972b-97f3b965a05d-node-exporter-accelerators-collector-config\") pod \"node-exporter-jltvz\" (UID: \"bfaa24d2-46c0-4a59-972b-97f3b965a05d\") " pod="openshift-monitoring/node-exporter-jltvz"
Sep 10 20:37:59.591207 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.591097    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfaa24d2-46c0-4a59-972b-97f3b965a05d-metrics-client-ca\") pod \"node-exporter-jltvz\" (UID: \"bfaa24d2-46c0-4a59-972b-97f3b965a05d\") " pod="openshift-monitoring/node-exporter-jltvz"
Sep 10 20:37:59.592764 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.592739    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bfaa24d2-46c0-4a59-972b-97f3b965a05d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jltvz\" (UID: \"bfaa24d2-46c0-4a59-972b-97f3b965a05d\") " pod="openshift-monitoring/node-exporter-jltvz"
Sep 10 20:37:59.593020 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.593003    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bfaa24d2-46c0-4a59-972b-97f3b965a05d-node-exporter-tls\") pod \"node-exporter-jltvz\" (UID: \"bfaa24d2-46c0-4a59-972b-97f3b965a05d\") " pod="openshift-monitoring/node-exporter-jltvz"
Sep 10 20:37:59.600975 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.600956    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skkzz\" (UniqueName: \"kubernetes.io/projected/bfaa24d2-46c0-4a59-972b-97f3b965a05d-kube-api-access-skkzz\") pod \"node-exporter-jltvz\" (UID: \"bfaa24d2-46c0-4a59-972b-97f3b965a05d\") " pod="openshift-monitoring/node-exporter-jltvz"
Sep 10 20:37:59.732755 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:37:59.732697    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jltvz"
Sep 10 20:37:59.745695 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:37:59.745670    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfaa24d2_46c0_4a59_972b_97f3b965a05d.slice/crio-00a5283a13944ddfc565fb86ef3376f4be4524254021b426d31f225170393a98 WatchSource:0}: Error finding container 00a5283a13944ddfc565fb86ef3376f4be4524254021b426d31f225170393a98: Status 404 returned error can't find the container with id 00a5283a13944ddfc565fb86ef3376f4be4524254021b426d31f225170393a98
Sep 10 20:38:00.104558 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:00.104521    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jltvz" event={"ID":"bfaa24d2-46c0-4a59-972b-97f3b965a05d","Type":"ContainerStarted","Data":"00a5283a13944ddfc565fb86ef3376f4be4524254021b426d31f225170393a98"}
Sep 10 20:38:01.108912 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:01.108838    2564 generic.go:356] "Generic (PLEG): container finished" podID="bfaa24d2-46c0-4a59-972b-97f3b965a05d" containerID="1b125bef7bb8cfa563eed8847d9d599d6cf3e520710805128202b41444b74e17" exitCode=0
Sep 10 20:38:01.109220 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:01.108929    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jltvz" event={"ID":"bfaa24d2-46c0-4a59-972b-97f3b965a05d","Type":"ContainerDied","Data":"1b125bef7bb8cfa563eed8847d9d599d6cf3e520710805128202b41444b74e17"}
Sep 10 20:38:02.115660 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:02.115619    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jltvz" event={"ID":"bfaa24d2-46c0-4a59-972b-97f3b965a05d","Type":"ContainerStarted","Data":"bc8ed932167610f0b030d16962bb5b091c8c8b5ef41c3ca0b0115699488642c1"}
Sep 10 20:38:02.116105 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:02.115686    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jltvz" event={"ID":"bfaa24d2-46c0-4a59-972b-97f3b965a05d","Type":"ContainerStarted","Data":"ef2af0b4d7990ba1a5ad80da4468a8503e93512e948e6f6d16fd0b2d52640662"}
Sep 10 20:38:02.138556 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:02.138498    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jltvz" podStartSLOduration=2.007012244 podStartE2EDuration="3.138482424s" totalImagesPullingTime="1.07273943s" totalInitContainerRuntime="58.73075ms" isStatefulPod=true podCreationTimestamp="2026-09-10 20:37:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:38:02.137142963 +0000 UTC m=+122.102211157" watchObservedRunningTime="2026-09-10 20:38:02.138482424 +0000 UTC m=+122.103550618"
Sep 10 20:38:07.921788 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:07.921753    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:38:10.269865 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:10.269830    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b-metrics-certs\") pod \"network-metrics-daemon-vrrvq\" (UID: \"e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b\") " pod="openshift-multus/network-metrics-daemon-vrrvq"
Sep 10 20:38:10.272111 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:10.272091    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b-metrics-certs\") pod \"network-metrics-daemon-vrrvq\" (UID: \"e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b\") " pod="openshift-multus/network-metrics-daemon-vrrvq"
Sep 10 20:38:10.349525 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:10.349494    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ks6z6\""
Sep 10 20:38:10.358339 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:10.358318    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrrvq"
Sep 10 20:38:10.662581 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:10.662559    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vrrvq"]
Sep 10 20:38:10.663616 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:38:10.663584    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1d7c7d2_64d4_43bf_8f9a_5d8e0f8b9e7b.slice/crio-4eebb688f22763de7efa124bc8a543262e279864b808d237584da76688488cf1 WatchSource:0}: Error finding container 4eebb688f22763de7efa124bc8a543262e279864b808d237584da76688488cf1: Status 404 returned error can't find the container with id 4eebb688f22763de7efa124bc8a543262e279864b808d237584da76688488cf1
Sep 10 20:38:11.145940 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:11.145908    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vrrvq" event={"ID":"e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b","Type":"ContainerStarted","Data":"4eebb688f22763de7efa124bc8a543262e279864b808d237584da76688488cf1"}
Sep 10 20:38:12.151286 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:12.151253    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vrrvq" event={"ID":"e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b","Type":"ContainerStarted","Data":"1d31619e06838a9d14b22f97b5c6dcbdaf39cca8e1950842aa6048fc2165deb9"}
Sep 10 20:38:12.151592 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:12.151316    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vrrvq" event={"ID":"e1d7c7d2-64d4-43bf-8f9a-5d8e0f8b9e7b","Type":"ContainerStarted","Data":"3c007af879b1a45443621f32724f68d58d20a409b2a5d03face6bd603505291b"}
Sep 10 20:38:12.170305 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:12.170231    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vrrvq" podStartSLOduration=129.920246967 podStartE2EDuration="2m11.170215627s" totalImagesPullingTime="1.24996866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:38:12.169220029 +0000 UTC m=+132.134288223" watchObservedRunningTime="2026-09-10 20:38:12.170215627 +0000 UTC m=+132.135283824"
Sep 10 20:38:20.332322 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:20.332287    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x9fdk_88836648-21c4-45c8-9cd9-1e30c6fb4529/serve-healthcheck-canary/0.log"
Sep 10 20:38:22.935681 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:22.935610    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr" podUID="1d5ba49b-5e81-426b-b8b0-ba260f754c14" containerName="registry" containerID="cri-o://43dd900a020a1dc3a73351f3346db55d43b6441a45845e29dfbc0d1856e23a42" gracePeriod=30
Sep 10 20:38:23.175630 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:23.175607    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:38:23.189931 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:23.189878    2564 generic.go:356] "Generic (PLEG): container finished" podID="1d5ba49b-5e81-426b-b8b0-ba260f754c14" containerID="43dd900a020a1dc3a73351f3346db55d43b6441a45845e29dfbc0d1856e23a42" exitCode=0
Sep 10 20:38:23.189931 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:23.189909    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr"
Sep 10 20:38:23.190067 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:23.189957    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr" event={"ID":"1d5ba49b-5e81-426b-b8b0-ba260f754c14","Type":"ContainerDied","Data":"43dd900a020a1dc3a73351f3346db55d43b6441a45845e29dfbc0d1856e23a42"}
Sep 10 20:38:23.190067 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:23.190003    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67bfdbc8bd-nprqr" event={"ID":"1d5ba49b-5e81-426b-b8b0-ba260f754c14","Type":"ContainerDied","Data":"337dee02d5aa287c3c7bf18255409d312ed0c00a4e11695c3840f8f4c4c7c59a"}
Sep 10 20:38:23.190067 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:23.190025    2564 scope.go:117] "RemoveContainer" containerID="43dd900a020a1dc3a73351f3346db55d43b6441a45845e29dfbc0d1856e23a42"
Sep 10 20:38:23.198255 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:23.198234    2564 scope.go:117] "RemoveContainer" containerID="43dd900a020a1dc3a73351f3346db55d43b6441a45845e29dfbc0d1856e23a42"
Sep 10 20:38:23.198531 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:38:23.198504    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43dd900a020a1dc3a73351f3346db55d43b6441a45845e29dfbc0d1856e23a42\": container with ID starting with 43dd900a020a1dc3a73351f3346db55d43b6441a45845e29dfbc0d1856e23a42 not found: ID does not exist" containerID="43dd900a020a1dc3a73351f3346db55d43b6441a45845e29dfbc0d1856e23a42"
Sep 10 20:38:23.198615 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:23.198571    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43dd900a020a1dc3a73351f3346db55d43b6441a45845e29dfbc0d1856e23a42"} err="failed to get container status \"43dd900a020a1dc3a73351f3346db55d43b6441a45845e29dfbc0d1856e23a42\": rpc error: code = NotFound desc = could not find container \"43dd900a020a1dc3a73351f3346db55d43b6441a45845e29dfbc0d1856e23a42\": container with ID starting with 43dd900a020a1dc3a73351f3346db55d43b6441a45845e29dfbc0d1856e23a42 not found: ID does not exist"
Sep 10 20:38:23.256292 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:23.256271    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1d5ba49b-5e81-426b-b8b0-ba260f754c14-ca-trust-extracted\") pod \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") "
Sep 10 20:38:23.256395 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:23.256297    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d5ba49b-5e81-426b-b8b0-ba260f754c14-trusted-ca\") pod \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") "
Sep 10 20:38:23.256395 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:23.256317    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1d5ba49b-5e81-426b-b8b0-ba260f754c14-registry-certificates\") pod \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") "
Sep 10 20:38:23.256395 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:23.256342    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2j5x\" (UniqueName: \"kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-kube-api-access-j2j5x\") pod \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") "
Sep 10 20:38:23.256395 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:23.256379    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-registry-tls\") pod \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") "
Sep 10 20:38:23.256596 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:23.256433    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1d5ba49b-5e81-426b-b8b0-ba260f754c14-image-registry-private-configuration\") pod \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") "
Sep 10 20:38:23.256596 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:23.256478    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1d5ba49b-5e81-426b-b8b0-ba260f754c14-installation-pull-secrets\") pod \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") "
Sep 10 20:38:23.256596 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:23.256503    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-bound-sa-token\") pod \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\" (UID: \"1d5ba49b-5e81-426b-b8b0-ba260f754c14\") "
Sep 10 20:38:23.256746 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:23.256721    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d5ba49b-5e81-426b-b8b0-ba260f754c14-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1d5ba49b-5e81-426b-b8b0-ba260f754c14" (UID: "1d5ba49b-5e81-426b-b8b0-ba260f754c14"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:38:23.256746 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:23.256737    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d5ba49b-5e81-426b-b8b0-ba260f754c14-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1d5ba49b-5e81-426b-b8b0-ba260f754c14" (UID: "1d5ba49b-5e81-426b-b8b0-ba260f754c14"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:38:23.258935 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:23.258905    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1d5ba49b-5e81-426b-b8b0-ba260f754c14" (UID: "1d5ba49b-5e81-426b-b8b0-ba260f754c14"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:38:23.259052 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:23.258910    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1d5ba49b-5e81-426b-b8b0-ba260f754c14" (UID: "1d5ba49b-5e81-426b-b8b0-ba260f754c14"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:38:23.259112 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:23.259054    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d5ba49b-5e81-426b-b8b0-ba260f754c14-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1d5ba49b-5e81-426b-b8b0-ba260f754c14" (UID: "1d5ba49b-5e81-426b-b8b0-ba260f754c14"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:38:23.259112 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:23.259085    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-kube-api-access-j2j5x" (OuterVolumeSpecName: "kube-api-access-j2j5x") pod "1d5ba49b-5e81-426b-b8b0-ba260f754c14" (UID: "1d5ba49b-5e81-426b-b8b0-ba260f754c14"). InnerVolumeSpecName "kube-api-access-j2j5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:38:23.259236 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:23.259212    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d5ba49b-5e81-426b-b8b0-ba260f754c14-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1d5ba49b-5e81-426b-b8b0-ba260f754c14" (UID: "1d5ba49b-5e81-426b-b8b0-ba260f754c14"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:38:23.264993 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:23.264970    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d5ba49b-5e81-426b-b8b0-ba260f754c14-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1d5ba49b-5e81-426b-b8b0-ba260f754c14" (UID: "1d5ba49b-5e81-426b-b8b0-ba260f754c14"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:38:23.357671 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:23.357646    2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1d5ba49b-5e81-426b-b8b0-ba260f754c14-ca-trust-extracted\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:38:23.357671 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:23.357668    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d5ba49b-5e81-426b-b8b0-ba260f754c14-trusted-ca\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:38:23.357782 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:23.357677    2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1d5ba49b-5e81-426b-b8b0-ba260f754c14-registry-certificates\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:38:23.357782 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:23.357686    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2j5x\" (UniqueName: \"kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-kube-api-access-j2j5x\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:38:23.357782 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:23.357694    2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-registry-tls\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:38:23.357782 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:23.357703    2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1d5ba49b-5e81-426b-b8b0-ba260f754c14-image-registry-private-configuration\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:38:23.357782 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:23.357712    2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1d5ba49b-5e81-426b-b8b0-ba260f754c14-installation-pull-secrets\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:38:23.357782 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:23.357721    2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d5ba49b-5e81-426b-b8b0-ba260f754c14-bound-sa-token\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:38:23.512573 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:23.512547    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67bfdbc8bd-nprqr"]
Sep 10 20:38:23.520986 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:23.520962    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-67bfdbc8bd-nprqr"]
Sep 10 20:38:24.539252 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:38:24.539219    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d5ba49b-5e81-426b-b8b0-ba260f754c14" path="/var/lib/kubelet/pods/1d5ba49b-5e81-426b-b8b0-ba260f754c14/volumes"
Sep 10 20:41:00.454001 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:41:00.453964    2564 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 20:43:06.738344 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:06.738305    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xx7gn"]
Sep 10 20:43:06.738776 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:06.738553    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d5ba49b-5e81-426b-b8b0-ba260f754c14" containerName="registry"
Sep 10 20:43:06.738776 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:06.738565    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d5ba49b-5e81-426b-b8b0-ba260f754c14" containerName="registry"
Sep 10 20:43:06.738776 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:06.738618    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d5ba49b-5e81-426b-b8b0-ba260f754c14" containerName="registry"
Sep 10 20:43:06.741576 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:06.741551    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xx7gn"
Sep 10 20:43:06.743628 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:06.743604    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-pfn85\""
Sep 10 20:43:06.743628 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:06.743617    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 20:43:06.744260 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:06.744238    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 20:43:06.755254 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:06.754808    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xx7gn"]
Sep 10 20:43:06.862762 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:06.862733    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d44a3c4-0b97-40f0-8a1b-e514654ec96c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xx7gn\" (UID: \"2d44a3c4-0b97-40f0-8a1b-e514654ec96c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xx7gn"
Sep 10 20:43:06.862872 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:06.862769    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-855rt\" (UniqueName: \"kubernetes.io/projected/2d44a3c4-0b97-40f0-8a1b-e514654ec96c-kube-api-access-855rt\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xx7gn\" (UID: \"2d44a3c4-0b97-40f0-8a1b-e514654ec96c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xx7gn"
Sep 10 20:43:06.862872 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:06.862792    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d44a3c4-0b97-40f0-8a1b-e514654ec96c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xx7gn\" (UID: \"2d44a3c4-0b97-40f0-8a1b-e514654ec96c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xx7gn"
Sep 10 20:43:06.963282 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:06.963246    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d44a3c4-0b97-40f0-8a1b-e514654ec96c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xx7gn\" (UID: \"2d44a3c4-0b97-40f0-8a1b-e514654ec96c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xx7gn"
Sep 10 20:43:06.963376 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:06.963292    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-855rt\" (UniqueName: \"kubernetes.io/projected/2d44a3c4-0b97-40f0-8a1b-e514654ec96c-kube-api-access-855rt\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xx7gn\" (UID: \"2d44a3c4-0b97-40f0-8a1b-e514654ec96c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xx7gn"
Sep 10 20:43:06.963376 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:06.963322    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d44a3c4-0b97-40f0-8a1b-e514654ec96c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xx7gn\" (UID: \"2d44a3c4-0b97-40f0-8a1b-e514654ec96c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xx7gn"
Sep 10 20:43:06.963608 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:06.963588    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d44a3c4-0b97-40f0-8a1b-e514654ec96c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xx7gn\" (UID: \"2d44a3c4-0b97-40f0-8a1b-e514654ec96c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xx7gn"
Sep 10 20:43:06.963644 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:06.963621    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d44a3c4-0b97-40f0-8a1b-e514654ec96c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xx7gn\" (UID: \"2d44a3c4-0b97-40f0-8a1b-e514654ec96c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xx7gn"
Sep 10 20:43:06.972544 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:06.972521    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-855rt\" (UniqueName: \"kubernetes.io/projected/2d44a3c4-0b97-40f0-8a1b-e514654ec96c-kube-api-access-855rt\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xx7gn\" (UID: \"2d44a3c4-0b97-40f0-8a1b-e514654ec96c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xx7gn"
Sep 10 20:43:07.052799 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:07.052741    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xx7gn"
Sep 10 20:43:07.327681 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:07.327577    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xx7gn"]
Sep 10 20:43:07.330658 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:43:07.330621    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d44a3c4_0b97_40f0_8a1b_e514654ec96c.slice/crio-36d7d90558fbab8d1078c4aebf72aa7d7d30ca9691ed6568cd207d6bfc5610a1 WatchSource:0}: Error finding container 36d7d90558fbab8d1078c4aebf72aa7d7d30ca9691ed6568cd207d6bfc5610a1: Status 404 returned error can't find the container with id 36d7d90558fbab8d1078c4aebf72aa7d7d30ca9691ed6568cd207d6bfc5610a1
Sep 10 20:43:07.332619 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:07.332603    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 20:43:08.154255 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:08.154215    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xx7gn" event={"ID":"2d44a3c4-0b97-40f0-8a1b-e514654ec96c","Type":"ContainerStarted","Data":"36d7d90558fbab8d1078c4aebf72aa7d7d30ca9691ed6568cd207d6bfc5610a1"}
Sep 10 20:43:13.174739 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:13.174704    2564 generic.go:356] "Generic (PLEG): container finished" podID="2d44a3c4-0b97-40f0-8a1b-e514654ec96c" containerID="81b48cdca71c629836306860b8b741a8ecb0628c84f117f586f52eff0f59a4a6" exitCode=0
Sep 10 20:43:13.175135 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:13.174794    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xx7gn" event={"ID":"2d44a3c4-0b97-40f0-8a1b-e514654ec96c","Type":"ContainerDied","Data":"81b48cdca71c629836306860b8b741a8ecb0628c84f117f586f52eff0f59a4a6"}
Sep 10 20:43:18.194575 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:18.194541    2564 generic.go:356] "Generic (PLEG): container finished" podID="2d44a3c4-0b97-40f0-8a1b-e514654ec96c" containerID="80db1626ed0b4451f8d6468bb03d00d5f86cbb06baf95c8e3f12d822cec25a20" exitCode=0
Sep 10 20:43:18.195033 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:18.194596    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xx7gn" event={"ID":"2d44a3c4-0b97-40f0-8a1b-e514654ec96c","Type":"ContainerDied","Data":"80db1626ed0b4451f8d6468bb03d00d5f86cbb06baf95c8e3f12d822cec25a20"}
Sep 10 20:43:27.229675 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:27.229634    2564 generic.go:356] "Generic (PLEG): container finished" podID="2d44a3c4-0b97-40f0-8a1b-e514654ec96c" containerID="6b699e63933b4f74fb01778ada20d9a50ea601941bb70353cbe4ed264e066e89" exitCode=0
Sep 10 20:43:27.230078 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:27.229721    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xx7gn" event={"ID":"2d44a3c4-0b97-40f0-8a1b-e514654ec96c","Type":"ContainerDied","Data":"6b699e63933b4f74fb01778ada20d9a50ea601941bb70353cbe4ed264e066e89"}
Sep 10 20:43:28.361529 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:28.361505    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xx7gn"
Sep 10 20:43:28.507400 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:28.507349    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-855rt\" (UniqueName: \"kubernetes.io/projected/2d44a3c4-0b97-40f0-8a1b-e514654ec96c-kube-api-access-855rt\") pod \"2d44a3c4-0b97-40f0-8a1b-e514654ec96c\" (UID: \"2d44a3c4-0b97-40f0-8a1b-e514654ec96c\") "
Sep 10 20:43:28.507497 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:28.507417    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d44a3c4-0b97-40f0-8a1b-e514654ec96c-bundle\") pod \"2d44a3c4-0b97-40f0-8a1b-e514654ec96c\" (UID: \"2d44a3c4-0b97-40f0-8a1b-e514654ec96c\") "
Sep 10 20:43:28.507497 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:28.507440    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d44a3c4-0b97-40f0-8a1b-e514654ec96c-util\") pod \"2d44a3c4-0b97-40f0-8a1b-e514654ec96c\" (UID: \"2d44a3c4-0b97-40f0-8a1b-e514654ec96c\") "
Sep 10 20:43:28.507979 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:28.507952    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d44a3c4-0b97-40f0-8a1b-e514654ec96c-bundle" (OuterVolumeSpecName: "bundle") pod "2d44a3c4-0b97-40f0-8a1b-e514654ec96c" (UID: "2d44a3c4-0b97-40f0-8a1b-e514654ec96c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:43:28.509444 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:28.509411    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d44a3c4-0b97-40f0-8a1b-e514654ec96c-kube-api-access-855rt" (OuterVolumeSpecName: "kube-api-access-855rt") pod "2d44a3c4-0b97-40f0-8a1b-e514654ec96c" (UID: "2d44a3c4-0b97-40f0-8a1b-e514654ec96c"). InnerVolumeSpecName "kube-api-access-855rt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:43:28.511586 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:28.511563    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d44a3c4-0b97-40f0-8a1b-e514654ec96c-util" (OuterVolumeSpecName: "util") pod "2d44a3c4-0b97-40f0-8a1b-e514654ec96c" (UID: "2d44a3c4-0b97-40f0-8a1b-e514654ec96c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:43:28.608609 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:28.608584    2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d44a3c4-0b97-40f0-8a1b-e514654ec96c-bundle\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:43:28.608609 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:28.608608    2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d44a3c4-0b97-40f0-8a1b-e514654ec96c-util\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:43:28.608725 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:28.608623    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-855rt\" (UniqueName: \"kubernetes.io/projected/2d44a3c4-0b97-40f0-8a1b-e514654ec96c-kube-api-access-855rt\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:43:29.239133 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:29.239095    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xx7gn" event={"ID":"2d44a3c4-0b97-40f0-8a1b-e514654ec96c","Type":"ContainerDied","Data":"36d7d90558fbab8d1078c4aebf72aa7d7d30ca9691ed6568cd207d6bfc5610a1"}
Sep 10 20:43:29.239323 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:29.239148    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36d7d90558fbab8d1078c4aebf72aa7d7d30ca9691ed6568cd207d6bfc5610a1"
Sep 10 20:43:29.239323 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:29.239148    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xx7gn"
Sep 10 20:43:35.059404 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:35.059365    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pgdc2"]
Sep 10 20:43:35.059887 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:35.059621    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d44a3c4-0b97-40f0-8a1b-e514654ec96c" containerName="util"
Sep 10 20:43:35.059887 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:35.059633    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d44a3c4-0b97-40f0-8a1b-e514654ec96c" containerName="util"
Sep 10 20:43:35.059887 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:35.059646    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d44a3c4-0b97-40f0-8a1b-e514654ec96c" containerName="pull"
Sep 10 20:43:35.059887 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:35.059651    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d44a3c4-0b97-40f0-8a1b-e514654ec96c" containerName="pull"
Sep 10 20:43:35.059887 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:35.059659    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d44a3c4-0b97-40f0-8a1b-e514654ec96c" containerName="extract"
Sep 10 20:43:35.059887 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:35.059664    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d44a3c4-0b97-40f0-8a1b-e514654ec96c" containerName="extract"
Sep 10 20:43:35.059887 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:35.059717    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d44a3c4-0b97-40f0-8a1b-e514654ec96c" containerName="extract"
Sep 10 20:43:35.064886 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:35.064861    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pgdc2"
Sep 10 20:43:35.067517 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:35.067487    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 10 20:43:35.067641 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:35.067561    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 10 20:43:35.067641 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:35.067567    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-944j6\""
Sep 10 20:43:35.077242 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:35.077216    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pgdc2"]
Sep 10 20:43:35.148584 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:35.148561    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/475cf380-9849-4ade-8e26-ffae473fffbc-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-pgdc2\" (UID: \"475cf380-9849-4ade-8e26-ffae473fffbc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pgdc2"
Sep 10 20:43:35.148691 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:35.148599    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2kgs\" (UniqueName: \"kubernetes.io/projected/475cf380-9849-4ade-8e26-ffae473fffbc-kube-api-access-p2kgs\") pod \"cert-manager-operator-controller-manager-55c444bf86-pgdc2\" (UID: \"475cf380-9849-4ade-8e26-ffae473fffbc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pgdc2"
Sep 10 20:43:35.249883 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:35.249847    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/475cf380-9849-4ade-8e26-ffae473fffbc-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-pgdc2\" (UID: \"475cf380-9849-4ade-8e26-ffae473fffbc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pgdc2"
Sep 10 20:43:35.249983 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:35.249900    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2kgs\" (UniqueName: \"kubernetes.io/projected/475cf380-9849-4ade-8e26-ffae473fffbc-kube-api-access-p2kgs\") pod \"cert-manager-operator-controller-manager-55c444bf86-pgdc2\" (UID: \"475cf380-9849-4ade-8e26-ffae473fffbc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pgdc2"
Sep 10 20:43:35.250285 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:35.250260    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/475cf380-9849-4ade-8e26-ffae473fffbc-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-pgdc2\" (UID: \"475cf380-9849-4ade-8e26-ffae473fffbc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pgdc2"
Sep 10 20:43:35.261337 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:35.261312    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2kgs\" (UniqueName: \"kubernetes.io/projected/475cf380-9849-4ade-8e26-ffae473fffbc-kube-api-access-p2kgs\") pod \"cert-manager-operator-controller-manager-55c444bf86-pgdc2\" (UID: \"475cf380-9849-4ade-8e26-ffae473fffbc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pgdc2"
Sep 10 20:43:35.376365 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:35.376342    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pgdc2"
Sep 10 20:43:35.654135 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:35.654066    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pgdc2"]
Sep 10 20:43:35.657541 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:43:35.657510    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod475cf380_9849_4ade_8e26_ffae473fffbc.slice/crio-dd0f6c2175a809968cde8db43ef3a038777da784014c133e8a6ecb7e62685738 WatchSource:0}: Error finding container dd0f6c2175a809968cde8db43ef3a038777da784014c133e8a6ecb7e62685738: Status 404 returned error can't find the container with id dd0f6c2175a809968cde8db43ef3a038777da784014c133e8a6ecb7e62685738
Sep 10 20:43:36.264883 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:36.264847    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pgdc2" event={"ID":"475cf380-9849-4ade-8e26-ffae473fffbc","Type":"ContainerStarted","Data":"dd0f6c2175a809968cde8db43ef3a038777da784014c133e8a6ecb7e62685738"}
Sep 10 20:43:44.295124 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:44.295091    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pgdc2" event={"ID":"475cf380-9849-4ade-8e26-ffae473fffbc","Type":"ContainerStarted","Data":"db375be3e85d32ac8632a0fc5f6f004074dcdbc03768f6a29d089cdaed9f1461"}
Sep 10 20:43:44.316833 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:44.316792    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pgdc2" podStartSLOduration=0.815005916 podStartE2EDuration="9.31677568s" totalImagesPullingTime="8.501769764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:43:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:43:44.315546336 +0000 UTC m=+464.280614531" watchObservedRunningTime="2026-09-10 20:43:44.31677568 +0000 UTC m=+464.281843876"
Sep 10 20:43:45.219537 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:45.219500    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-w89tz"]
Sep 10 20:43:45.222683 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:45.222661    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-w89tz"
Sep 10 20:43:45.225576 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:45.225555    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 10 20:43:45.225672 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:45.225566    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-dnk6z\""
Sep 10 20:43:45.226111 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:45.226097    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 10 20:43:45.232826 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:45.232804    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-w89tz"]
Sep 10 20:43:45.312692 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:45.312670    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrvxp\" (UniqueName: \"kubernetes.io/projected/c5c64381-b121-47be-bd1a-ed2730378438-kube-api-access-qrvxp\") pod \"cert-manager-webhook-6486ffd997-w89tz\" (UID: \"c5c64381-b121-47be-bd1a-ed2730378438\") " pod="cert-manager/cert-manager-webhook-6486ffd997-w89tz"
Sep 10 20:43:45.312957 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:45.312697    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c5c64381-b121-47be-bd1a-ed2730378438-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-w89tz\" (UID: \"c5c64381-b121-47be-bd1a-ed2730378438\") " pod="cert-manager/cert-manager-webhook-6486ffd997-w89tz"
Sep 10 20:43:45.413259 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:45.413232    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrvxp\" (UniqueName: \"kubernetes.io/projected/c5c64381-b121-47be-bd1a-ed2730378438-kube-api-access-qrvxp\") pod \"cert-manager-webhook-6486ffd997-w89tz\" (UID: \"c5c64381-b121-47be-bd1a-ed2730378438\") " pod="cert-manager/cert-manager-webhook-6486ffd997-w89tz"
Sep 10 20:43:45.413359 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:45.413260    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c5c64381-b121-47be-bd1a-ed2730378438-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-w89tz\" (UID: \"c5c64381-b121-47be-bd1a-ed2730378438\") " pod="cert-manager/cert-manager-webhook-6486ffd997-w89tz"
Sep 10 20:43:45.423047 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:45.423022    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrvxp\" (UniqueName: \"kubernetes.io/projected/c5c64381-b121-47be-bd1a-ed2730378438-kube-api-access-qrvxp\") pod \"cert-manager-webhook-6486ffd997-w89tz\" (UID: \"c5c64381-b121-47be-bd1a-ed2730378438\") " pod="cert-manager/cert-manager-webhook-6486ffd997-w89tz"
Sep 10 20:43:45.427563 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:45.427534    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c5c64381-b121-47be-bd1a-ed2730378438-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-w89tz\" (UID: \"c5c64381-b121-47be-bd1a-ed2730378438\") " pod="cert-manager/cert-manager-webhook-6486ffd997-w89tz"
Sep 10 20:43:45.531684 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:45.531622    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-w89tz"
Sep 10 20:43:45.710366 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:45.710338    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzqqft"]
Sep 10 20:43:45.713666 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:45.713648    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzqqft"
Sep 10 20:43:45.716264 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:45.716244    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-pfn85\""
Sep 10 20:43:45.716657 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:45.716633    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 20:43:45.716657 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:45.716647    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 20:43:45.727272 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:45.727249    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzqqft"]
Sep 10 20:43:45.813398 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:45.813103    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-w89tz"]
Sep 10 20:43:45.815814 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:45.815794    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efccca58-0c71-4dd2-b175-76110d330e5e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzqqft\" (UID: \"efccca58-0c71-4dd2-b175-76110d330e5e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzqqft"
Sep 10 20:43:45.815885 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:45.815834    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h8pj\" (UniqueName: \"kubernetes.io/projected/efccca58-0c71-4dd2-b175-76110d330e5e-kube-api-access-7h8pj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzqqft\" (UID: \"efccca58-0c71-4dd2-b175-76110d330e5e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzqqft"
Sep 10 20:43:45.815885 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:45.815856    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efccca58-0c71-4dd2-b175-76110d330e5e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzqqft\" (UID: \"efccca58-0c71-4dd2-b175-76110d330e5e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzqqft"
Sep 10 20:43:45.817110 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:43:45.817087    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5c64381_b121_47be_bd1a_ed2730378438.slice/crio-9b568a6ee8f601ecf79e7a1c6cf2dcc73d9267844c99d0c030614c11ed27a2ef WatchSource:0}: Error finding container 9b568a6ee8f601ecf79e7a1c6cf2dcc73d9267844c99d0c030614c11ed27a2ef: Status 404 returned error can't find the container with id 9b568a6ee8f601ecf79e7a1c6cf2dcc73d9267844c99d0c030614c11ed27a2ef
Sep 10 20:43:45.916244 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:45.916221    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efccca58-0c71-4dd2-b175-76110d330e5e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzqqft\" (UID: \"efccca58-0c71-4dd2-b175-76110d330e5e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzqqft"
Sep 10 20:43:45.916354 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:45.916256    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h8pj\" (UniqueName: \"kubernetes.io/projected/efccca58-0c71-4dd2-b175-76110d330e5e-kube-api-access-7h8pj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzqqft\" (UID: \"efccca58-0c71-4dd2-b175-76110d330e5e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzqqft"
Sep 10 20:43:45.916354 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:45.916283    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efccca58-0c71-4dd2-b175-76110d330e5e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzqqft\" (UID: \"efccca58-0c71-4dd2-b175-76110d330e5e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzqqft"
Sep 10 20:43:45.916617 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:45.916597    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efccca58-0c71-4dd2-b175-76110d330e5e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzqqft\" (UID: \"efccca58-0c71-4dd2-b175-76110d330e5e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzqqft"
Sep 10 20:43:45.916669 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:45.916627    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efccca58-0c71-4dd2-b175-76110d330e5e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzqqft\" (UID: \"efccca58-0c71-4dd2-b175-76110d330e5e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzqqft"
Sep 10 20:43:45.927380 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:45.927362    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h8pj\" (UniqueName: \"kubernetes.io/projected/efccca58-0c71-4dd2-b175-76110d330e5e-kube-api-access-7h8pj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzqqft\" (UID: \"efccca58-0c71-4dd2-b175-76110d330e5e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzqqft"
Sep 10 20:43:46.023011 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:46.022987    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzqqft"
Sep 10 20:43:46.292808 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:46.292720    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzqqft"]
Sep 10 20:43:46.295203 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:43:46.295157    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefccca58_0c71_4dd2_b175_76110d330e5e.slice/crio-5dc58d2fb8fe21bed76f0608ffeb68991a8ebfe2907cc291fe317e9a5a1799eb WatchSource:0}: Error finding container 5dc58d2fb8fe21bed76f0608ffeb68991a8ebfe2907cc291fe317e9a5a1799eb: Status 404 returned error can't find the container with id 5dc58d2fb8fe21bed76f0608ffeb68991a8ebfe2907cc291fe317e9a5a1799eb
Sep 10 20:43:46.303962 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:46.303933    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzqqft" event={"ID":"efccca58-0c71-4dd2-b175-76110d330e5e","Type":"ContainerStarted","Data":"5dc58d2fb8fe21bed76f0608ffeb68991a8ebfe2907cc291fe317e9a5a1799eb"}
Sep 10 20:43:46.305103 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:46.305080    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-w89tz" event={"ID":"c5c64381-b121-47be-bd1a-ed2730378438","Type":"ContainerStarted","Data":"9b568a6ee8f601ecf79e7a1c6cf2dcc73d9267844c99d0c030614c11ed27a2ef"}
Sep 10 20:43:47.310767 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:47.310730    2564 generic.go:356] "Generic (PLEG): container finished" podID="efccca58-0c71-4dd2-b175-76110d330e5e" containerID="711e9e25fd973278ada9c12f90fc12501bbb84311d1acc723bb2be76a8f5a818" exitCode=0
Sep 10 20:43:47.311160 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:47.310771    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzqqft" event={"ID":"efccca58-0c71-4dd2-b175-76110d330e5e","Type":"ContainerDied","Data":"711e9e25fd973278ada9c12f90fc12501bbb84311d1acc723bb2be76a8f5a818"}
Sep 10 20:43:49.319805 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:49.319766    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-w89tz" event={"ID":"c5c64381-b121-47be-bd1a-ed2730378438","Type":"ContainerStarted","Data":"245fe5798e757f4ffd441dc2bf7bebe4ae2d37f663097156e35064201bfa9467"}
Sep 10 20:43:49.320216 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:49.319853    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-w89tz"
Sep 10 20:43:50.326164 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:50.326126    2564 generic.go:356] "Generic (PLEG): container finished" podID="efccca58-0c71-4dd2-b175-76110d330e5e" containerID="90f94a3af0ef3b5a46a9d63c627131a0d9c1fb747aa5ec765b6bf5f35494a464" exitCode=0
Sep 10 20:43:50.326512 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:50.326224    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzqqft" event={"ID":"efccca58-0c71-4dd2-b175-76110d330e5e","Type":"ContainerDied","Data":"90f94a3af0ef3b5a46a9d63c627131a0d9c1fb747aa5ec765b6bf5f35494a464"}
Sep 10 20:43:50.344082 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:50.344021    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-w89tz" podStartSLOduration=2.634305704 podStartE2EDuration="5.344004817s" totalImagesPullingTime="2.709699113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:43:49.340648434 +0000 UTC m=+469.305716629" watchObservedRunningTime="2026-09-10 20:43:50.344004817 +0000 UTC m=+470.309073010"
Sep 10 20:43:51.332507 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:51.332467    2564 generic.go:356] "Generic (PLEG): container finished" podID="efccca58-0c71-4dd2-b175-76110d330e5e" containerID="aa13ea1e38e15c09e62759e4f370a916e485c39e9cb0a263e16b5784e59c2a5b" exitCode=0
Sep 10 20:43:51.332891 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:51.332541    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzqqft" event={"ID":"efccca58-0c71-4dd2-b175-76110d330e5e","Type":"ContainerDied","Data":"aa13ea1e38e15c09e62759e4f370a916e485c39e9cb0a263e16b5784e59c2a5b"}
Sep 10 20:43:52.458012 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:52.457990    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzqqft"
Sep 10 20:43:52.564766 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:52.564741    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efccca58-0c71-4dd2-b175-76110d330e5e-util\") pod \"efccca58-0c71-4dd2-b175-76110d330e5e\" (UID: \"efccca58-0c71-4dd2-b175-76110d330e5e\") "
Sep 10 20:43:52.564901 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:52.564773    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efccca58-0c71-4dd2-b175-76110d330e5e-bundle\") pod \"efccca58-0c71-4dd2-b175-76110d330e5e\" (UID: \"efccca58-0c71-4dd2-b175-76110d330e5e\") "
Sep 10 20:43:52.564901 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:52.564792    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7h8pj\" (UniqueName: \"kubernetes.io/projected/efccca58-0c71-4dd2-b175-76110d330e5e-kube-api-access-7h8pj\") pod \"efccca58-0c71-4dd2-b175-76110d330e5e\" (UID: \"efccca58-0c71-4dd2-b175-76110d330e5e\") "
Sep 10 20:43:52.565134 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:52.565107    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efccca58-0c71-4dd2-b175-76110d330e5e-bundle" (OuterVolumeSpecName: "bundle") pod "efccca58-0c71-4dd2-b175-76110d330e5e" (UID: "efccca58-0c71-4dd2-b175-76110d330e5e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:43:52.566710 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:52.566677    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efccca58-0c71-4dd2-b175-76110d330e5e-kube-api-access-7h8pj" (OuterVolumeSpecName: "kube-api-access-7h8pj") pod "efccca58-0c71-4dd2-b175-76110d330e5e" (UID: "efccca58-0c71-4dd2-b175-76110d330e5e"). InnerVolumeSpecName "kube-api-access-7h8pj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:43:52.569556 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:52.569529    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efccca58-0c71-4dd2-b175-76110d330e5e-util" (OuterVolumeSpecName: "util") pod "efccca58-0c71-4dd2-b175-76110d330e5e" (UID: "efccca58-0c71-4dd2-b175-76110d330e5e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:43:52.665649 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:52.665596    2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efccca58-0c71-4dd2-b175-76110d330e5e-util\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:43:52.665649 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:52.665615    2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efccca58-0c71-4dd2-b175-76110d330e5e-bundle\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:43:52.665649 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:52.665625    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h8pj\" (UniqueName: \"kubernetes.io/projected/efccca58-0c71-4dd2-b175-76110d330e5e-kube-api-access-7h8pj\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:43:53.343279 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:53.343236    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzqqft" event={"ID":"efccca58-0c71-4dd2-b175-76110d330e5e","Type":"ContainerDied","Data":"5dc58d2fb8fe21bed76f0608ffeb68991a8ebfe2907cc291fe317e9a5a1799eb"}
Sep 10 20:43:53.343447 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:53.343299    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzqqft"
Sep 10 20:43:53.343447 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:53.343301    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5dc58d2fb8fe21bed76f0608ffeb68991a8ebfe2907cc291fe317e9a5a1799eb"
Sep 10 20:43:55.328936 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:43:55.328903    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-w89tz"
Sep 10 20:44:05.609492 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:05.609452    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-988jq"]
Sep 10 20:44:05.610023 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:05.609695    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efccca58-0c71-4dd2-b175-76110d330e5e" containerName="util"
Sep 10 20:44:05.610023 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:05.609706    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="efccca58-0c71-4dd2-b175-76110d330e5e" containerName="util"
Sep 10 20:44:05.610023 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:05.609719    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efccca58-0c71-4dd2-b175-76110d330e5e" containerName="pull"
Sep 10 20:44:05.610023 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:05.609725    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="efccca58-0c71-4dd2-b175-76110d330e5e" containerName="pull"
Sep 10 20:44:05.610023 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:05.609736    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efccca58-0c71-4dd2-b175-76110d330e5e" containerName="extract"
Sep 10 20:44:05.610023 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:05.609742    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="efccca58-0c71-4dd2-b175-76110d330e5e" containerName="extract"
Sep 10 20:44:05.610023 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:05.609788    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="efccca58-0c71-4dd2-b175-76110d330e5e" containerName="extract"
Sep 10 20:44:05.615197 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:05.615157    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-988jq"
Sep 10 20:44:05.617157 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:05.617123    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-qbphh\""
Sep 10 20:44:05.622991 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:05.622965    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-988jq"]
Sep 10 20:44:05.748679 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:05.748649    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hdvs\" (UniqueName: \"kubernetes.io/projected/2560edb6-a473-4c5f-b185-9b9e9e1ab63a-kube-api-access-8hdvs\") pod \"cert-manager-55b4d954d5-988jq\" (UID: \"2560edb6-a473-4c5f-b185-9b9e9e1ab63a\") " pod="cert-manager/cert-manager-55b4d954d5-988jq"
Sep 10 20:44:05.748792 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:05.748686    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2560edb6-a473-4c5f-b185-9b9e9e1ab63a-bound-sa-token\") pod \"cert-manager-55b4d954d5-988jq\" (UID: \"2560edb6-a473-4c5f-b185-9b9e9e1ab63a\") " pod="cert-manager/cert-manager-55b4d954d5-988jq"
Sep 10 20:44:05.849282 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:05.849256    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hdvs\" (UniqueName: \"kubernetes.io/projected/2560edb6-a473-4c5f-b185-9b9e9e1ab63a-kube-api-access-8hdvs\") pod \"cert-manager-55b4d954d5-988jq\" (UID: \"2560edb6-a473-4c5f-b185-9b9e9e1ab63a\") " pod="cert-manager/cert-manager-55b4d954d5-988jq"
Sep 10 20:44:05.849386 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:05.849290    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2560edb6-a473-4c5f-b185-9b9e9e1ab63a-bound-sa-token\") pod \"cert-manager-55b4d954d5-988jq\" (UID: \"2560edb6-a473-4c5f-b185-9b9e9e1ab63a\") " pod="cert-manager/cert-manager-55b4d954d5-988jq"
Sep 10 20:44:05.857872 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:05.857840    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2560edb6-a473-4c5f-b185-9b9e9e1ab63a-bound-sa-token\") pod \"cert-manager-55b4d954d5-988jq\" (UID: \"2560edb6-a473-4c5f-b185-9b9e9e1ab63a\") " pod="cert-manager/cert-manager-55b4d954d5-988jq"
Sep 10 20:44:05.858026 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:05.858004    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hdvs\" (UniqueName: \"kubernetes.io/projected/2560edb6-a473-4c5f-b185-9b9e9e1ab63a-kube-api-access-8hdvs\") pod \"cert-manager-55b4d954d5-988jq\" (UID: \"2560edb6-a473-4c5f-b185-9b9e9e1ab63a\") " pod="cert-manager/cert-manager-55b4d954d5-988jq"
Sep 10 20:44:05.925317 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:05.925258    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-988jq"
Sep 10 20:44:06.196142 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:06.196059    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-988jq"]
Sep 10 20:44:06.199023 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:44:06.198995    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2560edb6_a473_4c5f_b185_9b9e9e1ab63a.slice/crio-48d2d6d927b39e3f79e9939d03224ae9f0ce7052d742f944d08b487344d7b861 WatchSource:0}: Error finding container 48d2d6d927b39e3f79e9939d03224ae9f0ce7052d742f944d08b487344d7b861: Status 404 returned error can't find the container with id 48d2d6d927b39e3f79e9939d03224ae9f0ce7052d742f944d08b487344d7b861
Sep 10 20:44:06.396189 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:06.396143    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-988jq" event={"ID":"2560edb6-a473-4c5f-b185-9b9e9e1ab63a","Type":"ContainerStarted","Data":"cb60b35fadf7a5b3d5af52665a030d6e512c7e8bb5471243b8166af59c31fd7a"}
Sep 10 20:44:06.396324 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:06.396212    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-988jq" event={"ID":"2560edb6-a473-4c5f-b185-9b9e9e1ab63a","Type":"ContainerStarted","Data":"48d2d6d927b39e3f79e9939d03224ae9f0ce7052d742f944d08b487344d7b861"}
Sep 10 20:44:06.413777 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:06.413731    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-988jq" podStartSLOduration=1.4137167 podStartE2EDuration="1.4137167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:44:06.41245448 +0000 UTC m=+486.377522674" watchObservedRunningTime="2026-09-10 20:44:06.4137167 +0000 UTC m=+486.378784895"
Sep 10 20:44:19.143669 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:19.143629    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af522br"]
Sep 10 20:44:19.146968 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:19.146948    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af522br"
Sep 10 20:44:19.149275 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:19.149245    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 20:44:19.149414 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:19.149353    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 20:44:19.149931 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:19.149911    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-pfn85\""
Sep 10 20:44:19.157000 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:19.156972    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af522br"]
Sep 10 20:44:19.236362 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:19.236331    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ade2c67-a0da-454c-878a-ecf74fd1e2b8-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af522br\" (UID: \"2ade2c67-a0da-454c-878a-ecf74fd1e2b8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af522br"
Sep 10 20:44:19.236523 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:19.236383    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ade2c67-a0da-454c-878a-ecf74fd1e2b8-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af522br\" (UID: \"2ade2c67-a0da-454c-878a-ecf74fd1e2b8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af522br"
Sep 10 20:44:19.236523 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:19.236437    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djh7b\" (UniqueName: \"kubernetes.io/projected/2ade2c67-a0da-454c-878a-ecf74fd1e2b8-kube-api-access-djh7b\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af522br\" (UID: \"2ade2c67-a0da-454c-878a-ecf74fd1e2b8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af522br"
Sep 10 20:44:19.337639 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:19.337609    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djh7b\" (UniqueName: \"kubernetes.io/projected/2ade2c67-a0da-454c-878a-ecf74fd1e2b8-kube-api-access-djh7b\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af522br\" (UID: \"2ade2c67-a0da-454c-878a-ecf74fd1e2b8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af522br"
Sep 10 20:44:19.337731 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:19.337661    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ade2c67-a0da-454c-878a-ecf74fd1e2b8-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af522br\" (UID: \"2ade2c67-a0da-454c-878a-ecf74fd1e2b8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af522br"
Sep 10 20:44:19.337799 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:19.337709    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ade2c67-a0da-454c-878a-ecf74fd1e2b8-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af522br\" (UID: \"2ade2c67-a0da-454c-878a-ecf74fd1e2b8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af522br"
Sep 10 20:44:19.338112 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:19.338087    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ade2c67-a0da-454c-878a-ecf74fd1e2b8-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af522br\" (UID: \"2ade2c67-a0da-454c-878a-ecf74fd1e2b8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af522br"
Sep 10 20:44:19.338112 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:19.338104    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ade2c67-a0da-454c-878a-ecf74fd1e2b8-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af522br\" (UID: \"2ade2c67-a0da-454c-878a-ecf74fd1e2b8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af522br"
Sep 10 20:44:19.346843 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:19.346819    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djh7b\" (UniqueName: \"kubernetes.io/projected/2ade2c67-a0da-454c-878a-ecf74fd1e2b8-kube-api-access-djh7b\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af522br\" (UID: \"2ade2c67-a0da-454c-878a-ecf74fd1e2b8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af522br"
Sep 10 20:44:19.457377 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:19.457325    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af522br"
Sep 10 20:44:19.729007 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:19.728939    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af522br"]
Sep 10 20:44:19.731812 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:44:19.731784    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ade2c67_a0da_454c_878a_ecf74fd1e2b8.slice/crio-077522fa49709973e18794aeeb4d2b9c3176d6106c8c26f898f4393c4fc0a3b0 WatchSource:0}: Error finding container 077522fa49709973e18794aeeb4d2b9c3176d6106c8c26f898f4393c4fc0a3b0: Status 404 returned error can't find the container with id 077522fa49709973e18794aeeb4d2b9c3176d6106c8c26f898f4393c4fc0a3b0
Sep 10 20:44:20.456134 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:20.456100    2564 generic.go:356] "Generic (PLEG): container finished" podID="2ade2c67-a0da-454c-878a-ecf74fd1e2b8" containerID="6af6515bcb5174d9f69ee5b307b682623e5fb49a9604a0e645809ee1bdd41308" exitCode=0
Sep 10 20:44:20.456538 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:20.456206    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af522br" event={"ID":"2ade2c67-a0da-454c-878a-ecf74fd1e2b8","Type":"ContainerDied","Data":"6af6515bcb5174d9f69ee5b307b682623e5fb49a9604a0e645809ee1bdd41308"}
Sep 10 20:44:20.456538 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:20.456261    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af522br" event={"ID":"2ade2c67-a0da-454c-878a-ecf74fd1e2b8","Type":"ContainerStarted","Data":"077522fa49709973e18794aeeb4d2b9c3176d6106c8c26f898f4393c4fc0a3b0"}
Sep 10 20:44:21.462666 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:21.462591    2564 generic.go:356] "Generic (PLEG): container finished" podID="2ade2c67-a0da-454c-878a-ecf74fd1e2b8" containerID="eb207268b5e738d9309ab0b2f7899235a6c8f53be55ac2656aedd41e20014f3c" exitCode=0
Sep 10 20:44:21.463014 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:21.462682    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af522br" event={"ID":"2ade2c67-a0da-454c-878a-ecf74fd1e2b8","Type":"ContainerDied","Data":"eb207268b5e738d9309ab0b2f7899235a6c8f53be55ac2656aedd41e20014f3c"}
Sep 10 20:44:22.469835 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:22.469797    2564 generic.go:356] "Generic (PLEG): container finished" podID="2ade2c67-a0da-454c-878a-ecf74fd1e2b8" containerID="0439e3501304ab766fcc3c1b111f566e507ce14ce3f770db3c0befcd005ba3db" exitCode=0
Sep 10 20:44:22.470278 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:22.469885    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af522br" event={"ID":"2ade2c67-a0da-454c-878a-ecf74fd1e2b8","Type":"ContainerDied","Data":"0439e3501304ab766fcc3c1b111f566e507ce14ce3f770db3c0befcd005ba3db"}
Sep 10 20:44:23.595594 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:23.595568    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af522br"
Sep 10 20:44:23.766825 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:23.766762    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ade2c67-a0da-454c-878a-ecf74fd1e2b8-bundle\") pod \"2ade2c67-a0da-454c-878a-ecf74fd1e2b8\" (UID: \"2ade2c67-a0da-454c-878a-ecf74fd1e2b8\") "
Sep 10 20:44:23.766955 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:23.766824    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ade2c67-a0da-454c-878a-ecf74fd1e2b8-util\") pod \"2ade2c67-a0da-454c-878a-ecf74fd1e2b8\" (UID: \"2ade2c67-a0da-454c-878a-ecf74fd1e2b8\") "
Sep 10 20:44:23.766955 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:23.766851    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djh7b\" (UniqueName: \"kubernetes.io/projected/2ade2c67-a0da-454c-878a-ecf74fd1e2b8-kube-api-access-djh7b\") pod \"2ade2c67-a0da-454c-878a-ecf74fd1e2b8\" (UID: \"2ade2c67-a0da-454c-878a-ecf74fd1e2b8\") "
Sep 10 20:44:23.767523 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:23.767493    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ade2c67-a0da-454c-878a-ecf74fd1e2b8-bundle" (OuterVolumeSpecName: "bundle") pod "2ade2c67-a0da-454c-878a-ecf74fd1e2b8" (UID: "2ade2c67-a0da-454c-878a-ecf74fd1e2b8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:44:23.768852 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:23.768816    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ade2c67-a0da-454c-878a-ecf74fd1e2b8-kube-api-access-djh7b" (OuterVolumeSpecName: "kube-api-access-djh7b") pod "2ade2c67-a0da-454c-878a-ecf74fd1e2b8" (UID: "2ade2c67-a0da-454c-878a-ecf74fd1e2b8"). InnerVolumeSpecName "kube-api-access-djh7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:44:23.772000 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:23.771972    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ade2c67-a0da-454c-878a-ecf74fd1e2b8-util" (OuterVolumeSpecName: "util") pod "2ade2c67-a0da-454c-878a-ecf74fd1e2b8" (UID: "2ade2c67-a0da-454c-878a-ecf74fd1e2b8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:44:23.867805 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:23.867781    2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ade2c67-a0da-454c-878a-ecf74fd1e2b8-util\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:44:23.867805 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:23.867802    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djh7b\" (UniqueName: \"kubernetes.io/projected/2ade2c67-a0da-454c-878a-ecf74fd1e2b8-kube-api-access-djh7b\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:44:23.867926 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:23.867814    2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ade2c67-a0da-454c-878a-ecf74fd1e2b8-bundle\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:44:24.480650 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:24.480623    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af522br"
Sep 10 20:44:24.480650 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:24.480628    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af522br" event={"ID":"2ade2c67-a0da-454c-878a-ecf74fd1e2b8","Type":"ContainerDied","Data":"077522fa49709973e18794aeeb4d2b9c3176d6106c8c26f898f4393c4fc0a3b0"}
Sep 10 20:44:24.480833 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:24.480673    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="077522fa49709973e18794aeeb4d2b9c3176d6106c8c26f898f4393c4fc0a3b0"
Sep 10 20:44:28.943749 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:28.943708    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mq88p"]
Sep 10 20:44:28.944102 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:28.943970    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ade2c67-a0da-454c-878a-ecf74fd1e2b8" containerName="extract"
Sep 10 20:44:28.944102 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:28.943983    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ade2c67-a0da-454c-878a-ecf74fd1e2b8" containerName="extract"
Sep 10 20:44:28.944102 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:28.944001    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ade2c67-a0da-454c-878a-ecf74fd1e2b8" containerName="util"
Sep 10 20:44:28.944102 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:28.944011    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ade2c67-a0da-454c-878a-ecf74fd1e2b8" containerName="util"
Sep 10 20:44:28.944102 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:28.944028    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ade2c67-a0da-454c-878a-ecf74fd1e2b8" containerName="pull"
Sep 10 20:44:28.944102 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:28.944034    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ade2c67-a0da-454c-878a-ecf74fd1e2b8" containerName="pull"
Sep 10 20:44:28.944102 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:28.944073    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ade2c67-a0da-454c-878a-ecf74fd1e2b8" containerName="extract"
Sep 10 20:44:28.947138 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:28.947119    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mq88p"
Sep 10 20:44:28.949720 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:28.949691    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-pfn85\""
Sep 10 20:44:28.949958 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:28.949937    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 20:44:28.950450 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:28.950430    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 20:44:28.964461 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:28.964438    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mq88p"]
Sep 10 20:44:29.103526 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:29.103497    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c5fb7ad9-769e-48c2-96f0-d237e59c3be8-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mq88p\" (UID: \"c5fb7ad9-769e-48c2-96f0-d237e59c3be8\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mq88p"
Sep 10 20:44:29.103628 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:29.103545    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5fb7ad9-769e-48c2-96f0-d237e59c3be8-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mq88p\" (UID: \"c5fb7ad9-769e-48c2-96f0-d237e59c3be8\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mq88p"
Sep 10 20:44:29.103628 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:29.103572    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m88s2\" (UniqueName: \"kubernetes.io/projected/c5fb7ad9-769e-48c2-96f0-d237e59c3be8-kube-api-access-m88s2\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mq88p\" (UID: \"c5fb7ad9-769e-48c2-96f0-d237e59c3be8\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mq88p"
Sep 10 20:44:29.204869 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:29.204800    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5fb7ad9-769e-48c2-96f0-d237e59c3be8-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mq88p\" (UID: \"c5fb7ad9-769e-48c2-96f0-d237e59c3be8\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mq88p"
Sep 10 20:44:29.204869 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:29.204849    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m88s2\" (UniqueName: \"kubernetes.io/projected/c5fb7ad9-769e-48c2-96f0-d237e59c3be8-kube-api-access-m88s2\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mq88p\" (UID: \"c5fb7ad9-769e-48c2-96f0-d237e59c3be8\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mq88p"
Sep 10 20:44:29.205030 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:29.204886    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c5fb7ad9-769e-48c2-96f0-d237e59c3be8-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mq88p\" (UID: \"c5fb7ad9-769e-48c2-96f0-d237e59c3be8\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mq88p"
Sep 10 20:44:29.205269 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:29.205246    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5fb7ad9-769e-48c2-96f0-d237e59c3be8-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mq88p\" (UID: \"c5fb7ad9-769e-48c2-96f0-d237e59c3be8\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mq88p"
Sep 10 20:44:29.205359 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:29.205302    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c5fb7ad9-769e-48c2-96f0-d237e59c3be8-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mq88p\" (UID: \"c5fb7ad9-769e-48c2-96f0-d237e59c3be8\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mq88p"
Sep 10 20:44:29.213106 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:29.213081    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m88s2\" (UniqueName: \"kubernetes.io/projected/c5fb7ad9-769e-48c2-96f0-d237e59c3be8-kube-api-access-m88s2\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mq88p\" (UID: \"c5fb7ad9-769e-48c2-96f0-d237e59c3be8\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mq88p"
Sep 10 20:44:29.256703 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:29.256682    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mq88p"
Sep 10 20:44:29.551078 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:29.551052    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mq88p"]
Sep 10 20:44:29.553268 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:44:29.553239    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5fb7ad9_769e_48c2_96f0_d237e59c3be8.slice/crio-778adc421a68ad7e141c7e0050a109e50c343949fd2253ad4cd3f55d8dd6eb9e WatchSource:0}: Error finding container 778adc421a68ad7e141c7e0050a109e50c343949fd2253ad4cd3f55d8dd6eb9e: Status 404 returned error can't find the container with id 778adc421a68ad7e141c7e0050a109e50c343949fd2253ad4cd3f55d8dd6eb9e
Sep 10 20:44:30.514420 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:30.514333    2564 generic.go:356] "Generic (PLEG): container finished" podID="c5fb7ad9-769e-48c2-96f0-d237e59c3be8" containerID="edcc3522736a9a4c8407f65fad292aff931e0746fa02e0880bbdd1d5b167a5eb" exitCode=0
Sep 10 20:44:30.514420 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:30.514407    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mq88p" event={"ID":"c5fb7ad9-769e-48c2-96f0-d237e59c3be8","Type":"ContainerDied","Data":"edcc3522736a9a4c8407f65fad292aff931e0746fa02e0880bbdd1d5b167a5eb"}
Sep 10 20:44:30.514844 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:30.514489    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mq88p" event={"ID":"c5fb7ad9-769e-48c2-96f0-d237e59c3be8","Type":"ContainerStarted","Data":"778adc421a68ad7e141c7e0050a109e50c343949fd2253ad4cd3f55d8dd6eb9e"}
Sep 10 20:44:31.218040 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:31.218015    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-59df7859b6-8lrgz"]
Sep 10 20:44:31.222244 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:31.222224    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-8lrgz"
Sep 10 20:44:31.233846 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:31.233827    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 20:44:31.239188 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:31.239145    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 20:44:31.239284 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:31.239243    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 20:44:31.239383 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:31.239330    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-v6fqb\""
Sep 10 20:44:31.239383 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:31.239339    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 20:44:31.257890 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:31.257862    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-59df7859b6-8lrgz"]
Sep 10 20:44:31.318319 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:31.318298    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lhr2\" (UniqueName: \"kubernetes.io/projected/25cfc772-5c2c-4146-b2c4-952ae3f7c996-kube-api-access-6lhr2\") pod \"opendatahub-operator-controller-manager-59df7859b6-8lrgz\" (UID: \"25cfc772-5c2c-4146-b2c4-952ae3f7c996\") " pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-8lrgz"
Sep 10 20:44:31.318413 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:31.318335    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/25cfc772-5c2c-4146-b2c4-952ae3f7c996-apiservice-cert\") pod \"opendatahub-operator-controller-manager-59df7859b6-8lrgz\" (UID: \"25cfc772-5c2c-4146-b2c4-952ae3f7c996\") " pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-8lrgz"
Sep 10 20:44:31.318413 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:31.318367    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25cfc772-5c2c-4146-b2c4-952ae3f7c996-tmp\") pod \"opendatahub-operator-controller-manager-59df7859b6-8lrgz\" (UID: \"25cfc772-5c2c-4146-b2c4-952ae3f7c996\") " pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-8lrgz"
Sep 10 20:44:31.318479 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:31.318417    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/25cfc772-5c2c-4146-b2c4-952ae3f7c996-manifests\") pod \"opendatahub-operator-controller-manager-59df7859b6-8lrgz\" (UID: \"25cfc772-5c2c-4146-b2c4-952ae3f7c996\") " pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-8lrgz"
Sep 10 20:44:31.318479 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:31.318437    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/25cfc772-5c2c-4146-b2c4-952ae3f7c996-webhook-cert\") pod \"opendatahub-operator-controller-manager-59df7859b6-8lrgz\" (UID: \"25cfc772-5c2c-4146-b2c4-952ae3f7c996\") " pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-8lrgz"
Sep 10 20:44:31.419433 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:31.419408    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/25cfc772-5c2c-4146-b2c4-952ae3f7c996-manifests\") pod \"opendatahub-operator-controller-manager-59df7859b6-8lrgz\" (UID: \"25cfc772-5c2c-4146-b2c4-952ae3f7c996\") " pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-8lrgz"
Sep 10 20:44:31.419530 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:31.419442    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/25cfc772-5c2c-4146-b2c4-952ae3f7c996-webhook-cert\") pod \"opendatahub-operator-controller-manager-59df7859b6-8lrgz\" (UID: \"25cfc772-5c2c-4146-b2c4-952ae3f7c996\") " pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-8lrgz"
Sep 10 20:44:31.419530 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:31.419477    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lhr2\" (UniqueName: \"kubernetes.io/projected/25cfc772-5c2c-4146-b2c4-952ae3f7c996-kube-api-access-6lhr2\") pod \"opendatahub-operator-controller-manager-59df7859b6-8lrgz\" (UID: \"25cfc772-5c2c-4146-b2c4-952ae3f7c996\") " pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-8lrgz"
Sep 10 20:44:31.419530 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:31.419499    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/25cfc772-5c2c-4146-b2c4-952ae3f7c996-apiservice-cert\") pod \"opendatahub-operator-controller-manager-59df7859b6-8lrgz\" (UID: \"25cfc772-5c2c-4146-b2c4-952ae3f7c996\") " pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-8lrgz"
Sep 10 20:44:31.419530 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:31.419525    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25cfc772-5c2c-4146-b2c4-952ae3f7c996-tmp\") pod \"opendatahub-operator-controller-manager-59df7859b6-8lrgz\" (UID: \"25cfc772-5c2c-4146-b2c4-952ae3f7c996\") " pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-8lrgz"
Sep 10 20:44:31.419748 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:31.419728    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/25cfc772-5c2c-4146-b2c4-952ae3f7c996-manifests\") pod \"opendatahub-operator-controller-manager-59df7859b6-8lrgz\" (UID: \"25cfc772-5c2c-4146-b2c4-952ae3f7c996\") " pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-8lrgz"
Sep 10 20:44:31.421672 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:31.421651    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25cfc772-5c2c-4146-b2c4-952ae3f7c996-tmp\") pod \"opendatahub-operator-controller-manager-59df7859b6-8lrgz\" (UID: \"25cfc772-5c2c-4146-b2c4-952ae3f7c996\") " pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-8lrgz"
Sep 10 20:44:31.421767 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:31.421749    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/25cfc772-5c2c-4146-b2c4-952ae3f7c996-webhook-cert\") pod \"opendatahub-operator-controller-manager-59df7859b6-8lrgz\" (UID: \"25cfc772-5c2c-4146-b2c4-952ae3f7c996\") " pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-8lrgz"
Sep 10 20:44:31.421807 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:31.421774    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/25cfc772-5c2c-4146-b2c4-952ae3f7c996-apiservice-cert\") pod \"opendatahub-operator-controller-manager-59df7859b6-8lrgz\" (UID: \"25cfc772-5c2c-4146-b2c4-952ae3f7c996\") " pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-8lrgz"
Sep 10 20:44:31.431261 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:31.431233    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lhr2\" (UniqueName: \"kubernetes.io/projected/25cfc772-5c2c-4146-b2c4-952ae3f7c996-kube-api-access-6lhr2\") pod \"opendatahub-operator-controller-manager-59df7859b6-8lrgz\" (UID: \"25cfc772-5c2c-4146-b2c4-952ae3f7c996\") " pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-8lrgz"
Sep 10 20:44:31.520080 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:31.520031    2564 generic.go:356] "Generic (PLEG): container finished" podID="c5fb7ad9-769e-48c2-96f0-d237e59c3be8" containerID="acc4c854bf8a132394e5c768eb383ad2ba096040751e872785fcac3cbc1764bd" exitCode=0
Sep 10 20:44:31.520370 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:31.520113    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mq88p" event={"ID":"c5fb7ad9-769e-48c2-96f0-d237e59c3be8","Type":"ContainerDied","Data":"acc4c854bf8a132394e5c768eb383ad2ba096040751e872785fcac3cbc1764bd"}
Sep 10 20:44:31.549259 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:31.549237    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-8lrgz"
Sep 10 20:44:31.829261 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:31.829232    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-59df7859b6-8lrgz"]
Sep 10 20:44:31.832349 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:44:31.832310    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25cfc772_5c2c_4146_b2c4_952ae3f7c996.slice/crio-69c7207aa4c64046199b42910d8c54147a9e59b0ed09289852fb0c8ceca3342d WatchSource:0}: Error finding container 69c7207aa4c64046199b42910d8c54147a9e59b0ed09289852fb0c8ceca3342d: Status 404 returned error can't find the container with id 69c7207aa4c64046199b42910d8c54147a9e59b0ed09289852fb0c8ceca3342d
Sep 10 20:44:32.527798 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:32.527720    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-8lrgz" event={"ID":"25cfc772-5c2c-4146-b2c4-952ae3f7c996","Type":"ContainerStarted","Data":"69c7207aa4c64046199b42910d8c54147a9e59b0ed09289852fb0c8ceca3342d"}
Sep 10 20:44:32.530509 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:32.530480    2564 generic.go:356] "Generic (PLEG): container finished" podID="c5fb7ad9-769e-48c2-96f0-d237e59c3be8" containerID="39033c72731063022fef60236ab4cd390ea6303f0d5c63b7022961bf9151a0c2" exitCode=0
Sep 10 20:44:32.530617 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:32.530552    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mq88p" event={"ID":"c5fb7ad9-769e-48c2-96f0-d237e59c3be8","Type":"ContainerDied","Data":"39033c72731063022fef60236ab4cd390ea6303f0d5c63b7022961bf9151a0c2"}
Sep 10 20:44:34.759014 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:34.758991    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mq88p"
Sep 10 20:44:34.844929 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:34.844911    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5fb7ad9-769e-48c2-96f0-d237e59c3be8-bundle\") pod \"c5fb7ad9-769e-48c2-96f0-d237e59c3be8\" (UID: \"c5fb7ad9-769e-48c2-96f0-d237e59c3be8\") "
Sep 10 20:44:34.845017 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:34.844954    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c5fb7ad9-769e-48c2-96f0-d237e59c3be8-util\") pod \"c5fb7ad9-769e-48c2-96f0-d237e59c3be8\" (UID: \"c5fb7ad9-769e-48c2-96f0-d237e59c3be8\") "
Sep 10 20:44:34.845017 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:34.844980    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m88s2\" (UniqueName: \"kubernetes.io/projected/c5fb7ad9-769e-48c2-96f0-d237e59c3be8-kube-api-access-m88s2\") pod \"c5fb7ad9-769e-48c2-96f0-d237e59c3be8\" (UID: \"c5fb7ad9-769e-48c2-96f0-d237e59c3be8\") "
Sep 10 20:44:34.845708 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:34.845677    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5fb7ad9-769e-48c2-96f0-d237e59c3be8-bundle" (OuterVolumeSpecName: "bundle") pod "c5fb7ad9-769e-48c2-96f0-d237e59c3be8" (UID: "c5fb7ad9-769e-48c2-96f0-d237e59c3be8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:44:34.847370 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:34.847342    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5fb7ad9-769e-48c2-96f0-d237e59c3be8-kube-api-access-m88s2" (OuterVolumeSpecName: "kube-api-access-m88s2") pod "c5fb7ad9-769e-48c2-96f0-d237e59c3be8" (UID: "c5fb7ad9-769e-48c2-96f0-d237e59c3be8"). InnerVolumeSpecName "kube-api-access-m88s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:44:34.851598 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:34.851574    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5fb7ad9-769e-48c2-96f0-d237e59c3be8-util" (OuterVolumeSpecName: "util") pod "c5fb7ad9-769e-48c2-96f0-d237e59c3be8" (UID: "c5fb7ad9-769e-48c2-96f0-d237e59c3be8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:44:34.945885 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:34.945862    2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5fb7ad9-769e-48c2-96f0-d237e59c3be8-bundle\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:44:34.945885 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:34.945882    2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c5fb7ad9-769e-48c2-96f0-d237e59c3be8-util\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:44:34.945995 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:34.945892    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m88s2\" (UniqueName: \"kubernetes.io/projected/c5fb7ad9-769e-48c2-96f0-d237e59c3be8-kube-api-access-m88s2\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:44:35.546293 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:35.546248    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-8lrgz" event={"ID":"25cfc772-5c2c-4146-b2c4-952ae3f7c996","Type":"ContainerStarted","Data":"00cb4bd2e5336d83564637fe4b709a442cf1d30e7b1bac0e1396b4a2f70266b9"}
Sep 10 20:44:35.548145 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:35.548117    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mq88p" event={"ID":"c5fb7ad9-769e-48c2-96f0-d237e59c3be8","Type":"ContainerDied","Data":"778adc421a68ad7e141c7e0050a109e50c343949fd2253ad4cd3f55d8dd6eb9e"}
Sep 10 20:44:35.548276 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:35.548159    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="778adc421a68ad7e141c7e0050a109e50c343949fd2253ad4cd3f55d8dd6eb9e"
Sep 10 20:44:35.548276 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:35.548159    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0mq88p"
Sep 10 20:44:36.553246 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:36.553211    2564 generic.go:356] "Generic (PLEG): container finished" podID="25cfc772-5c2c-4146-b2c4-952ae3f7c996" containerID="00cb4bd2e5336d83564637fe4b709a442cf1d30e7b1bac0e1396b4a2f70266b9" exitCode=0
Sep 10 20:44:36.553683 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:36.553296    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-8lrgz" event={"ID":"25cfc772-5c2c-4146-b2c4-952ae3f7c996","Type":"ContainerDied","Data":"00cb4bd2e5336d83564637fe4b709a442cf1d30e7b1bac0e1396b4a2f70266b9"}
Sep 10 20:44:37.560014 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:37.559978    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-8lrgz" event={"ID":"25cfc772-5c2c-4146-b2c4-952ae3f7c996","Type":"ContainerStarted","Data":"24044e5a4580dc6598f5bdf660db7ce25cc24be9998bca57eaa4fa4d792c6b03"}
Sep 10 20:44:37.560465 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:37.560180    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-8lrgz"
Sep 10 20:44:37.587777 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:37.586769    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-8lrgz" podStartSLOduration=2.167334271 podStartE2EDuration="6.586748103s" totalImagesPullingTime="3.415462089s" totalInitContainerRuntime="1.003951743s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:44:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:44:37.585382935 +0000 UTC m=+517.550451132" watchObservedRunningTime="2026-09-10 20:44:37.586748103 +0000 UTC m=+517.551816299"
Sep 10 20:44:48.566382 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:48.566349    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-59df7859b6-8lrgz"
Sep 10 20:44:50.889494 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:50.889463    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-556d9c8688-7qzx6"]
Sep 10 20:44:50.889851 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:50.889702    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5fb7ad9-769e-48c2-96f0-d237e59c3be8" containerName="util"
Sep 10 20:44:50.889851 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:50.889712    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5fb7ad9-769e-48c2-96f0-d237e59c3be8" containerName="util"
Sep 10 20:44:50.889851 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:50.889720    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5fb7ad9-769e-48c2-96f0-d237e59c3be8" containerName="pull"
Sep 10 20:44:50.889851 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:50.889725    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5fb7ad9-769e-48c2-96f0-d237e59c3be8" containerName="pull"
Sep 10 20:44:50.889851 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:50.889740    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5fb7ad9-769e-48c2-96f0-d237e59c3be8" containerName="extract"
Sep 10 20:44:50.889851 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:50.889746    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5fb7ad9-769e-48c2-96f0-d237e59c3be8" containerName="extract"
Sep 10 20:44:50.889851 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:50.889789    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5fb7ad9-769e-48c2-96f0-d237e59c3be8" containerName="extract"
Sep 10 20:44:50.902814 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:50.902792    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-556d9c8688-7qzx6"
Sep 10 20:44:50.906227 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:50.906202    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 20:44:50.907649 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:50.907622    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 20:44:50.908116 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:50.907929    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 20:44:50.908116 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:50.907944    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 20:44:50.908926 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:50.908898    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-556d9c8688-7qzx6"]
Sep 10 20:44:51.047368 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:51.047335    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk7h8\" (UniqueName: \"kubernetes.io/projected/a0be5df7-c7e6-4558-9ae1-79e4980571cb-kube-api-access-vk7h8\") pod \"kube-auth-proxy-556d9c8688-7qzx6\" (UID: \"a0be5df7-c7e6-4558-9ae1-79e4980571cb\") " pod="openshift-ingress/kube-auth-proxy-556d9c8688-7qzx6"
Sep 10 20:44:51.047523 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:51.047393    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0be5df7-c7e6-4558-9ae1-79e4980571cb-tls-certs\") pod \"kube-auth-proxy-556d9c8688-7qzx6\" (UID: \"a0be5df7-c7e6-4558-9ae1-79e4980571cb\") " pod="openshift-ingress/kube-auth-proxy-556d9c8688-7qzx6"
Sep 10 20:44:51.047523 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:51.047417    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0be5df7-c7e6-4558-9ae1-79e4980571cb-tmp\") pod \"kube-auth-proxy-556d9c8688-7qzx6\" (UID: \"a0be5df7-c7e6-4558-9ae1-79e4980571cb\") " pod="openshift-ingress/kube-auth-proxy-556d9c8688-7qzx6"
Sep 10 20:44:51.148156 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:51.148095    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0be5df7-c7e6-4558-9ae1-79e4980571cb-tls-certs\") pod \"kube-auth-proxy-556d9c8688-7qzx6\" (UID: \"a0be5df7-c7e6-4558-9ae1-79e4980571cb\") " pod="openshift-ingress/kube-auth-proxy-556d9c8688-7qzx6"
Sep 10 20:44:51.148156 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:51.148122    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0be5df7-c7e6-4558-9ae1-79e4980571cb-tmp\") pod \"kube-auth-proxy-556d9c8688-7qzx6\" (UID: \"a0be5df7-c7e6-4558-9ae1-79e4980571cb\") " pod="openshift-ingress/kube-auth-proxy-556d9c8688-7qzx6"
Sep 10 20:44:51.148314 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:51.148162    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk7h8\" (UniqueName: \"kubernetes.io/projected/a0be5df7-c7e6-4558-9ae1-79e4980571cb-kube-api-access-vk7h8\") pod \"kube-auth-proxy-556d9c8688-7qzx6\" (UID: \"a0be5df7-c7e6-4558-9ae1-79e4980571cb\") " pod="openshift-ingress/kube-auth-proxy-556d9c8688-7qzx6"
Sep 10 20:44:51.150352 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:51.150324    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0be5df7-c7e6-4558-9ae1-79e4980571cb-tmp\") pod \"kube-auth-proxy-556d9c8688-7qzx6\" (UID: \"a0be5df7-c7e6-4558-9ae1-79e4980571cb\") " pod="openshift-ingress/kube-auth-proxy-556d9c8688-7qzx6"
Sep 10 20:44:51.150591 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:51.150575    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0be5df7-c7e6-4558-9ae1-79e4980571cb-tls-certs\") pod \"kube-auth-proxy-556d9c8688-7qzx6\" (UID: \"a0be5df7-c7e6-4558-9ae1-79e4980571cb\") " pod="openshift-ingress/kube-auth-proxy-556d9c8688-7qzx6"
Sep 10 20:44:51.156527 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:51.156507    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk7h8\" (UniqueName: \"kubernetes.io/projected/a0be5df7-c7e6-4558-9ae1-79e4980571cb-kube-api-access-vk7h8\") pod \"kube-auth-proxy-556d9c8688-7qzx6\" (UID: \"a0be5df7-c7e6-4558-9ae1-79e4980571cb\") " pod="openshift-ingress/kube-auth-proxy-556d9c8688-7qzx6"
Sep 10 20:44:51.214284 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:51.214263    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-556d9c8688-7qzx6"
Sep 10 20:44:51.487093 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:51.487068    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-556d9c8688-7qzx6"]
Sep 10 20:44:51.489276 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:44:51.489245    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0be5df7_c7e6_4558_9ae1_79e4980571cb.slice/crio-db74d114789ee5d48c797a2887e1016e12e92daf690ca851e38aa6c72c99ee98 WatchSource:0}: Error finding container db74d114789ee5d48c797a2887e1016e12e92daf690ca851e38aa6c72c99ee98: Status 404 returned error can't find the container with id db74d114789ee5d48c797a2887e1016e12e92daf690ca851e38aa6c72c99ee98
Sep 10 20:44:51.619531 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:51.619505    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-556d9c8688-7qzx6" event={"ID":"a0be5df7-c7e6-4558-9ae1-79e4980571cb","Type":"ContainerStarted","Data":"db74d114789ee5d48c797a2887e1016e12e92daf690ca851e38aa6c72c99ee98"}
Sep 10 20:44:54.636048 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:54.636012    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-556d9c8688-7qzx6" event={"ID":"a0be5df7-c7e6-4558-9ae1-79e4980571cb","Type":"ContainerStarted","Data":"1ae824ca6b49263c82711fda014bce34b30bb6bf0c930a516dba5c839d732f89"}
Sep 10 20:44:54.660944 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:54.660896    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-556d9c8688-7qzx6" podStartSLOduration=1.876750438 podStartE2EDuration="4.660876662s" totalImagesPullingTime="2.784126224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:44:54.660202944 +0000 UTC m=+534.625271142" watchObservedRunningTime="2026-09-10 20:44:54.660876662 +0000 UTC m=+534.625944856"
Sep 10 20:44:57.727775 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:57.727742    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"]
Sep 10 20:44:57.731098 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:57.731072    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"
Sep 10 20:44:57.734417 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:57.734394    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-5xbwp\""
Sep 10 20:44:57.734765 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:57.734744    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 20:44:57.746728 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:57.746707    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"]
Sep 10 20:44:57.792077 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:57.792051    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7x6z\" (UniqueName: \"kubernetes.io/projected/f502ab18-8459-4acf-a582-57b7ce34b53b-kube-api-access-s7x6z\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9\" (UID: \"f502ab18-8459-4acf-a582-57b7ce34b53b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"
Sep 10 20:44:57.792161 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:57.792089    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f502ab18-8459-4acf-a582-57b7ce34b53b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9\" (UID: \"f502ab18-8459-4acf-a582-57b7ce34b53b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"
Sep 10 20:44:57.792161 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:57.792117    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f502ab18-8459-4acf-a582-57b7ce34b53b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9\" (UID: \"f502ab18-8459-4acf-a582-57b7ce34b53b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"
Sep 10 20:44:57.792294 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:57.792194    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f502ab18-8459-4acf-a582-57b7ce34b53b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9\" (UID: \"f502ab18-8459-4acf-a582-57b7ce34b53b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"
Sep 10 20:44:57.792294 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:57.792219    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f502ab18-8459-4acf-a582-57b7ce34b53b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9\" (UID: \"f502ab18-8459-4acf-a582-57b7ce34b53b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"
Sep 10 20:44:57.792294 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:57.792244    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f502ab18-8459-4acf-a582-57b7ce34b53b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9\" (UID: \"f502ab18-8459-4acf-a582-57b7ce34b53b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"
Sep 10 20:44:57.792294 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:57.792283    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f502ab18-8459-4acf-a582-57b7ce34b53b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9\" (UID: \"f502ab18-8459-4acf-a582-57b7ce34b53b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"
Sep 10 20:44:57.792428 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:57.792317    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f502ab18-8459-4acf-a582-57b7ce34b53b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9\" (UID: \"f502ab18-8459-4acf-a582-57b7ce34b53b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"
Sep 10 20:44:57.792428 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:57.792335    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f502ab18-8459-4acf-a582-57b7ce34b53b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9\" (UID: \"f502ab18-8459-4acf-a582-57b7ce34b53b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"
Sep 10 20:44:57.893120 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:57.893096    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f502ab18-8459-4acf-a582-57b7ce34b53b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9\" (UID: \"f502ab18-8459-4acf-a582-57b7ce34b53b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"
Sep 10 20:44:57.893219 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:57.893136    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f502ab18-8459-4acf-a582-57b7ce34b53b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9\" (UID: \"f502ab18-8459-4acf-a582-57b7ce34b53b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"
Sep 10 20:44:57.893219 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:57.893155    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f502ab18-8459-4acf-a582-57b7ce34b53b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9\" (UID: \"f502ab18-8459-4acf-a582-57b7ce34b53b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"
Sep 10 20:44:57.893219 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:57.893198    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7x6z\" (UniqueName: \"kubernetes.io/projected/f502ab18-8459-4acf-a582-57b7ce34b53b-kube-api-access-s7x6z\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9\" (UID: \"f502ab18-8459-4acf-a582-57b7ce34b53b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"
Sep 10 20:44:57.893357 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:57.893228    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f502ab18-8459-4acf-a582-57b7ce34b53b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9\" (UID: \"f502ab18-8459-4acf-a582-57b7ce34b53b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"
Sep 10 20:44:57.893357 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:57.893245    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f502ab18-8459-4acf-a582-57b7ce34b53b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9\" (UID: \"f502ab18-8459-4acf-a582-57b7ce34b53b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"
Sep 10 20:44:57.893357 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:57.893264    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f502ab18-8459-4acf-a582-57b7ce34b53b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9\" (UID: \"f502ab18-8459-4acf-a582-57b7ce34b53b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"
Sep 10 20:44:57.893357 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:57.893288    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f502ab18-8459-4acf-a582-57b7ce34b53b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9\" (UID: \"f502ab18-8459-4acf-a582-57b7ce34b53b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"
Sep 10 20:44:57.893357 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:57.893325    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f502ab18-8459-4acf-a582-57b7ce34b53b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9\" (UID: \"f502ab18-8459-4acf-a582-57b7ce34b53b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"
Sep 10 20:44:57.893635 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:57.893614    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f502ab18-8459-4acf-a582-57b7ce34b53b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9\" (UID: \"f502ab18-8459-4acf-a582-57b7ce34b53b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"
Sep 10 20:44:57.893705 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:57.893640    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f502ab18-8459-4acf-a582-57b7ce34b53b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9\" (UID: \"f502ab18-8459-4acf-a582-57b7ce34b53b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"
Sep 10 20:44:57.893773 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:57.893749    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f502ab18-8459-4acf-a582-57b7ce34b53b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9\" (UID: \"f502ab18-8459-4acf-a582-57b7ce34b53b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"
Sep 10 20:44:57.893850 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:57.893830    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f502ab18-8459-4acf-a582-57b7ce34b53b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9\" (UID: \"f502ab18-8459-4acf-a582-57b7ce34b53b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"
Sep 10 20:44:57.894242 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:57.894222    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f502ab18-8459-4acf-a582-57b7ce34b53b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9\" (UID: \"f502ab18-8459-4acf-a582-57b7ce34b53b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"
Sep 10 20:44:57.895555 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:57.895537    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f502ab18-8459-4acf-a582-57b7ce34b53b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9\" (UID: \"f502ab18-8459-4acf-a582-57b7ce34b53b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"
Sep 10 20:44:57.895966 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:57.895945    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f502ab18-8459-4acf-a582-57b7ce34b53b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9\" (UID: \"f502ab18-8459-4acf-a582-57b7ce34b53b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"
Sep 10 20:44:57.903140 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:57.903120    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f502ab18-8459-4acf-a582-57b7ce34b53b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9\" (UID: \"f502ab18-8459-4acf-a582-57b7ce34b53b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"
Sep 10 20:44:57.903430 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:57.903410    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7x6z\" (UniqueName: \"kubernetes.io/projected/f502ab18-8459-4acf-a582-57b7ce34b53b-kube-api-access-s7x6z\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9\" (UID: \"f502ab18-8459-4acf-a582-57b7ce34b53b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"
Sep 10 20:44:58.042477 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:58.042419    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"
Sep 10 20:44:58.322361 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:58.322335    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"]
Sep 10 20:44:58.326285 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:44:58.326250    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf502ab18_8459_4acf_a582_57b7ce34b53b.slice/crio-a3ac7314b5f7de8edb06ada4c48b3517f674f10be32931a3acad4fad340907f2 WatchSource:0}: Error finding container a3ac7314b5f7de8edb06ada4c48b3517f674f10be32931a3acad4fad340907f2: Status 404 returned error can't find the container with id a3ac7314b5f7de8edb06ada4c48b3517f674f10be32931a3acad4fad340907f2
Sep 10 20:44:58.659386 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:44:58.659305    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9" event={"ID":"f502ab18-8459-4acf-a582-57b7ce34b53b","Type":"ContainerStarted","Data":"a3ac7314b5f7de8edb06ada4c48b3517f674f10be32931a3acad4fad340907f2"}
Sep 10 20:45:01.105879 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:01.105827    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 20:45:01.106270 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:01.105919    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 20:45:01.106270 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:01.105956    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 20:45:01.640618 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:01.640578    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8hktp"]
Sep 10 20:45:01.643999 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:01.643975    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-8hktp"
Sep 10 20:45:01.648446 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:01.648421    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 10 20:45:01.648546 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:01.648505    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-4f9vd\""
Sep 10 20:45:01.671470 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:01.671438    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8hktp"]
Sep 10 20:45:01.676225 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:01.676191    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9" event={"ID":"f502ab18-8459-4acf-a582-57b7ce34b53b","Type":"ContainerStarted","Data":"fa74f76525d312838977412b42c856608e757f048b38c52665f6413aa8d98adf"}
Sep 10 20:45:01.724004 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:01.723971    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/81b9c2e1-8b00-4fd2-91b4-a0d233ae7e05-cert\") pod \"odh-model-controller-7998c67dc4-8hktp\" (UID: \"81b9c2e1-8b00-4fd2-91b4-a0d233ae7e05\") " pod="opendatahub/odh-model-controller-7998c67dc4-8hktp"
Sep 10 20:45:01.724352 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:01.724311    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25jcb\" (UniqueName: \"kubernetes.io/projected/81b9c2e1-8b00-4fd2-91b4-a0d233ae7e05-kube-api-access-25jcb\") pod \"odh-model-controller-7998c67dc4-8hktp\" (UID: \"81b9c2e1-8b00-4fd2-91b4-a0d233ae7e05\") " pod="opendatahub/odh-model-controller-7998c67dc4-8hktp"
Sep 10 20:45:01.772645 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:01.772593    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9" podStartSLOduration=1.9950571099999999 podStartE2EDuration="4.772575526s" totalImagesPullingTime="2.777518416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:44:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:45:01.770894445 +0000 UTC m=+541.735962639" watchObservedRunningTime="2026-09-10 20:45:01.772575526 +0000 UTC m=+541.737643721"
Sep 10 20:45:01.825355 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:01.825325    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/81b9c2e1-8b00-4fd2-91b4-a0d233ae7e05-cert\") pod \"odh-model-controller-7998c67dc4-8hktp\" (UID: \"81b9c2e1-8b00-4fd2-91b4-a0d233ae7e05\") " pod="opendatahub/odh-model-controller-7998c67dc4-8hktp"
Sep 10 20:45:01.825474 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:01.825364    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25jcb\" (UniqueName: \"kubernetes.io/projected/81b9c2e1-8b00-4fd2-91b4-a0d233ae7e05-kube-api-access-25jcb\") pod \"odh-model-controller-7998c67dc4-8hktp\" (UID: \"81b9c2e1-8b00-4fd2-91b4-a0d233ae7e05\") " pod="opendatahub/odh-model-controller-7998c67dc4-8hktp"
Sep 10 20:45:01.825474 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:45:01.825462    2564 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 10 20:45:01.825542 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:45:01.825525    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81b9c2e1-8b00-4fd2-91b4-a0d233ae7e05-cert podName:81b9c2e1-8b00-4fd2-91b4-a0d233ae7e05 nodeName:}" failed. No retries permitted until 2026-09-10 20:45:02.325509649 +0000 UTC m=+542.290577822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/81b9c2e1-8b00-4fd2-91b4-a0d233ae7e05-cert") pod "odh-model-controller-7998c67dc4-8hktp" (UID: "81b9c2e1-8b00-4fd2-91b4-a0d233ae7e05") : secret "odh-model-controller-webhook-cert" not found
Sep 10 20:45:01.837017 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:01.836995    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25jcb\" (UniqueName: \"kubernetes.io/projected/81b9c2e1-8b00-4fd2-91b4-a0d233ae7e05-kube-api-access-25jcb\") pod \"odh-model-controller-7998c67dc4-8hktp\" (UID: \"81b9c2e1-8b00-4fd2-91b4-a0d233ae7e05\") " pod="opendatahub/odh-model-controller-7998c67dc4-8hktp"
Sep 10 20:45:02.042690 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:02.042603    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"
Sep 10 20:45:02.046978 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:02.046953    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"
Sep 10 20:45:02.329483 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:02.329453    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/81b9c2e1-8b00-4fd2-91b4-a0d233ae7e05-cert\") pod \"odh-model-controller-7998c67dc4-8hktp\" (UID: \"81b9c2e1-8b00-4fd2-91b4-a0d233ae7e05\") " pod="opendatahub/odh-model-controller-7998c67dc4-8hktp"
Sep 10 20:45:02.331959 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:02.331935    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/81b9c2e1-8b00-4fd2-91b4-a0d233ae7e05-cert\") pod \"odh-model-controller-7998c67dc4-8hktp\" (UID: \"81b9c2e1-8b00-4fd2-91b4-a0d233ae7e05\") " pod="opendatahub/odh-model-controller-7998c67dc4-8hktp"
Sep 10 20:45:02.350381 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:02.350358    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-f59c6"]
Sep 10 20:45:02.353486 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:02.353469    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-f59c6"
Sep 10 20:45:02.358036 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:02.358009    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 10 20:45:02.374743 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:02.373901    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-f59c6"]
Sep 10 20:45:02.429937 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:02.429906    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l8sl\" (UniqueName: \"kubernetes.io/projected/73bee6fb-d337-4d9f-96b5-8068b9468996-kube-api-access-9l8sl\") pod \"model-serving-api-54d89bd5fc-f59c6\" (UID: \"73bee6fb-d337-4d9f-96b5-8068b9468996\") " pod="opendatahub/model-serving-api-54d89bd5fc-f59c6"
Sep 10 20:45:02.430076 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:02.429950    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73bee6fb-d337-4d9f-96b5-8068b9468996-tls-certs\") pod \"model-serving-api-54d89bd5fc-f59c6\" (UID: \"73bee6fb-d337-4d9f-96b5-8068b9468996\") " pod="opendatahub/model-serving-api-54d89bd5fc-f59c6"
Sep 10 20:45:02.530765 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:02.530732    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73bee6fb-d337-4d9f-96b5-8068b9468996-tls-certs\") pod \"model-serving-api-54d89bd5fc-f59c6\" (UID: \"73bee6fb-d337-4d9f-96b5-8068b9468996\") " pod="opendatahub/model-serving-api-54d89bd5fc-f59c6"
Sep 10 20:45:02.530960 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:02.530798    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l8sl\" (UniqueName: \"kubernetes.io/projected/73bee6fb-d337-4d9f-96b5-8068b9468996-kube-api-access-9l8sl\") pod \"model-serving-api-54d89bd5fc-f59c6\" (UID: \"73bee6fb-d337-4d9f-96b5-8068b9468996\") " pod="opendatahub/model-serving-api-54d89bd5fc-f59c6"
Sep 10 20:45:02.533356 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:02.533329    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73bee6fb-d337-4d9f-96b5-8068b9468996-tls-certs\") pod \"model-serving-api-54d89bd5fc-f59c6\" (UID: \"73bee6fb-d337-4d9f-96b5-8068b9468996\") " pod="opendatahub/model-serving-api-54d89bd5fc-f59c6"
Sep 10 20:45:02.541580 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:02.541558    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l8sl\" (UniqueName: \"kubernetes.io/projected/73bee6fb-d337-4d9f-96b5-8068b9468996-kube-api-access-9l8sl\") pod \"model-serving-api-54d89bd5fc-f59c6\" (UID: \"73bee6fb-d337-4d9f-96b5-8068b9468996\") " pod="opendatahub/model-serving-api-54d89bd5fc-f59c6"
Sep 10 20:45:02.554347 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:02.554327    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-8hktp"
Sep 10 20:45:02.663815 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:02.663781    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-f59c6"
Sep 10 20:45:02.681816 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:02.681782    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"
Sep 10 20:45:02.682948 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:02.682925    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9"
Sep 10 20:45:02.836795 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:02.836742    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8hktp"]
Sep 10 20:45:02.839327 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:45:02.839289    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b9c2e1_8b00_4fd2_91b4_a0d233ae7e05.slice/crio-4a0d435e87c5fe60ab2c6c00c64ebd452d7d5d58c15ca405f57ba3f019fe874a WatchSource:0}: Error finding container 4a0d435e87c5fe60ab2c6c00c64ebd452d7d5d58c15ca405f57ba3f019fe874a: Status 404 returned error can't find the container with id 4a0d435e87c5fe60ab2c6c00c64ebd452d7d5d58c15ca405f57ba3f019fe874a
Sep 10 20:45:02.944091 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:02.944067    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-f59c6"]
Sep 10 20:45:02.946627 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:45:02.946602    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73bee6fb_d337_4d9f_96b5_8068b9468996.slice/crio-03d7709ec6e87cc1bd843b7314c6c45c7fd97f65bc79514faaa2dce64d8439e3 WatchSource:0}: Error finding container 03d7709ec6e87cc1bd843b7314c6c45c7fd97f65bc79514faaa2dce64d8439e3: Status 404 returned error can't find the container with id 03d7709ec6e87cc1bd843b7314c6c45c7fd97f65bc79514faaa2dce64d8439e3
Sep 10 20:45:03.690995 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:03.690952    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-f59c6" event={"ID":"73bee6fb-d337-4d9f-96b5-8068b9468996","Type":"ContainerStarted","Data":"03d7709ec6e87cc1bd843b7314c6c45c7fd97f65bc79514faaa2dce64d8439e3"}
Sep 10 20:45:03.693547 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:03.693464    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-8hktp" event={"ID":"81b9c2e1-8b00-4fd2-91b4-a0d233ae7e05","Type":"ContainerStarted","Data":"4a0d435e87c5fe60ab2c6c00c64ebd452d7d5d58c15ca405f57ba3f019fe874a"}
Sep 10 20:45:06.708899 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:06.708859    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-f59c6" event={"ID":"73bee6fb-d337-4d9f-96b5-8068b9468996","Type":"ContainerStarted","Data":"a3e9d64fae4464096aa56f8eabef5eb6c07a0249f4ec48868387862e8acc6aed"}
Sep 10 20:45:06.709341 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:06.708924    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-f59c6"
Sep 10 20:45:06.710534 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:06.710508    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-8hktp" event={"ID":"81b9c2e1-8b00-4fd2-91b4-a0d233ae7e05","Type":"ContainerStarted","Data":"1873be2bae1b7ce4a279d514a45188b523cda1fc020fe1b36cb1ff3f4011a2c1"}
Sep 10 20:45:06.710670 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:06.710562    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-8hktp"
Sep 10 20:45:06.726577 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:06.726531    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-f59c6" podStartSLOduration=1.68049127 podStartE2EDuration="4.726511618s" totalImagesPullingTime="3.046020348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:45:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:45:06.725982003 +0000 UTC m=+546.691050199" watchObservedRunningTime="2026-09-10 20:45:06.726511618 +0000 UTC m=+546.691579813"
Sep 10 20:45:06.742430 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:06.742389    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-8hktp" podStartSLOduration=2.543816769 podStartE2EDuration="5.74237744s" totalImagesPullingTime="3.198560671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:45:06.741235774 +0000 UTC m=+546.706303994" watchObservedRunningTime="2026-09-10 20:45:06.74237744 +0000 UTC m=+546.707445634"
Sep 10 20:45:09.625798 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:09.625763    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-8d95fb99-5m4qs"]
Sep 10 20:45:09.628367 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:09.628350    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8d95fb99-5m4qs"
Sep 10 20:45:09.630327 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:09.630301    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 20:45:09.630458 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:09.630301    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-h2f25\""
Sep 10 20:45:09.631004 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:09.630987    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 20:45:09.631109 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:09.631088    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 20:45:09.631161 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:09.631099    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 20:45:09.631161 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:09.631127    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 20:45:09.639744 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:09.639725    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8d95fb99-5m4qs"]
Sep 10 20:45:09.686891 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:09.686867    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5ab23fce-8336-47a9-b0d4-a651446ffea7-manager-config\") pod \"lws-controller-manager-8d95fb99-5m4qs\" (UID: \"5ab23fce-8336-47a9-b0d4-a651446ffea7\") " pod="openshift-lws-operator/lws-controller-manager-8d95fb99-5m4qs"
Sep 10 20:45:09.687018 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:09.686901    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58kls\" (UniqueName: \"kubernetes.io/projected/5ab23fce-8336-47a9-b0d4-a651446ffea7-kube-api-access-58kls\") pod \"lws-controller-manager-8d95fb99-5m4qs\" (UID: \"5ab23fce-8336-47a9-b0d4-a651446ffea7\") " pod="openshift-lws-operator/lws-controller-manager-8d95fb99-5m4qs"
Sep 10 20:45:09.687018 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:09.686928    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5ab23fce-8336-47a9-b0d4-a651446ffea7-metrics-cert\") pod \"lws-controller-manager-8d95fb99-5m4qs\" (UID: \"5ab23fce-8336-47a9-b0d4-a651446ffea7\") " pod="openshift-lws-operator/lws-controller-manager-8d95fb99-5m4qs"
Sep 10 20:45:09.687018 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:09.686973    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ab23fce-8336-47a9-b0d4-a651446ffea7-cert\") pod \"lws-controller-manager-8d95fb99-5m4qs\" (UID: \"5ab23fce-8336-47a9-b0d4-a651446ffea7\") " pod="openshift-lws-operator/lws-controller-manager-8d95fb99-5m4qs"
Sep 10 20:45:09.788311 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:09.788284    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5ab23fce-8336-47a9-b0d4-a651446ffea7-manager-config\") pod \"lws-controller-manager-8d95fb99-5m4qs\" (UID: \"5ab23fce-8336-47a9-b0d4-a651446ffea7\") " pod="openshift-lws-operator/lws-controller-manager-8d95fb99-5m4qs"
Sep 10 20:45:09.788426 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:09.788322    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58kls\" (UniqueName: \"kubernetes.io/projected/5ab23fce-8336-47a9-b0d4-a651446ffea7-kube-api-access-58kls\") pod \"lws-controller-manager-8d95fb99-5m4qs\" (UID: \"5ab23fce-8336-47a9-b0d4-a651446ffea7\") " pod="openshift-lws-operator/lws-controller-manager-8d95fb99-5m4qs"
Sep 10 20:45:09.788426 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:09.788357    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5ab23fce-8336-47a9-b0d4-a651446ffea7-metrics-cert\") pod \"lws-controller-manager-8d95fb99-5m4qs\" (UID: \"5ab23fce-8336-47a9-b0d4-a651446ffea7\") " pod="openshift-lws-operator/lws-controller-manager-8d95fb99-5m4qs"
Sep 10 20:45:09.788554 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:09.788523    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ab23fce-8336-47a9-b0d4-a651446ffea7-cert\") pod \"lws-controller-manager-8d95fb99-5m4qs\" (UID: \"5ab23fce-8336-47a9-b0d4-a651446ffea7\") " pod="openshift-lws-operator/lws-controller-manager-8d95fb99-5m4qs"
Sep 10 20:45:09.788964 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:09.788943    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5ab23fce-8336-47a9-b0d4-a651446ffea7-manager-config\") pod \"lws-controller-manager-8d95fb99-5m4qs\" (UID: \"5ab23fce-8336-47a9-b0d4-a651446ffea7\") " pod="openshift-lws-operator/lws-controller-manager-8d95fb99-5m4qs"
Sep 10 20:45:09.790751 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:09.790728    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5ab23fce-8336-47a9-b0d4-a651446ffea7-metrics-cert\") pod \"lws-controller-manager-8d95fb99-5m4qs\" (UID: \"5ab23fce-8336-47a9-b0d4-a651446ffea7\") " pod="openshift-lws-operator/lws-controller-manager-8d95fb99-5m4qs"
Sep 10 20:45:09.790834 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:09.790787    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ab23fce-8336-47a9-b0d4-a651446ffea7-cert\") pod \"lws-controller-manager-8d95fb99-5m4qs\" (UID: \"5ab23fce-8336-47a9-b0d4-a651446ffea7\") " pod="openshift-lws-operator/lws-controller-manager-8d95fb99-5m4qs"
Sep 10 20:45:09.799073 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:09.799055    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58kls\" (UniqueName: \"kubernetes.io/projected/5ab23fce-8336-47a9-b0d4-a651446ffea7-kube-api-access-58kls\") pod \"lws-controller-manager-8d95fb99-5m4qs\" (UID: \"5ab23fce-8336-47a9-b0d4-a651446ffea7\") " pod="openshift-lws-operator/lws-controller-manager-8d95fb99-5m4qs"
Sep 10 20:45:09.937033 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:09.936937    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8d95fb99-5m4qs"
Sep 10 20:45:10.220767 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:10.220743    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8d95fb99-5m4qs"]
Sep 10 20:45:10.222621 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:45:10.222599    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab23fce_8336_47a9_b0d4_a651446ffea7.slice/crio-49ebdec9b6d5284f754ee0c73a08e99b8e983cc9fcc09c638ca9548bad06bff8 WatchSource:0}: Error finding container 49ebdec9b6d5284f754ee0c73a08e99b8e983cc9fcc09c638ca9548bad06bff8: Status 404 returned error can't find the container with id 49ebdec9b6d5284f754ee0c73a08e99b8e983cc9fcc09c638ca9548bad06bff8
Sep 10 20:45:10.730804 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:10.730772    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8d95fb99-5m4qs" event={"ID":"5ab23fce-8336-47a9-b0d4-a651446ffea7","Type":"ContainerStarted","Data":"49ebdec9b6d5284f754ee0c73a08e99b8e983cc9fcc09c638ca9548bad06bff8"}
Sep 10 20:45:11.736479 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:11.736444    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8d95fb99-5m4qs" event={"ID":"5ab23fce-8336-47a9-b0d4-a651446ffea7","Type":"ContainerStarted","Data":"d5944ddb3f060ef41bafa47863f2be5bc806e82d9b8a4a19a4f9c4a3211809d2"}
Sep 10 20:45:11.736773 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:11.736525    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-8d95fb99-5m4qs"
Sep 10 20:45:11.759435 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:11.759390    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-8d95fb99-5m4qs" podStartSLOduration=1.367197231 podStartE2EDuration="2.759375662s" totalImagesPullingTime="1.392178431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:45:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:45:11.757948721 +0000 UTC m=+551.723016930" watchObservedRunningTime="2026-09-10 20:45:11.759375662 +0000 UTC m=+551.724443856"
Sep 10 20:45:17.716866 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:17.716835    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-8hktp"
Sep 10 20:45:17.718974 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:17.718954    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-f59c6"
Sep 10 20:45:22.743137 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:45:22.743109    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-8d95fb99-5m4qs"
Sep 10 20:46:09.615267 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.615227    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tltjf"]
Sep 10 20:46:09.619368 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.619345    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tltjf"
Sep 10 20:46:09.622126 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.622102    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 20:46:09.622254 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.622186    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-pfn85\""
Sep 10 20:46:09.622664 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.622649    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 20:46:09.628132 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.628107    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tltjf"]
Sep 10 20:46:09.704286 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.704263    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dca6741b-bc4d-4d35-b337-9e2e0861d6e2-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tltjf\" (UID: \"dca6741b-bc4d-4d35-b337-9e2e0861d6e2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tltjf"
Sep 10 20:46:09.704385 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.704299    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zftxl\" (UniqueName: \"kubernetes.io/projected/dca6741b-bc4d-4d35-b337-9e2e0861d6e2-kube-api-access-zftxl\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tltjf\" (UID: \"dca6741b-bc4d-4d35-b337-9e2e0861d6e2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tltjf"
Sep 10 20:46:09.704433 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.704387    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dca6741b-bc4d-4d35-b337-9e2e0861d6e2-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tltjf\" (UID: \"dca6741b-bc4d-4d35-b337-9e2e0861d6e2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tltjf"
Sep 10 20:46:09.720766 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.720743    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswr67"]
Sep 10 20:46:09.723847 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.723832    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswr67"
Sep 10 20:46:09.735099 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.735079    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswr67"]
Sep 10 20:46:09.805637 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.805613    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dca6741b-bc4d-4d35-b337-9e2e0861d6e2-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tltjf\" (UID: \"dca6741b-bc4d-4d35-b337-9e2e0861d6e2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tltjf"
Sep 10 20:46:09.805725 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.805649    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zftxl\" (UniqueName: \"kubernetes.io/projected/dca6741b-bc4d-4d35-b337-9e2e0861d6e2-kube-api-access-zftxl\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tltjf\" (UID: \"dca6741b-bc4d-4d35-b337-9e2e0861d6e2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tltjf"
Sep 10 20:46:09.805725 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.805674    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d179db8-f756-493c-876d-5dd905eb506b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswr67\" (UID: \"6d179db8-f756-493c-876d-5dd905eb506b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswr67"
Sep 10 20:46:09.805725 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.805708    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dca6741b-bc4d-4d35-b337-9e2e0861d6e2-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tltjf\" (UID: \"dca6741b-bc4d-4d35-b337-9e2e0861d6e2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tltjf"
Sep 10 20:46:09.805930 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.805730    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d179db8-f756-493c-876d-5dd905eb506b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswr67\" (UID: \"6d179db8-f756-493c-876d-5dd905eb506b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswr67"
Sep 10 20:46:09.805930 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.805747    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckc54\" (UniqueName: \"kubernetes.io/projected/6d179db8-f756-493c-876d-5dd905eb506b-kube-api-access-ckc54\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswr67\" (UID: \"6d179db8-f756-493c-876d-5dd905eb506b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswr67"
Sep 10 20:46:09.806039 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.805949    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dca6741b-bc4d-4d35-b337-9e2e0861d6e2-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tltjf\" (UID: \"dca6741b-bc4d-4d35-b337-9e2e0861d6e2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tltjf"
Sep 10 20:46:09.806092 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.806052    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dca6741b-bc4d-4d35-b337-9e2e0861d6e2-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tltjf\" (UID: \"dca6741b-bc4d-4d35-b337-9e2e0861d6e2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tltjf"
Sep 10 20:46:09.815326 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.815289    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zftxl\" (UniqueName: \"kubernetes.io/projected/dca6741b-bc4d-4d35-b337-9e2e0861d6e2-kube-api-access-zftxl\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tltjf\" (UID: \"dca6741b-bc4d-4d35-b337-9e2e0861d6e2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tltjf"
Sep 10 20:46:09.816690 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.816669    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tn2hs"]
Sep 10 20:46:09.819970 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.819956    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tn2hs"
Sep 10 20:46:09.831911 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.831880    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tn2hs"]
Sep 10 20:46:09.906822 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.906767    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9eff462-3f3c-4b54-a7b0-6d5b265909ca-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tn2hs\" (UID: \"e9eff462-3f3c-4b54-a7b0-6d5b265909ca\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tn2hs"
Sep 10 20:46:09.906822 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.906808    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d179db8-f756-493c-876d-5dd905eb506b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswr67\" (UID: \"6d179db8-f756-493c-876d-5dd905eb506b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswr67"
Sep 10 20:46:09.906935 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.906836    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckc54\" (UniqueName: \"kubernetes.io/projected/6d179db8-f756-493c-876d-5dd905eb506b-kube-api-access-ckc54\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswr67\" (UID: \"6d179db8-f756-493c-876d-5dd905eb506b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswr67"
Sep 10 20:46:09.906935 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.906871    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk8qf\" (UniqueName: \"kubernetes.io/projected/e9eff462-3f3c-4b54-a7b0-6d5b265909ca-kube-api-access-dk8qf\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tn2hs\" (UID: \"e9eff462-3f3c-4b54-a7b0-6d5b265909ca\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tn2hs"
Sep 10 20:46:09.906935 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.906898    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9eff462-3f3c-4b54-a7b0-6d5b265909ca-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tn2hs\" (UID: \"e9eff462-3f3c-4b54-a7b0-6d5b265909ca\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tn2hs"
Sep 10 20:46:09.907034 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.906962    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d179db8-f756-493c-876d-5dd905eb506b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswr67\" (UID: \"6d179db8-f756-493c-876d-5dd905eb506b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswr67"
Sep 10 20:46:09.907160 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.907137    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d179db8-f756-493c-876d-5dd905eb506b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswr67\" (UID: \"6d179db8-f756-493c-876d-5dd905eb506b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswr67"
Sep 10 20:46:09.907250 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.907235    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d179db8-f756-493c-876d-5dd905eb506b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswr67\" (UID: \"6d179db8-f756-493c-876d-5dd905eb506b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswr67"
Sep 10 20:46:09.918587 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.918560    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprh78"]
Sep 10 20:46:09.920161 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.920138    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckc54\" (UniqueName: \"kubernetes.io/projected/6d179db8-f756-493c-876d-5dd905eb506b-kube-api-access-ckc54\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswr67\" (UID: \"6d179db8-f756-493c-876d-5dd905eb506b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswr67"
Sep 10 20:46:09.922073 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.922058    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprh78"
Sep 10 20:46:09.929821 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.929798    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tltjf"
Sep 10 20:46:09.933596 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:09.933573    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprh78"]
Sep 10 20:46:10.008113 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:10.008086    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk8qf\" (UniqueName: \"kubernetes.io/projected/e9eff462-3f3c-4b54-a7b0-6d5b265909ca-kube-api-access-dk8qf\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tn2hs\" (UID: \"e9eff462-3f3c-4b54-a7b0-6d5b265909ca\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tn2hs"
Sep 10 20:46:10.008243 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:10.008126    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9eff462-3f3c-4b54-a7b0-6d5b265909ca-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tn2hs\" (UID: \"e9eff462-3f3c-4b54-a7b0-6d5b265909ca\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tn2hs"
Sep 10 20:46:10.008243 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:10.008160    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d71ad7b8-ca50-412a-ac68-06a362b4f64d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprh78\" (UID: \"d71ad7b8-ca50-412a-ac68-06a362b4f64d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprh78"
Sep 10 20:46:10.008243 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:10.008213    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d71ad7b8-ca50-412a-ac68-06a362b4f64d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprh78\" (UID: \"d71ad7b8-ca50-412a-ac68-06a362b4f64d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprh78"
Sep 10 20:46:10.008243 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:10.008239    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9eff462-3f3c-4b54-a7b0-6d5b265909ca-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tn2hs\" (UID: \"e9eff462-3f3c-4b54-a7b0-6d5b265909ca\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tn2hs"
Sep 10 20:46:10.008418 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:10.008271    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zncf\" (UniqueName: \"kubernetes.io/projected/d71ad7b8-ca50-412a-ac68-06a362b4f64d-kube-api-access-8zncf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprh78\" (UID: \"d71ad7b8-ca50-412a-ac68-06a362b4f64d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprh78"
Sep 10 20:46:10.008519 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:10.008501    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9eff462-3f3c-4b54-a7b0-6d5b265909ca-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tn2hs\" (UID: \"e9eff462-3f3c-4b54-a7b0-6d5b265909ca\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tn2hs"
Sep 10 20:46:10.008590 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:10.008572    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9eff462-3f3c-4b54-a7b0-6d5b265909ca-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tn2hs\" (UID: \"e9eff462-3f3c-4b54-a7b0-6d5b265909ca\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tn2hs"
Sep 10 20:46:10.019419 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:10.019391    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk8qf\" (UniqueName: \"kubernetes.io/projected/e9eff462-3f3c-4b54-a7b0-6d5b265909ca-kube-api-access-dk8qf\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tn2hs\" (UID: \"e9eff462-3f3c-4b54-a7b0-6d5b265909ca\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tn2hs"
Sep 10 20:46:10.032140 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:10.032118    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswr67"
Sep 10 20:46:10.109777 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:10.109749    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d71ad7b8-ca50-412a-ac68-06a362b4f64d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprh78\" (UID: \"d71ad7b8-ca50-412a-ac68-06a362b4f64d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprh78"
Sep 10 20:46:10.109894 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:10.109800    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d71ad7b8-ca50-412a-ac68-06a362b4f64d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprh78\" (UID: \"d71ad7b8-ca50-412a-ac68-06a362b4f64d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprh78"
Sep 10 20:46:10.109894 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:10.109833    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zncf\" (UniqueName: \"kubernetes.io/projected/d71ad7b8-ca50-412a-ac68-06a362b4f64d-kube-api-access-8zncf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprh78\" (UID: \"d71ad7b8-ca50-412a-ac68-06a362b4f64d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprh78"
Sep 10 20:46:10.110206 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:10.110159    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d71ad7b8-ca50-412a-ac68-06a362b4f64d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprh78\" (UID: \"d71ad7b8-ca50-412a-ac68-06a362b4f64d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprh78"
Sep 10 20:46:10.110307 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:10.110232    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d71ad7b8-ca50-412a-ac68-06a362b4f64d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprh78\" (UID: \"d71ad7b8-ca50-412a-ac68-06a362b4f64d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprh78"
Sep 10 20:46:10.118982 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:10.118959    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zncf\" (UniqueName: \"kubernetes.io/projected/d71ad7b8-ca50-412a-ac68-06a362b4f64d-kube-api-access-8zncf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprh78\" (UID: \"d71ad7b8-ca50-412a-ac68-06a362b4f64d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprh78"
Sep 10 20:46:10.134786 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:10.134766    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tn2hs"
Sep 10 20:46:10.206989 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:10.206957    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tltjf"]
Sep 10 20:46:10.233994 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:10.233971    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprh78"
Sep 10 20:46:10.317292 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:10.317261    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswr67"]
Sep 10 20:46:10.319709 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:46:10.319675    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d179db8_f756_493c_876d_5dd905eb506b.slice/crio-69f86e3df4840dcbfdb117f8b7c55fef0297e10c047a2b5b62f9c9bc47562f17 WatchSource:0}: Error finding container 69f86e3df4840dcbfdb117f8b7c55fef0297e10c047a2b5b62f9c9bc47562f17: Status 404 returned error can't find the container with id 69f86e3df4840dcbfdb117f8b7c55fef0297e10c047a2b5b62f9c9bc47562f17
Sep 10 20:46:10.411531 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:10.411507    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tn2hs"]
Sep 10 20:46:10.412738 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:46:10.412712    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9eff462_3f3c_4b54_a7b0_6d5b265909ca.slice/crio-f6cd2432c9002fc77df857de545d167a2ef70862627eac950d70de10fe0f20c7 WatchSource:0}: Error finding container f6cd2432c9002fc77df857de545d167a2ef70862627eac950d70de10fe0f20c7: Status 404 returned error can't find the container with id f6cd2432c9002fc77df857de545d167a2ef70862627eac950d70de10fe0f20c7
Sep 10 20:46:10.510530 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:10.510505    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprh78"]
Sep 10 20:46:10.514653 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:46:10.514627    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71ad7b8_ca50_412a_ac68_06a362b4f64d.slice/crio-ae0e43be9f5b441dd4682dcda36e2ae3157ccb3cdaab434c10892b73efd9a81d WatchSource:0}: Error finding container ae0e43be9f5b441dd4682dcda36e2ae3157ccb3cdaab434c10892b73efd9a81d: Status 404 returned error can't find the container with id ae0e43be9f5b441dd4682dcda36e2ae3157ccb3cdaab434c10892b73efd9a81d
Sep 10 20:46:11.000965 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:11.000930    2564 generic.go:356] "Generic (PLEG): container finished" podID="d71ad7b8-ca50-412a-ac68-06a362b4f64d" containerID="a9f6fed6cf804cac3488b2bc22d521937c6ffd38e05f52d8fca428946cf99787" exitCode=0
Sep 10 20:46:11.001408 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:11.000977    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprh78" event={"ID":"d71ad7b8-ca50-412a-ac68-06a362b4f64d","Type":"ContainerDied","Data":"a9f6fed6cf804cac3488b2bc22d521937c6ffd38e05f52d8fca428946cf99787"}
Sep 10 20:46:11.001408 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:11.001033    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprh78" event={"ID":"d71ad7b8-ca50-412a-ac68-06a362b4f64d","Type":"ContainerStarted","Data":"ae0e43be9f5b441dd4682dcda36e2ae3157ccb3cdaab434c10892b73efd9a81d"}
Sep 10 20:46:11.002590 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:11.002554    2564 generic.go:356] "Generic (PLEG): container finished" podID="6d179db8-f756-493c-876d-5dd905eb506b" containerID="087a79129b6931b281db46f548c8d18cfce230f70643316e077112e96692517d" exitCode=0
Sep 10 20:46:11.002659 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:11.002637    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswr67" event={"ID":"6d179db8-f756-493c-876d-5dd905eb506b","Type":"ContainerDied","Data":"087a79129b6931b281db46f548c8d18cfce230f70643316e077112e96692517d"}
Sep 10 20:46:11.002716 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:11.002687    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswr67" event={"ID":"6d179db8-f756-493c-876d-5dd905eb506b","Type":"ContainerStarted","Data":"69f86e3df4840dcbfdb117f8b7c55fef0297e10c047a2b5b62f9c9bc47562f17"}
Sep 10 20:46:11.004248 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:11.004227    2564 generic.go:356] "Generic (PLEG): container finished" podID="e9eff462-3f3c-4b54-a7b0-6d5b265909ca" containerID="801811df7ffed193dbb4f482ff82275ddb31659be3a5421c1052c23796938e1e" exitCode=0
Sep 10 20:46:11.004364 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:11.004313    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tn2hs" event={"ID":"e9eff462-3f3c-4b54-a7b0-6d5b265909ca","Type":"ContainerDied","Data":"801811df7ffed193dbb4f482ff82275ddb31659be3a5421c1052c23796938e1e"}
Sep 10 20:46:11.004431 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:11.004396    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tn2hs" event={"ID":"e9eff462-3f3c-4b54-a7b0-6d5b265909ca","Type":"ContainerStarted","Data":"f6cd2432c9002fc77df857de545d167a2ef70862627eac950d70de10fe0f20c7"}
Sep 10 20:46:11.006082 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:11.006061    2564 generic.go:356] "Generic (PLEG): container finished" podID="dca6741b-bc4d-4d35-b337-9e2e0861d6e2" containerID="f57185e163346737d0d2b944e998942a8e2c6546a6a063c75c5b11f56937ac4c" exitCode=0
Sep 10 20:46:11.006140 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:11.006104    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tltjf" event={"ID":"dca6741b-bc4d-4d35-b337-9e2e0861d6e2","Type":"ContainerDied","Data":"f57185e163346737d0d2b944e998942a8e2c6546a6a063c75c5b11f56937ac4c"}
Sep 10 20:46:11.006205 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:11.006136    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tltjf" event={"ID":"dca6741b-bc4d-4d35-b337-9e2e0861d6e2","Type":"ContainerStarted","Data":"29f5bd01651061f4a78455e1f64c42eb1e89f8e108d1e69da180e0721e755a40"}
Sep 10 20:46:13.016801 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:13.016767    2564 generic.go:356] "Generic (PLEG): container finished" podID="d71ad7b8-ca50-412a-ac68-06a362b4f64d" containerID="b1459895d73c439b1400564c51ded827df4825d21b9c7cd74397be883b41dbeb" exitCode=0
Sep 10 20:46:13.017260 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:13.016837    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprh78" event={"ID":"d71ad7b8-ca50-412a-ac68-06a362b4f64d","Type":"ContainerDied","Data":"b1459895d73c439b1400564c51ded827df4825d21b9c7cd74397be883b41dbeb"}
Sep 10 20:46:13.019055 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:13.019029    2564 generic.go:356] "Generic (PLEG): container finished" podID="6d179db8-f756-493c-876d-5dd905eb506b" containerID="44b51f9e72b4ebcbf0522f6cb9fe658107914026490354839f6d781366dfb1c7" exitCode=0
Sep 10 20:46:13.019181 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:13.019121    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswr67" event={"ID":"6d179db8-f756-493c-876d-5dd905eb506b","Type":"ContainerDied","Data":"44b51f9e72b4ebcbf0522f6cb9fe658107914026490354839f6d781366dfb1c7"}
Sep 10 20:46:13.021320 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:13.021297    2564 generic.go:356] "Generic (PLEG): container finished" podID="e9eff462-3f3c-4b54-a7b0-6d5b265909ca" containerID="f5da9ff78ec80732fd417bfd853ed005b70627f244896e0dea06c837c2a370c8" exitCode=0
Sep 10 20:46:13.021399 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:13.021348    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tn2hs" event={"ID":"e9eff462-3f3c-4b54-a7b0-6d5b265909ca","Type":"ContainerDied","Data":"f5da9ff78ec80732fd417bfd853ed005b70627f244896e0dea06c837c2a370c8"}
Sep 10 20:46:14.028996 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:14.028961    2564 generic.go:356] "Generic (PLEG): container finished" podID="d71ad7b8-ca50-412a-ac68-06a362b4f64d" containerID="0eeca653da899a019bc5446d1909267abb3534ea39351f504b166175e4f33bc3" exitCode=0
Sep 10 20:46:14.029402 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:14.029045    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprh78" event={"ID":"d71ad7b8-ca50-412a-ac68-06a362b4f64d","Type":"ContainerDied","Data":"0eeca653da899a019bc5446d1909267abb3534ea39351f504b166175e4f33bc3"}
Sep 10 20:46:14.031334 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:14.031307    2564 generic.go:356] "Generic (PLEG): container finished" podID="6d179db8-f756-493c-876d-5dd905eb506b" containerID="5be5fe15c93d950df2217353df6eb663b924d6a93497107334fe4c92cc36c657" exitCode=0
Sep 10 20:46:14.031442 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:14.031393    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswr67" event={"ID":"6d179db8-f756-493c-876d-5dd905eb506b","Type":"ContainerDied","Data":"5be5fe15c93d950df2217353df6eb663b924d6a93497107334fe4c92cc36c657"}
Sep 10 20:46:14.033713 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:14.033692    2564 generic.go:356] "Generic (PLEG): container finished" podID="e9eff462-3f3c-4b54-a7b0-6d5b265909ca" containerID="2304bc69d416daed78c1e332c06623cdc3a24d6f5e125627ac2569b30d877c73" exitCode=0
Sep 10 20:46:14.033830 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:14.033769    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tn2hs" event={"ID":"e9eff462-3f3c-4b54-a7b0-6d5b265909ca","Type":"ContainerDied","Data":"2304bc69d416daed78c1e332c06623cdc3a24d6f5e125627ac2569b30d877c73"}
Sep 10 20:46:15.183206 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:15.183185    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswr67"
Sep 10 20:46:15.216543 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:15.216521    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tn2hs"
Sep 10 20:46:15.219340 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:15.219320    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprh78"
Sep 10 20:46:15.352323 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:15.352294    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d179db8-f756-493c-876d-5dd905eb506b-bundle\") pod \"6d179db8-f756-493c-876d-5dd905eb506b\" (UID: \"6d179db8-f756-493c-876d-5dd905eb506b\") "
Sep 10 20:46:15.352475 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:15.352338    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d71ad7b8-ca50-412a-ac68-06a362b4f64d-bundle\") pod \"d71ad7b8-ca50-412a-ac68-06a362b4f64d\" (UID: \"d71ad7b8-ca50-412a-ac68-06a362b4f64d\") "
Sep 10 20:46:15.352475 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:15.352373    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d71ad7b8-ca50-412a-ac68-06a362b4f64d-util\") pod \"d71ad7b8-ca50-412a-ac68-06a362b4f64d\" (UID: \"d71ad7b8-ca50-412a-ac68-06a362b4f64d\") "
Sep 10 20:46:15.352475 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:15.352402    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9eff462-3f3c-4b54-a7b0-6d5b265909ca-util\") pod \"e9eff462-3f3c-4b54-a7b0-6d5b265909ca\" (UID: \"e9eff462-3f3c-4b54-a7b0-6d5b265909ca\") "
Sep 10 20:46:15.352475 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:15.352444    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zncf\" (UniqueName: \"kubernetes.io/projected/d71ad7b8-ca50-412a-ac68-06a362b4f64d-kube-api-access-8zncf\") pod \"d71ad7b8-ca50-412a-ac68-06a362b4f64d\" (UID: \"d71ad7b8-ca50-412a-ac68-06a362b4f64d\") "
Sep 10 20:46:15.352677 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:15.352486    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dk8qf\" (UniqueName: \"kubernetes.io/projected/e9eff462-3f3c-4b54-a7b0-6d5b265909ca-kube-api-access-dk8qf\") pod \"e9eff462-3f3c-4b54-a7b0-6d5b265909ca\" (UID: \"e9eff462-3f3c-4b54-a7b0-6d5b265909ca\") "
Sep 10 20:46:15.352677 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:15.352518    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckc54\" (UniqueName: \"kubernetes.io/projected/6d179db8-f756-493c-876d-5dd905eb506b-kube-api-access-ckc54\") pod \"6d179db8-f756-493c-876d-5dd905eb506b\" (UID: \"6d179db8-f756-493c-876d-5dd905eb506b\") "
Sep 10 20:46:15.352677 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:15.352555    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d179db8-f756-493c-876d-5dd905eb506b-util\") pod \"6d179db8-f756-493c-876d-5dd905eb506b\" (UID: \"6d179db8-f756-493c-876d-5dd905eb506b\") "
Sep 10 20:46:15.352677 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:15.352607    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9eff462-3f3c-4b54-a7b0-6d5b265909ca-bundle\") pod \"e9eff462-3f3c-4b54-a7b0-6d5b265909ca\" (UID: \"e9eff462-3f3c-4b54-a7b0-6d5b265909ca\") "
Sep 10 20:46:15.353315 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:15.352950    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d179db8-f756-493c-876d-5dd905eb506b-bundle" (OuterVolumeSpecName: "bundle") pod "6d179db8-f756-493c-876d-5dd905eb506b" (UID: "6d179db8-f756-493c-876d-5dd905eb506b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:46:15.353434 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:15.353408    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9eff462-3f3c-4b54-a7b0-6d5b265909ca-bundle" (OuterVolumeSpecName: "bundle") pod "e9eff462-3f3c-4b54-a7b0-6d5b265909ca" (UID: "e9eff462-3f3c-4b54-a7b0-6d5b265909ca"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:46:15.353497 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:15.353436    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d71ad7b8-ca50-412a-ac68-06a362b4f64d-bundle" (OuterVolumeSpecName: "bundle") pod "d71ad7b8-ca50-412a-ac68-06a362b4f64d" (UID: "d71ad7b8-ca50-412a-ac68-06a362b4f64d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:46:15.354962 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:15.354932    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9eff462-3f3c-4b54-a7b0-6d5b265909ca-kube-api-access-dk8qf" (OuterVolumeSpecName: "kube-api-access-dk8qf") pod "e9eff462-3f3c-4b54-a7b0-6d5b265909ca" (UID: "e9eff462-3f3c-4b54-a7b0-6d5b265909ca"). InnerVolumeSpecName "kube-api-access-dk8qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:46:15.355057 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:15.354965    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d179db8-f756-493c-876d-5dd905eb506b-kube-api-access-ckc54" (OuterVolumeSpecName: "kube-api-access-ckc54") pod "6d179db8-f756-493c-876d-5dd905eb506b" (UID: "6d179db8-f756-493c-876d-5dd905eb506b"). InnerVolumeSpecName "kube-api-access-ckc54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:46:15.355057 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:15.355039    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d71ad7b8-ca50-412a-ac68-06a362b4f64d-kube-api-access-8zncf" (OuterVolumeSpecName: "kube-api-access-8zncf") pod "d71ad7b8-ca50-412a-ac68-06a362b4f64d" (UID: "d71ad7b8-ca50-412a-ac68-06a362b4f64d"). InnerVolumeSpecName "kube-api-access-8zncf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:46:15.359049 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:15.359015    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d71ad7b8-ca50-412a-ac68-06a362b4f64d-util" (OuterVolumeSpecName: "util") pod "d71ad7b8-ca50-412a-ac68-06a362b4f64d" (UID: "d71ad7b8-ca50-412a-ac68-06a362b4f64d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:46:15.361795 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:15.361773    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d179db8-f756-493c-876d-5dd905eb506b-util" (OuterVolumeSpecName: "util") pod "6d179db8-f756-493c-876d-5dd905eb506b" (UID: "6d179db8-f756-493c-876d-5dd905eb506b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:46:15.361966 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:15.361950    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9eff462-3f3c-4b54-a7b0-6d5b265909ca-util" (OuterVolumeSpecName: "util") pod "e9eff462-3f3c-4b54-a7b0-6d5b265909ca" (UID: "e9eff462-3f3c-4b54-a7b0-6d5b265909ca"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:46:15.453919 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:15.453884    2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9eff462-3f3c-4b54-a7b0-6d5b265909ca-bundle\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:46:15.453919 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:15.453916    2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d179db8-f756-493c-876d-5dd905eb506b-bundle\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:46:15.453919 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:15.453925    2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d71ad7b8-ca50-412a-ac68-06a362b4f64d-bundle\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:46:15.454065 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:15.453933    2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d71ad7b8-ca50-412a-ac68-06a362b4f64d-util\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:46:15.454065 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:15.453941    2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9eff462-3f3c-4b54-a7b0-6d5b265909ca-util\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:46:15.454065 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:15.453949    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zncf\" (UniqueName: \"kubernetes.io/projected/d71ad7b8-ca50-412a-ac68-06a362b4f64d-kube-api-access-8zncf\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:46:15.454065 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:15.453958    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk8qf\" (UniqueName: \"kubernetes.io/projected/e9eff462-3f3c-4b54-a7b0-6d5b265909ca-kube-api-access-dk8qf\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:46:15.454065 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:15.453967    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckc54\" (UniqueName: \"kubernetes.io/projected/6d179db8-f756-493c-876d-5dd905eb506b-kube-api-access-ckc54\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:46:15.454065 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:15.453974    2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d179db8-f756-493c-876d-5dd905eb506b-util\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:46:16.051974 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:16.051940    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswr67" event={"ID":"6d179db8-f756-493c-876d-5dd905eb506b","Type":"ContainerDied","Data":"69f86e3df4840dcbfdb117f8b7c55fef0297e10c047a2b5b62f9c9bc47562f17"}
Sep 10 20:46:16.052121 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:16.052019    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69f86e3df4840dcbfdb117f8b7c55fef0297e10c047a2b5b62f9c9bc47562f17"
Sep 10 20:46:16.052121 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:16.051945    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswr67"
Sep 10 20:46:16.054111 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:16.054088    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tn2hs"
Sep 10 20:46:16.054261 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:16.054125    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tn2hs" event={"ID":"e9eff462-3f3c-4b54-a7b0-6d5b265909ca","Type":"ContainerDied","Data":"f6cd2432c9002fc77df857de545d167a2ef70862627eac950d70de10fe0f20c7"}
Sep 10 20:46:16.054261 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:16.054187    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6cd2432c9002fc77df857de545d167a2ef70862627eac950d70de10fe0f20c7"
Sep 10 20:46:16.056617 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:16.056587    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprh78" event={"ID":"d71ad7b8-ca50-412a-ac68-06a362b4f64d","Type":"ContainerDied","Data":"ae0e43be9f5b441dd4682dcda36e2ae3157ccb3cdaab434c10892b73efd9a81d"}
Sep 10 20:46:16.056716 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:16.056634    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aprh78"
Sep 10 20:46:16.056760 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:16.056635    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae0e43be9f5b441dd4682dcda36e2ae3157ccb3cdaab434c10892b73efd9a81d"
Sep 10 20:46:17.063081 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:17.063022    2564 generic.go:356] "Generic (PLEG): container finished" podID="dca6741b-bc4d-4d35-b337-9e2e0861d6e2" containerID="c023179230ac34e994c2f75ce40131a6e62466988a3802b828c6075670e98d9c" exitCode=0
Sep 10 20:46:17.063395 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:17.063094    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tltjf" event={"ID":"dca6741b-bc4d-4d35-b337-9e2e0861d6e2","Type":"ContainerDied","Data":"c023179230ac34e994c2f75ce40131a6e62466988a3802b828c6075670e98d9c"}
Sep 10 20:46:18.070905 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:18.070870    2564 generic.go:356] "Generic (PLEG): container finished" podID="dca6741b-bc4d-4d35-b337-9e2e0861d6e2" containerID="eef07105f298495285a38ea3804c455ed00910b2c10b6312660f80499bd06c3f" exitCode=0
Sep 10 20:46:18.071315 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:18.070958    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tltjf" event={"ID":"dca6741b-bc4d-4d35-b337-9e2e0861d6e2","Type":"ContainerDied","Data":"eef07105f298495285a38ea3804c455ed00910b2c10b6312660f80499bd06c3f"}
Sep 10 20:46:19.198519 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:19.198496    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tltjf"
Sep 10 20:46:19.377321 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:19.377253    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zftxl\" (UniqueName: \"kubernetes.io/projected/dca6741b-bc4d-4d35-b337-9e2e0861d6e2-kube-api-access-zftxl\") pod \"dca6741b-bc4d-4d35-b337-9e2e0861d6e2\" (UID: \"dca6741b-bc4d-4d35-b337-9e2e0861d6e2\") "
Sep 10 20:46:19.377321 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:19.377329    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dca6741b-bc4d-4d35-b337-9e2e0861d6e2-util\") pod \"dca6741b-bc4d-4d35-b337-9e2e0861d6e2\" (UID: \"dca6741b-bc4d-4d35-b337-9e2e0861d6e2\") "
Sep 10 20:46:19.377520 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:19.377375    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dca6741b-bc4d-4d35-b337-9e2e0861d6e2-bundle\") pod \"dca6741b-bc4d-4d35-b337-9e2e0861d6e2\" (UID: \"dca6741b-bc4d-4d35-b337-9e2e0861d6e2\") "
Sep 10 20:46:19.377844 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:19.377812    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dca6741b-bc4d-4d35-b337-9e2e0861d6e2-bundle" (OuterVolumeSpecName: "bundle") pod "dca6741b-bc4d-4d35-b337-9e2e0861d6e2" (UID: "dca6741b-bc4d-4d35-b337-9e2e0861d6e2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:46:19.379321 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:19.379297    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dca6741b-bc4d-4d35-b337-9e2e0861d6e2-kube-api-access-zftxl" (OuterVolumeSpecName: "kube-api-access-zftxl") pod "dca6741b-bc4d-4d35-b337-9e2e0861d6e2" (UID: "dca6741b-bc4d-4d35-b337-9e2e0861d6e2"). InnerVolumeSpecName "kube-api-access-zftxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:46:19.381835 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:19.381796    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dca6741b-bc4d-4d35-b337-9e2e0861d6e2-util" (OuterVolumeSpecName: "util") pod "dca6741b-bc4d-4d35-b337-9e2e0861d6e2" (UID: "dca6741b-bc4d-4d35-b337-9e2e0861d6e2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:46:19.477846 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:19.477818    2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dca6741b-bc4d-4d35-b337-9e2e0861d6e2-util\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:46:19.477846 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:19.477844    2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dca6741b-bc4d-4d35-b337-9e2e0861d6e2-bundle\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:46:19.477954 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:19.477859    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zftxl\" (UniqueName: \"kubernetes.io/projected/dca6741b-bc4d-4d35-b337-9e2e0861d6e2-kube-api-access-zftxl\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:46:20.083541 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:20.083517    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tltjf"
Sep 10 20:46:20.083693 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:20.083518    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tltjf" event={"ID":"dca6741b-bc4d-4d35-b337-9e2e0861d6e2","Type":"ContainerDied","Data":"29f5bd01651061f4a78455e1f64c42eb1e89f8e108d1e69da180e0721e755a40"}
Sep 10 20:46:20.083693 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:46:20.083642    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29f5bd01651061f4a78455e1f64c42eb1e89f8e108d1e69da180e0721e755a40"
Sep 10 20:47:41.278098 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.278068    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-rxxgg"]
Sep 10 20:47:41.278495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.278354    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d179db8-f756-493c-876d-5dd905eb506b" containerName="extract"
Sep 10 20:47:41.278495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.278366    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d179db8-f756-493c-876d-5dd905eb506b" containerName="extract"
Sep 10 20:47:41.278495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.278378    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9eff462-3f3c-4b54-a7b0-6d5b265909ca" containerName="util"
Sep 10 20:47:41.278495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.278383    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9eff462-3f3c-4b54-a7b0-6d5b265909ca" containerName="util"
Sep 10 20:47:41.278495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.278390    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d71ad7b8-ca50-412a-ac68-06a362b4f64d" containerName="extract"
Sep 10 20:47:41.278495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.278395    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d71ad7b8-ca50-412a-ac68-06a362b4f64d" containerName="extract"
Sep 10 20:47:41.278495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.278401    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9eff462-3f3c-4b54-a7b0-6d5b265909ca" containerName="extract"
Sep 10 20:47:41.278495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.278406    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9eff462-3f3c-4b54-a7b0-6d5b265909ca" containerName="extract"
Sep 10 20:47:41.278495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.278414    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dca6741b-bc4d-4d35-b337-9e2e0861d6e2" containerName="extract"
Sep 10 20:47:41.278495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.278419    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dca6741b-bc4d-4d35-b337-9e2e0861d6e2" containerName="extract"
Sep 10 20:47:41.278495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.278425    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dca6741b-bc4d-4d35-b337-9e2e0861d6e2" containerName="util"
Sep 10 20:47:41.278495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.278430    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dca6741b-bc4d-4d35-b337-9e2e0861d6e2" containerName="util"
Sep 10 20:47:41.278495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.278436    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9eff462-3f3c-4b54-a7b0-6d5b265909ca" containerName="pull"
Sep 10 20:47:41.278495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.278441    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9eff462-3f3c-4b54-a7b0-6d5b265909ca" containerName="pull"
Sep 10 20:47:41.278495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.278446    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d179db8-f756-493c-876d-5dd905eb506b" containerName="util"
Sep 10 20:47:41.278495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.278451    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d179db8-f756-493c-876d-5dd905eb506b" containerName="util"
Sep 10 20:47:41.278495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.278456    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dca6741b-bc4d-4d35-b337-9e2e0861d6e2" containerName="pull"
Sep 10 20:47:41.278495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.278461    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dca6741b-bc4d-4d35-b337-9e2e0861d6e2" containerName="pull"
Sep 10 20:47:41.278495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.278469    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d71ad7b8-ca50-412a-ac68-06a362b4f64d" containerName="util"
Sep 10 20:47:41.278495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.278473    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d71ad7b8-ca50-412a-ac68-06a362b4f64d" containerName="util"
Sep 10 20:47:41.278495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.278479    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d179db8-f756-493c-876d-5dd905eb506b" containerName="pull"
Sep 10 20:47:41.278495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.278483    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d179db8-f756-493c-876d-5dd905eb506b" containerName="pull"
Sep 10 20:47:41.278495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.278490    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d71ad7b8-ca50-412a-ac68-06a362b4f64d" containerName="pull"
Sep 10 20:47:41.278495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.278495    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d71ad7b8-ca50-412a-ac68-06a362b4f64d" containerName="pull"
Sep 10 20:47:41.279182 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.278538    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d71ad7b8-ca50-412a-ac68-06a362b4f64d" containerName="extract"
Sep 10 20:47:41.279182 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.278546    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9eff462-3f3c-4b54-a7b0-6d5b265909ca" containerName="extract"
Sep 10 20:47:41.279182 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.278552    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d179db8-f756-493c-876d-5dd905eb506b" containerName="extract"
Sep 10 20:47:41.279182 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.278559    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="dca6741b-bc4d-4d35-b337-9e2e0861d6e2" containerName="extract"
Sep 10 20:47:41.281358 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.281340    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-rxxgg"
Sep 10 20:47:41.283841 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.283818    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 20:47:41.283942 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.283891    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 20:47:41.283993 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.283962    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zbpwx\""
Sep 10 20:47:41.288910 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.288888    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-rxxgg"]
Sep 10 20:47:41.330004 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.329979    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5zk8\" (UniqueName: \"kubernetes.io/projected/9cdc86da-2cc7-428b-8ff5-e505483493d9-kube-api-access-t5zk8\") pod \"authorino-59c974bd48-rxxgg\" (UID: \"9cdc86da-2cc7-428b-8ff5-e505483493d9\") " pod="kuadrant-system/authorino-59c974bd48-rxxgg"
Sep 10 20:47:41.430886 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.430854    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5zk8\" (UniqueName: \"kubernetes.io/projected/9cdc86da-2cc7-428b-8ff5-e505483493d9-kube-api-access-t5zk8\") pod \"authorino-59c974bd48-rxxgg\" (UID: \"9cdc86da-2cc7-428b-8ff5-e505483493d9\") " pod="kuadrant-system/authorino-59c974bd48-rxxgg"
Sep 10 20:47:41.438369 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.438339    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5zk8\" (UniqueName: \"kubernetes.io/projected/9cdc86da-2cc7-428b-8ff5-e505483493d9-kube-api-access-t5zk8\") pod \"authorino-59c974bd48-rxxgg\" (UID: \"9cdc86da-2cc7-428b-8ff5-e505483493d9\") " pod="kuadrant-system/authorino-59c974bd48-rxxgg"
Sep 10 20:47:41.590461 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.590438    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-rxxgg"
Sep 10 20:47:41.859091 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:41.859068    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-rxxgg"]
Sep 10 20:47:41.860645 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:47:41.860618    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9cdc86da_2cc7_428b_8ff5_e505483493d9.slice/crio-8da624727e20367a10d316eb8f5331f50db4a91fa2e98293beb4f51549aed2c1 WatchSource:0}: Error finding container 8da624727e20367a10d316eb8f5331f50db4a91fa2e98293beb4f51549aed2c1: Status 404 returned error can't find the container with id 8da624727e20367a10d316eb8f5331f50db4a91fa2e98293beb4f51549aed2c1
Sep 10 20:47:42.472203 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:42.472151    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-rxxgg" event={"ID":"9cdc86da-2cc7-428b-8ff5-e505483493d9","Type":"ContainerStarted","Data":"8da624727e20367a10d316eb8f5331f50db4a91fa2e98293beb4f51549aed2c1"}
Sep 10 20:47:44.484153 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:44.484119    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-rxxgg" event={"ID":"9cdc86da-2cc7-428b-8ff5-e505483493d9","Type":"ContainerStarted","Data":"73d5763b917f7323b660450a7f97e3288c2d69df467c10bb5d9ccb72e97130b8"}
Sep 10 20:47:44.500742 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:47:44.500689    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-rxxgg" podStartSLOduration=1.068270129 podStartE2EDuration="3.500674875s" totalImagesPullingTime="2.432404746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:47:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:47:44.499102041 +0000 UTC m=+704.464170407" watchObservedRunningTime="2026-09-10 20:47:44.500674875 +0000 UTC m=+704.465743071"
Sep 10 20:48:11.171065 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:11.170991    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7b954dcc4d-ltqm4"]
Sep 10 20:48:11.174221 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:11.174202    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b954dcc4d-ltqm4"
Sep 10 20:48:11.180426 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:11.180396    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b954dcc4d-ltqm4"]
Sep 10 20:48:11.263491 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:11.263464    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq7cw\" (UniqueName: \"kubernetes.io/projected/2b7d64f3-00cb-4bf5-b373-836a2bfa5e67-kube-api-access-hq7cw\") pod \"authorino-7b954dcc4d-ltqm4\" (UID: \"2b7d64f3-00cb-4bf5-b373-836a2bfa5e67\") " pod="kuadrant-system/authorino-7b954dcc4d-ltqm4"
Sep 10 20:48:11.364483 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:11.364449    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq7cw\" (UniqueName: \"kubernetes.io/projected/2b7d64f3-00cb-4bf5-b373-836a2bfa5e67-kube-api-access-hq7cw\") pod \"authorino-7b954dcc4d-ltqm4\" (UID: \"2b7d64f3-00cb-4bf5-b373-836a2bfa5e67\") " pod="kuadrant-system/authorino-7b954dcc4d-ltqm4"
Sep 10 20:48:11.373282 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:11.373254    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq7cw\" (UniqueName: \"kubernetes.io/projected/2b7d64f3-00cb-4bf5-b373-836a2bfa5e67-kube-api-access-hq7cw\") pod \"authorino-7b954dcc4d-ltqm4\" (UID: \"2b7d64f3-00cb-4bf5-b373-836a2bfa5e67\") " pod="kuadrant-system/authorino-7b954dcc4d-ltqm4"
Sep 10 20:48:11.484154 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:11.484093    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b954dcc4d-ltqm4"
Sep 10 20:48:11.752645 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:11.752622    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b954dcc4d-ltqm4"]
Sep 10 20:48:11.755207 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:48:11.755163    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b7d64f3_00cb_4bf5_b373_836a2bfa5e67.slice/crio-95d6c23060c0255762863d9f103050b77d984e70f97a63d5f8082158a4439f99 WatchSource:0}: Error finding container 95d6c23060c0255762863d9f103050b77d984e70f97a63d5f8082158a4439f99: Status 404 returned error can't find the container with id 95d6c23060c0255762863d9f103050b77d984e70f97a63d5f8082158a4439f99
Sep 10 20:48:11.756737 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:11.756714    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 20:48:12.617043 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:12.616969    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b954dcc4d-ltqm4" event={"ID":"2b7d64f3-00cb-4bf5-b373-836a2bfa5e67","Type":"ContainerStarted","Data":"5c64cfb001d7ca74e3d2ec74a7c07e4e29a9403fc544409fe75e23dfb335029b"}
Sep 10 20:48:12.617043 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:12.617005    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b954dcc4d-ltqm4" event={"ID":"2b7d64f3-00cb-4bf5-b373-836a2bfa5e67","Type":"ContainerStarted","Data":"95d6c23060c0255762863d9f103050b77d984e70f97a63d5f8082158a4439f99"}
Sep 10 20:48:12.632198 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:12.632140    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7b954dcc4d-ltqm4" podStartSLOduration=1.050820982 podStartE2EDuration="1.632126522s" totalImagesPullingTime="581.30554ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:48:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:48:12.632074077 +0000 UTC m=+732.597142274" watchObservedRunningTime="2026-09-10 20:48:12.632126522 +0000 UTC m=+732.597194716"
Sep 10 20:48:12.665656 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:12.665624    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-rxxgg"]
Sep 10 20:48:12.665830 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:12.665809    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-rxxgg" podUID="9cdc86da-2cc7-428b-8ff5-e505483493d9" containerName="authorino" containerID="cri-o://73d5763b917f7323b660450a7f97e3288c2d69df467c10bb5d9ccb72e97130b8" gracePeriod=30
Sep 10 20:48:12.900667 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:12.900636    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-rxxgg"
Sep 10 20:48:13.080202 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:13.080157    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5zk8\" (UniqueName: \"kubernetes.io/projected/9cdc86da-2cc7-428b-8ff5-e505483493d9-kube-api-access-t5zk8\") pod \"9cdc86da-2cc7-428b-8ff5-e505483493d9\" (UID: \"9cdc86da-2cc7-428b-8ff5-e505483493d9\") "
Sep 10 20:48:13.082299 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:13.082279    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cdc86da-2cc7-428b-8ff5-e505483493d9-kube-api-access-t5zk8" (OuterVolumeSpecName: "kube-api-access-t5zk8") pod "9cdc86da-2cc7-428b-8ff5-e505483493d9" (UID: "9cdc86da-2cc7-428b-8ff5-e505483493d9"). InnerVolumeSpecName "kube-api-access-t5zk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:48:13.180913 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:13.180832    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5zk8\" (UniqueName: \"kubernetes.io/projected/9cdc86da-2cc7-428b-8ff5-e505483493d9-kube-api-access-t5zk8\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:48:13.622826 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:13.622790    2564 generic.go:356] "Generic (PLEG): container finished" podID="9cdc86da-2cc7-428b-8ff5-e505483493d9" containerID="73d5763b917f7323b660450a7f97e3288c2d69df467c10bb5d9ccb72e97130b8" exitCode=0
Sep 10 20:48:13.623283 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:13.622832    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-rxxgg"
Sep 10 20:48:13.623283 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:13.622878    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-rxxgg" event={"ID":"9cdc86da-2cc7-428b-8ff5-e505483493d9","Type":"ContainerDied","Data":"73d5763b917f7323b660450a7f97e3288c2d69df467c10bb5d9ccb72e97130b8"}
Sep 10 20:48:13.623283 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:13.622923    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-rxxgg" event={"ID":"9cdc86da-2cc7-428b-8ff5-e505483493d9","Type":"ContainerDied","Data":"8da624727e20367a10d316eb8f5331f50db4a91fa2e98293beb4f51549aed2c1"}
Sep 10 20:48:13.623283 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:13.622949    2564 scope.go:117] "RemoveContainer" containerID="73d5763b917f7323b660450a7f97e3288c2d69df467c10bb5d9ccb72e97130b8"
Sep 10 20:48:13.633452 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:13.633435    2564 scope.go:117] "RemoveContainer" containerID="73d5763b917f7323b660450a7f97e3288c2d69df467c10bb5d9ccb72e97130b8"
Sep 10 20:48:13.633924 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:48:13.633902    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73d5763b917f7323b660450a7f97e3288c2d69df467c10bb5d9ccb72e97130b8\": container with ID starting with 73d5763b917f7323b660450a7f97e3288c2d69df467c10bb5d9ccb72e97130b8 not found: ID does not exist" containerID="73d5763b917f7323b660450a7f97e3288c2d69df467c10bb5d9ccb72e97130b8"
Sep 10 20:48:13.633997 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:13.633935    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73d5763b917f7323b660450a7f97e3288c2d69df467c10bb5d9ccb72e97130b8"} err="failed to get container status \"73d5763b917f7323b660450a7f97e3288c2d69df467c10bb5d9ccb72e97130b8\": rpc error: code = NotFound desc = could not find container \"73d5763b917f7323b660450a7f97e3288c2d69df467c10bb5d9ccb72e97130b8\": container with ID starting with 73d5763b917f7323b660450a7f97e3288c2d69df467c10bb5d9ccb72e97130b8 not found: ID does not exist"
Sep 10 20:48:13.643210 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:13.643185    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-rxxgg"]
Sep 10 20:48:13.647039 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:13.647020    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-rxxgg"]
Sep 10 20:48:14.142016 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:14.138550    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7b954dcc4d-ltqm4"]
Sep 10 20:48:14.540188 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:14.540100    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cdc86da-2cc7-428b-8ff5-e505483493d9" path="/var/lib/kubelet/pods/9cdc86da-2cc7-428b-8ff5-e505483493d9/volumes"
Sep 10 20:48:14.628781 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:14.628739    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7b954dcc4d-ltqm4" podUID="2b7d64f3-00cb-4bf5-b373-836a2bfa5e67" containerName="authorino" containerID="cri-o://5c64cfb001d7ca74e3d2ec74a7c07e4e29a9403fc544409fe75e23dfb335029b" gracePeriod=30
Sep 10 20:48:14.880686 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:14.880657    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b954dcc4d-ltqm4"
Sep 10 20:48:14.997579 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:14.997556    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hq7cw\" (UniqueName: \"kubernetes.io/projected/2b7d64f3-00cb-4bf5-b373-836a2bfa5e67-kube-api-access-hq7cw\") pod \"2b7d64f3-00cb-4bf5-b373-836a2bfa5e67\" (UID: \"2b7d64f3-00cb-4bf5-b373-836a2bfa5e67\") "
Sep 10 20:48:14.999463 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:14.999440    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b7d64f3-00cb-4bf5-b373-836a2bfa5e67-kube-api-access-hq7cw" (OuterVolumeSpecName: "kube-api-access-hq7cw") pod "2b7d64f3-00cb-4bf5-b373-836a2bfa5e67" (UID: "2b7d64f3-00cb-4bf5-b373-836a2bfa5e67"). InnerVolumeSpecName "kube-api-access-hq7cw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:48:15.098669 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:15.098645    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq7cw\" (UniqueName: \"kubernetes.io/projected/2b7d64f3-00cb-4bf5-b373-836a2bfa5e67-kube-api-access-hq7cw\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:48:15.633852 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:15.633819    2564 generic.go:356] "Generic (PLEG): container finished" podID="2b7d64f3-00cb-4bf5-b373-836a2bfa5e67" containerID="5c64cfb001d7ca74e3d2ec74a7c07e4e29a9403fc544409fe75e23dfb335029b" exitCode=0
Sep 10 20:48:15.634345 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:15.633855    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b954dcc4d-ltqm4"
Sep 10 20:48:15.634345 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:15.633878    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b954dcc4d-ltqm4" event={"ID":"2b7d64f3-00cb-4bf5-b373-836a2bfa5e67","Type":"ContainerDied","Data":"5c64cfb001d7ca74e3d2ec74a7c07e4e29a9403fc544409fe75e23dfb335029b"}
Sep 10 20:48:15.634345 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:15.633908    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b954dcc4d-ltqm4" event={"ID":"2b7d64f3-00cb-4bf5-b373-836a2bfa5e67","Type":"ContainerDied","Data":"95d6c23060c0255762863d9f103050b77d984e70f97a63d5f8082158a4439f99"}
Sep 10 20:48:15.634345 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:15.633923    2564 scope.go:117] "RemoveContainer" containerID="5c64cfb001d7ca74e3d2ec74a7c07e4e29a9403fc544409fe75e23dfb335029b"
Sep 10 20:48:15.642127 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:15.642109    2564 scope.go:117] "RemoveContainer" containerID="5c64cfb001d7ca74e3d2ec74a7c07e4e29a9403fc544409fe75e23dfb335029b"
Sep 10 20:48:15.642386 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:48:15.642370    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c64cfb001d7ca74e3d2ec74a7c07e4e29a9403fc544409fe75e23dfb335029b\": container with ID starting with 5c64cfb001d7ca74e3d2ec74a7c07e4e29a9403fc544409fe75e23dfb335029b not found: ID does not exist" containerID="5c64cfb001d7ca74e3d2ec74a7c07e4e29a9403fc544409fe75e23dfb335029b"
Sep 10 20:48:15.642440 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:15.642395    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c64cfb001d7ca74e3d2ec74a7c07e4e29a9403fc544409fe75e23dfb335029b"} err="failed to get container status \"5c64cfb001d7ca74e3d2ec74a7c07e4e29a9403fc544409fe75e23dfb335029b\": rpc error: code = NotFound desc = could not find container \"5c64cfb001d7ca74e3d2ec74a7c07e4e29a9403fc544409fe75e23dfb335029b\": container with ID starting with 5c64cfb001d7ca74e3d2ec74a7c07e4e29a9403fc544409fe75e23dfb335029b not found: ID does not exist"
Sep 10 20:48:15.654569 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:15.654549    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7b954dcc4d-ltqm4"]
Sep 10 20:48:15.658691 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:15.658670    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7b954dcc4d-ltqm4"]
Sep 10 20:48:16.540474 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:16.540441    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b7d64f3-00cb-4bf5-b373-836a2bfa5e67" path="/var/lib/kubelet/pods/2b7d64f3-00cb-4bf5-b373-836a2bfa5e67/volumes"
Sep 10 20:48:55.648528 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:55.648492    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-mcl7w"]
Sep 10 20:48:55.649012 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:55.648779    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cdc86da-2cc7-428b-8ff5-e505483493d9" containerName="authorino"
Sep 10 20:48:55.649012 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:55.648791    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cdc86da-2cc7-428b-8ff5-e505483493d9" containerName="authorino"
Sep 10 20:48:55.649012 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:55.648804    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b7d64f3-00cb-4bf5-b373-836a2bfa5e67" containerName="authorino"
Sep 10 20:48:55.649012 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:55.648810    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b7d64f3-00cb-4bf5-b373-836a2bfa5e67" containerName="authorino"
Sep 10 20:48:55.649012 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:55.648867    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b7d64f3-00cb-4bf5-b373-836a2bfa5e67" containerName="authorino"
Sep 10 20:48:55.649012 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:55.648876    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cdc86da-2cc7-428b-8ff5-e505483493d9" containerName="authorino"
Sep 10 20:48:55.651549 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:55.651525    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-mcl7w"
Sep 10 20:48:55.653804 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:55.653785    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 10 20:48:55.662540 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:55.662507    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-mcl7w"]
Sep 10 20:48:55.701037 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:55.700995    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4b72952-37d2-4c1e-8e16-36b020cf4553-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-mcl7w\" (UID: \"c4b72952-37d2-4c1e-8e16-36b020cf4553\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-mcl7w"
Sep 10 20:48:55.701228 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:55.701059    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwzv2\" (UniqueName: \"kubernetes.io/projected/c4b72952-37d2-4c1e-8e16-36b020cf4553-kube-api-access-cwzv2\") pod \"payload-pre-processing-59fbc47bf8-mcl7w\" (UID: \"c4b72952-37d2-4c1e-8e16-36b020cf4553\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-mcl7w"
Sep 10 20:48:55.802374 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:55.802330    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4b72952-37d2-4c1e-8e16-36b020cf4553-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-mcl7w\" (UID: \"c4b72952-37d2-4c1e-8e16-36b020cf4553\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-mcl7w"
Sep 10 20:48:55.802570 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:55.802513    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwzv2\" (UniqueName: \"kubernetes.io/projected/c4b72952-37d2-4c1e-8e16-36b020cf4553-kube-api-access-cwzv2\") pod \"payload-pre-processing-59fbc47bf8-mcl7w\" (UID: \"c4b72952-37d2-4c1e-8e16-36b020cf4553\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-mcl7w"
Sep 10 20:48:55.803304 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:55.803277    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4b72952-37d2-4c1e-8e16-36b020cf4553-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-mcl7w\" (UID: \"c4b72952-37d2-4c1e-8e16-36b020cf4553\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-mcl7w"
Sep 10 20:48:55.811637 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:55.811613    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwzv2\" (UniqueName: \"kubernetes.io/projected/c4b72952-37d2-4c1e-8e16-36b020cf4553-kube-api-access-cwzv2\") pod \"payload-pre-processing-59fbc47bf8-mcl7w\" (UID: \"c4b72952-37d2-4c1e-8e16-36b020cf4553\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-mcl7w"
Sep 10 20:48:55.961922 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:55.961821    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-mcl7w"
Sep 10 20:48:56.237905 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:56.237873    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-mcl7w"]
Sep 10 20:48:56.239834 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:48:56.239804    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b72952_37d2_4c1e_8e16_36b020cf4553.slice/crio-cc67f4c5d130611fe0a146f21ee66b96cc2839ceaf932f79a3be801c1f0d60c7 WatchSource:0}: Error finding container cc67f4c5d130611fe0a146f21ee66b96cc2839ceaf932f79a3be801c1f0d60c7: Status 404 returned error can't find the container with id cc67f4c5d130611fe0a146f21ee66b96cc2839ceaf932f79a3be801c1f0d60c7
Sep 10 20:48:56.838221 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:56.838183    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-mcl7w" event={"ID":"c4b72952-37d2-4c1e-8e16-36b020cf4553","Type":"ContainerStarted","Data":"cc67f4c5d130611fe0a146f21ee66b96cc2839ceaf932f79a3be801c1f0d60c7"}
Sep 10 20:48:58.853519 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:58.853488    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-mcl7w" event={"ID":"c4b72952-37d2-4c1e-8e16-36b020cf4553","Type":"ContainerStarted","Data":"3cd9b4be2023bee1abc28513a83eab0a841dd94d21279dcda51305a64e85fa68"}
Sep 10 20:48:58.853892 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:58.853583    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-mcl7w"
Sep 10 20:48:58.872254 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:48:58.872209    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-mcl7w" podStartSLOduration=2.18073123 podStartE2EDuration="3.87219244s" totalImagesPullingTime="1.69146121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:48:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:48:58.871064672 +0000 UTC m=+778.836132867" watchObservedRunningTime="2026-09-10 20:48:58.87219244 +0000 UTC m=+778.837260637"
Sep 10 20:49:09.860675 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:09.860642    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-mcl7w"
Sep 10 20:49:24.982971 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:24.982892    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn"]
Sep 10 20:49:24.987239 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:24.987215    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn"
Sep 10 20:49:24.989532 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:24.989507    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 20:49:24.989645 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:24.989515    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 20:49:24.990041 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:24.990015    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 10 20:49:24.990147 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:24.990046    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kzwmk\""
Sep 10 20:49:25.003726 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:25.003671    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn"]
Sep 10 20:49:25.139243 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:25.139197    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7da88b3d-5776-4896-aa4f-e025abc7940e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn\" (UID: \"7da88b3d-5776-4896-aa4f-e025abc7940e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn"
Sep 10 20:49:25.139243 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:25.139250    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7da88b3d-5776-4896-aa4f-e025abc7940e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn\" (UID: \"7da88b3d-5776-4896-aa4f-e025abc7940e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn"
Sep 10 20:49:25.139506 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:25.139302    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7da88b3d-5776-4896-aa4f-e025abc7940e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn\" (UID: \"7da88b3d-5776-4896-aa4f-e025abc7940e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn"
Sep 10 20:49:25.139506 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:25.139323    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7da88b3d-5776-4896-aa4f-e025abc7940e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn\" (UID: \"7da88b3d-5776-4896-aa4f-e025abc7940e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn"
Sep 10 20:49:25.139506 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:25.139405    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmbbr\" (UniqueName: \"kubernetes.io/projected/7da88b3d-5776-4896-aa4f-e025abc7940e-kube-api-access-mmbbr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn\" (UID: \"7da88b3d-5776-4896-aa4f-e025abc7940e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn"
Sep 10 20:49:25.139506 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:25.139453    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7da88b3d-5776-4896-aa4f-e025abc7940e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn\" (UID: \"7da88b3d-5776-4896-aa4f-e025abc7940e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn"
Sep 10 20:49:25.240624 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:25.240516    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7da88b3d-5776-4896-aa4f-e025abc7940e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn\" (UID: \"7da88b3d-5776-4896-aa4f-e025abc7940e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn"
Sep 10 20:49:25.240624 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:25.240566    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7da88b3d-5776-4896-aa4f-e025abc7940e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn\" (UID: \"7da88b3d-5776-4896-aa4f-e025abc7940e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn"
Sep 10 20:49:25.240624 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:25.240598    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7da88b3d-5776-4896-aa4f-e025abc7940e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn\" (UID: \"7da88b3d-5776-4896-aa4f-e025abc7940e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn"
Sep 10 20:49:25.240624 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:25.240619    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7da88b3d-5776-4896-aa4f-e025abc7940e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn\" (UID: \"7da88b3d-5776-4896-aa4f-e025abc7940e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn"
Sep 10 20:49:25.240954 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:25.240645    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmbbr\" (UniqueName: \"kubernetes.io/projected/7da88b3d-5776-4896-aa4f-e025abc7940e-kube-api-access-mmbbr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn\" (UID: \"7da88b3d-5776-4896-aa4f-e025abc7940e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn"
Sep 10 20:49:25.240954 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:25.240675    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7da88b3d-5776-4896-aa4f-e025abc7940e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn\" (UID: \"7da88b3d-5776-4896-aa4f-e025abc7940e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn"
Sep 10 20:49:25.241115 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:25.241083    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7da88b3d-5776-4896-aa4f-e025abc7940e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn\" (UID: \"7da88b3d-5776-4896-aa4f-e025abc7940e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn"
Sep 10 20:49:25.241200 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:25.241099    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7da88b3d-5776-4896-aa4f-e025abc7940e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn\" (UID: \"7da88b3d-5776-4896-aa4f-e025abc7940e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn"
Sep 10 20:49:25.241268 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:25.241205    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7da88b3d-5776-4896-aa4f-e025abc7940e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn\" (UID: \"7da88b3d-5776-4896-aa4f-e025abc7940e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn"
Sep 10 20:49:25.242973 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:25.242952    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7da88b3d-5776-4896-aa4f-e025abc7940e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn\" (UID: \"7da88b3d-5776-4896-aa4f-e025abc7940e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn"
Sep 10 20:49:25.243088 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:25.243069    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7da88b3d-5776-4896-aa4f-e025abc7940e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn\" (UID: \"7da88b3d-5776-4896-aa4f-e025abc7940e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn"
Sep 10 20:49:25.248253 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:25.248228    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmbbr\" (UniqueName: \"kubernetes.io/projected/7da88b3d-5776-4896-aa4f-e025abc7940e-kube-api-access-mmbbr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn\" (UID: \"7da88b3d-5776-4896-aa4f-e025abc7940e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn"
Sep 10 20:49:25.304702 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:25.304645    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn"
Sep 10 20:49:25.580475 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:25.580443    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn"]
Sep 10 20:49:25.582207 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:49:25.582157    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7da88b3d_5776_4896_aa4f_e025abc7940e.slice/crio-2146c98b369a94d224357dd762173e3ec16b3baeb38d2875381cacb2db9b2cb8 WatchSource:0}: Error finding container 2146c98b369a94d224357dd762173e3ec16b3baeb38d2875381cacb2db9b2cb8: Status 404 returned error can't find the container with id 2146c98b369a94d224357dd762173e3ec16b3baeb38d2875381cacb2db9b2cb8
Sep 10 20:49:25.987220 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:25.987112    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn" event={"ID":"7da88b3d-5776-4896-aa4f-e025abc7940e","Type":"ContainerStarted","Data":"2146c98b369a94d224357dd762173e3ec16b3baeb38d2875381cacb2db9b2cb8"}
Sep 10 20:49:26.994143 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:26.994055    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn" event={"ID":"7da88b3d-5776-4896-aa4f-e025abc7940e","Type":"ContainerStarted","Data":"c399308966cf4e169880759801ba94506a2a4a580401882b4a9807ff45e69a06"}
Sep 10 20:49:27.012918 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:27.012856    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn" podStartSLOduration=1.909863337 podStartE2EDuration="3.012838585s" totalImagesPullingTime="1.102975248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:49:27.010412741 +0000 UTC m=+806.975480936" watchObservedRunningTime="2026-09-10 20:49:27.012838585 +0000 UTC m=+806.977906780"
Sep 10 20:49:35.305185 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:35.305132    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn"
Sep 10 20:49:35.305592 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:35.305228    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn"
Sep 10 20:49:35.317832 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:35.317800    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn"
Sep 10 20:49:36.049302 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:49:36.049267    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn"
Sep 10 20:50:17.796013 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:17.795978    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-dgl5d"]
Sep 10 20:50:17.800282 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:17.800263    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dgl5d"
Sep 10 20:50:17.802265 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:17.802244    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 10 20:50:17.807219 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:17.807162    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-dgl5d"]
Sep 10 20:50:17.951270 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:17.951231    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qns8n\" (UniqueName: \"kubernetes.io/projected/1da53b5b-3b85-460e-8846-1e90d264e21b-kube-api-access-qns8n\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dgl5d\" (UID: \"1da53b5b-3b85-460e-8846-1e90d264e21b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dgl5d"
Sep 10 20:50:17.951434 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:17.951285    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1da53b5b-3b85-460e-8846-1e90d264e21b-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dgl5d\" (UID: \"1da53b5b-3b85-460e-8846-1e90d264e21b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dgl5d"
Sep 10 20:50:17.951434 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:17.951343    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1da53b5b-3b85-460e-8846-1e90d264e21b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dgl5d\" (UID: \"1da53b5b-3b85-460e-8846-1e90d264e21b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dgl5d"
Sep 10 20:50:17.951434 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:17.951400    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1da53b5b-3b85-460e-8846-1e90d264e21b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dgl5d\" (UID: \"1da53b5b-3b85-460e-8846-1e90d264e21b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dgl5d"
Sep 10 20:50:17.951538 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:17.951437    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1da53b5b-3b85-460e-8846-1e90d264e21b-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dgl5d\" (UID: \"1da53b5b-3b85-460e-8846-1e90d264e21b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dgl5d"
Sep 10 20:50:17.951538 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:17.951451    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1da53b5b-3b85-460e-8846-1e90d264e21b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dgl5d\" (UID: \"1da53b5b-3b85-460e-8846-1e90d264e21b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dgl5d"
Sep 10 20:50:18.052744 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:18.052637    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1da53b5b-3b85-460e-8846-1e90d264e21b-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dgl5d\" (UID: \"1da53b5b-3b85-460e-8846-1e90d264e21b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dgl5d"
Sep 10 20:50:18.052744 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:18.052701    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1da53b5b-3b85-460e-8846-1e90d264e21b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dgl5d\" (UID: \"1da53b5b-3b85-460e-8846-1e90d264e21b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dgl5d"
Sep 10 20:50:18.052744 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:18.052736    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1da53b5b-3b85-460e-8846-1e90d264e21b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dgl5d\" (UID: \"1da53b5b-3b85-460e-8846-1e90d264e21b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dgl5d"
Sep 10 20:50:18.052999 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:18.052796    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1da53b5b-3b85-460e-8846-1e90d264e21b-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dgl5d\" (UID: \"1da53b5b-3b85-460e-8846-1e90d264e21b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dgl5d"
Sep 10 20:50:18.052999 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:18.052818    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1da53b5b-3b85-460e-8846-1e90d264e21b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dgl5d\" (UID: \"1da53b5b-3b85-460e-8846-1e90d264e21b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dgl5d"
Sep 10 20:50:18.052999 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:18.052865    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qns8n\" (UniqueName: \"kubernetes.io/projected/1da53b5b-3b85-460e-8846-1e90d264e21b-kube-api-access-qns8n\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dgl5d\" (UID: \"1da53b5b-3b85-460e-8846-1e90d264e21b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dgl5d"
Sep 10 20:50:18.053213 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:18.053151    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1da53b5b-3b85-460e-8846-1e90d264e21b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dgl5d\" (UID: \"1da53b5b-3b85-460e-8846-1e90d264e21b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dgl5d"
Sep 10 20:50:18.053339 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:18.053256    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1da53b5b-3b85-460e-8846-1e90d264e21b-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dgl5d\" (UID: \"1da53b5b-3b85-460e-8846-1e90d264e21b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dgl5d"
Sep 10 20:50:18.053339 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:18.053283    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1da53b5b-3b85-460e-8846-1e90d264e21b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dgl5d\" (UID: \"1da53b5b-3b85-460e-8846-1e90d264e21b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dgl5d"
Sep 10 20:50:18.054925 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:18.054895    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1da53b5b-3b85-460e-8846-1e90d264e21b-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dgl5d\" (UID: \"1da53b5b-3b85-460e-8846-1e90d264e21b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dgl5d"
Sep 10 20:50:18.055274 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:18.055253    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1da53b5b-3b85-460e-8846-1e90d264e21b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dgl5d\" (UID: \"1da53b5b-3b85-460e-8846-1e90d264e21b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dgl5d"
Sep 10 20:50:18.061588 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:18.061566    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qns8n\" (UniqueName: \"kubernetes.io/projected/1da53b5b-3b85-460e-8846-1e90d264e21b-kube-api-access-qns8n\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dgl5d\" (UID: \"1da53b5b-3b85-460e-8846-1e90d264e21b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dgl5d"
Sep 10 20:50:18.110627 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:18.110585    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dgl5d"
Sep 10 20:50:18.385554 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:18.385527    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-dgl5d"]
Sep 10 20:50:18.387632 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:50:18.387604    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da53b5b_3b85_460e_8846_1e90d264e21b.slice/crio-6152e316a0244bf83e0abcc144652f001d16f5aa987ea154533bef34c6551f40 WatchSource:0}: Error finding container 6152e316a0244bf83e0abcc144652f001d16f5aa987ea154533bef34c6551f40: Status 404 returned error can't find the container with id 6152e316a0244bf83e0abcc144652f001d16f5aa987ea154533bef34c6551f40
Sep 10 20:50:19.251110 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:19.251076    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dgl5d" event={"ID":"1da53b5b-3b85-460e-8846-1e90d264e21b","Type":"ContainerStarted","Data":"31ed7f526b32775bbdd53e3adf5cada948007038a9132537635af5342a59a41e"}
Sep 10 20:50:19.251509 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:19.251127    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dgl5d" event={"ID":"1da53b5b-3b85-460e-8846-1e90d264e21b","Type":"ContainerStarted","Data":"6152e316a0244bf83e0abcc144652f001d16f5aa987ea154533bef34c6551f40"}
Sep 10 20:50:19.271238 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:19.271124    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dgl5d" podStartSLOduration=2.056249595 podStartE2EDuration="2.271109593s" totalImagesPullingTime="214.859998ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:50:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:50:19.269381267 +0000 UTC m=+859.234449474" watchObservedRunningTime="2026-09-10 20:50:19.271109593 +0000 UTC m=+859.236177787"
Sep 10 20:50:28.111417 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:28.111374    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dgl5d"
Sep 10 20:50:28.111937 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:28.111440    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dgl5d"
Sep 10 20:50:28.124137 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:28.124103    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dgl5d"
Sep 10 20:50:28.305117 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:28.305084    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dgl5d"
Sep 10 20:50:46.501512 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:46.501474    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-dzczl"]
Sep 10 20:50:46.504944 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:46.504921    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dzczl"
Sep 10 20:50:46.507435 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:46.507412    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 10 20:50:46.514433 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:46.514406    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-dzczl"]
Sep 10 20:50:46.663623 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:46.663595    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7820f223-3f55-48b0-8750-6e9a8440f32b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dzczl\" (UID: \"7820f223-3f55-48b0-8750-6e9a8440f32b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dzczl"
Sep 10 20:50:46.663744 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:46.663632    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtnq6\" (UniqueName: \"kubernetes.io/projected/7820f223-3f55-48b0-8750-6e9a8440f32b-kube-api-access-mtnq6\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dzczl\" (UID: \"7820f223-3f55-48b0-8750-6e9a8440f32b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dzczl"
Sep 10 20:50:46.663744 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:46.663665    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7820f223-3f55-48b0-8750-6e9a8440f32b-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dzczl\" (UID: \"7820f223-3f55-48b0-8750-6e9a8440f32b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dzczl"
Sep 10 20:50:46.663837 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:46.663757    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7820f223-3f55-48b0-8750-6e9a8440f32b-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dzczl\" (UID: \"7820f223-3f55-48b0-8750-6e9a8440f32b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dzczl"
Sep 10 20:50:46.663837 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:46.663781    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7820f223-3f55-48b0-8750-6e9a8440f32b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dzczl\" (UID: \"7820f223-3f55-48b0-8750-6e9a8440f32b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dzczl"
Sep 10 20:50:46.663921 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:46.663837    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7820f223-3f55-48b0-8750-6e9a8440f32b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dzczl\" (UID: \"7820f223-3f55-48b0-8750-6e9a8440f32b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dzczl"
Sep 10 20:50:46.764417 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:46.764356    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7820f223-3f55-48b0-8750-6e9a8440f32b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dzczl\" (UID: \"7820f223-3f55-48b0-8750-6e9a8440f32b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dzczl"
Sep 10 20:50:46.764417 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:46.764394    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7820f223-3f55-48b0-8750-6e9a8440f32b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dzczl\" (UID: \"7820f223-3f55-48b0-8750-6e9a8440f32b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dzczl"
Sep 10 20:50:46.764417 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:46.764412    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtnq6\" (UniqueName: \"kubernetes.io/projected/7820f223-3f55-48b0-8750-6e9a8440f32b-kube-api-access-mtnq6\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dzczl\" (UID: \"7820f223-3f55-48b0-8750-6e9a8440f32b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dzczl"
Sep 10 20:50:46.764643 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:46.764436    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7820f223-3f55-48b0-8750-6e9a8440f32b-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dzczl\" (UID: \"7820f223-3f55-48b0-8750-6e9a8440f32b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dzczl"
Sep 10 20:50:46.764643 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:46.764459    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7820f223-3f55-48b0-8750-6e9a8440f32b-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dzczl\" (UID: \"7820f223-3f55-48b0-8750-6e9a8440f32b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dzczl"
Sep 10 20:50:46.764643 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:46.764589    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7820f223-3f55-48b0-8750-6e9a8440f32b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dzczl\" (UID: \"7820f223-3f55-48b0-8750-6e9a8440f32b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dzczl"
Sep 10 20:50:46.764842 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:46.764822    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7820f223-3f55-48b0-8750-6e9a8440f32b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dzczl\" (UID: \"7820f223-3f55-48b0-8750-6e9a8440f32b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dzczl"
Sep 10 20:50:46.764902 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:46.764844    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7820f223-3f55-48b0-8750-6e9a8440f32b-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dzczl\" (UID: \"7820f223-3f55-48b0-8750-6e9a8440f32b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dzczl"
Sep 10 20:50:46.764941 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:46.764906    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7820f223-3f55-48b0-8750-6e9a8440f32b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dzczl\" (UID: \"7820f223-3f55-48b0-8750-6e9a8440f32b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dzczl"
Sep 10 20:50:46.766732 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:46.766712    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7820f223-3f55-48b0-8750-6e9a8440f32b-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dzczl\" (UID: \"7820f223-3f55-48b0-8750-6e9a8440f32b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dzczl"
Sep 10 20:50:46.767020 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:46.766998    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7820f223-3f55-48b0-8750-6e9a8440f32b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dzczl\" (UID: \"7820f223-3f55-48b0-8750-6e9a8440f32b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dzczl"
Sep 10 20:50:46.771773 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:46.771748    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtnq6\" (UniqueName: \"kubernetes.io/projected/7820f223-3f55-48b0-8750-6e9a8440f32b-kube-api-access-mtnq6\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dzczl\" (UID: \"7820f223-3f55-48b0-8750-6e9a8440f32b\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dzczl"
Sep 10 20:50:46.816285 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:46.816258    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dzczl"
Sep 10 20:50:47.087792 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:47.087766    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-dzczl"]
Sep 10 20:50:47.089880 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:50:47.089849    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7820f223_3f55_48b0_8750_6e9a8440f32b.slice/crio-c1dbc51a4cb93b370598ae0c4abd0ca46c72fcc325dbf0ec0f647c3ee98740a6 WatchSource:0}: Error finding container c1dbc51a4cb93b370598ae0c4abd0ca46c72fcc325dbf0ec0f647c3ee98740a6: Status 404 returned error can't find the container with id c1dbc51a4cb93b370598ae0c4abd0ca46c72fcc325dbf0ec0f647c3ee98740a6
Sep 10 20:50:47.392940 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:47.392879    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dzczl" event={"ID":"7820f223-3f55-48b0-8750-6e9a8440f32b","Type":"ContainerStarted","Data":"b14ff7c00ee70b89af53108b53ed2de8e692489fdff48b1b3e8d3cc92f25165f"}
Sep 10 20:50:47.392940 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:47.392919    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dzczl" event={"ID":"7820f223-3f55-48b0-8750-6e9a8440f32b","Type":"ContainerStarted","Data":"c1dbc51a4cb93b370598ae0c4abd0ca46c72fcc325dbf0ec0f647c3ee98740a6"}
Sep 10 20:50:47.411144 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:47.411099    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dzczl" podStartSLOduration=1.220177466 podStartE2EDuration="1.411083062s" totalImagesPullingTime="190.905596ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:50:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:50:47.409715429 +0000 UTC m=+887.374783626" watchObservedRunningTime="2026-09-10 20:50:47.411083062 +0000 UTC m=+887.376151256"
Sep 10 20:50:56.817242 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:56.817132    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dzczl"
Sep 10 20:50:56.817242 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:56.817204    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dzczl"
Sep 10 20:50:56.830435 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:56.830404    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dzczl"
Sep 10 20:50:57.462048 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:50:57.462012    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dzczl"
Sep 10 20:51:49.164664 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:51:49.164628    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7cbb6b7559-r9k87"]
Sep 10 20:51:49.167754 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:51:49.167732    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cbb6b7559-r9k87"
Sep 10 20:51:49.169918 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:51:49.169896    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 20:51:49.170014 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:51:49.169998    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 20:51:49.170628 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:51:49.170610    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zbpwx\""
Sep 10 20:51:49.170744 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:51:49.170636    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 20:51:49.176182 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:51:49.176150    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cbb6b7559-r9k87"]
Sep 10 20:51:49.214198 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:51:49.214156    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eee5736a-14cc-46ad-9842-193b42c8190a-tls-cert\") pod \"authorino-7cbb6b7559-r9k87\" (UID: \"eee5736a-14cc-46ad-9842-193b42c8190a\") " pod="kuadrant-system/authorino-7cbb6b7559-r9k87"
Sep 10 20:51:49.214302 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:51:49.214274    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pjrn\" (UniqueName: \"kubernetes.io/projected/eee5736a-14cc-46ad-9842-193b42c8190a-kube-api-access-4pjrn\") pod \"authorino-7cbb6b7559-r9k87\" (UID: \"eee5736a-14cc-46ad-9842-193b42c8190a\") " pod="kuadrant-system/authorino-7cbb6b7559-r9k87"
Sep 10 20:51:49.314930 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:51:49.314907    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pjrn\" (UniqueName: \"kubernetes.io/projected/eee5736a-14cc-46ad-9842-193b42c8190a-kube-api-access-4pjrn\") pod \"authorino-7cbb6b7559-r9k87\" (UID: \"eee5736a-14cc-46ad-9842-193b42c8190a\") " pod="kuadrant-system/authorino-7cbb6b7559-r9k87"
Sep 10 20:51:49.315030 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:51:49.314950    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eee5736a-14cc-46ad-9842-193b42c8190a-tls-cert\") pod \"authorino-7cbb6b7559-r9k87\" (UID: \"eee5736a-14cc-46ad-9842-193b42c8190a\") " pod="kuadrant-system/authorino-7cbb6b7559-r9k87"
Sep 10 20:51:49.317160 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:51:49.317139    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eee5736a-14cc-46ad-9842-193b42c8190a-tls-cert\") pod \"authorino-7cbb6b7559-r9k87\" (UID: \"eee5736a-14cc-46ad-9842-193b42c8190a\") " pod="kuadrant-system/authorino-7cbb6b7559-r9k87"
Sep 10 20:51:49.323529 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:51:49.323510    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pjrn\" (UniqueName: \"kubernetes.io/projected/eee5736a-14cc-46ad-9842-193b42c8190a-kube-api-access-4pjrn\") pod \"authorino-7cbb6b7559-r9k87\" (UID: \"eee5736a-14cc-46ad-9842-193b42c8190a\") " pod="kuadrant-system/authorino-7cbb6b7559-r9k87"
Sep 10 20:51:49.476848 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:51:49.476791    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cbb6b7559-r9k87"
Sep 10 20:51:49.745851 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:51:49.745824    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cbb6b7559-r9k87"]
Sep 10 20:51:49.747907 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:51:49.747876    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeee5736a_14cc_46ad_9842_193b42c8190a.slice/crio-d00f82f6802973e98566ce2137ecab53028033d5172fc5020cb1b451a436ffd8 WatchSource:0}: Error finding container d00f82f6802973e98566ce2137ecab53028033d5172fc5020cb1b451a436ffd8: Status 404 returned error can't find the container with id d00f82f6802973e98566ce2137ecab53028033d5172fc5020cb1b451a436ffd8
Sep 10 20:51:50.707132 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:51:50.707107    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cbb6b7559-r9k87" event={"ID":"eee5736a-14cc-46ad-9842-193b42c8190a","Type":"ContainerStarted","Data":"d00f82f6802973e98566ce2137ecab53028033d5172fc5020cb1b451a436ffd8"}
Sep 10 20:51:51.713873 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:51:51.713834    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cbb6b7559-r9k87" event={"ID":"eee5736a-14cc-46ad-9842-193b42c8190a","Type":"ContainerStarted","Data":"2e0f2817093102686ad4b34b9c2fd14de710863e9c0b91caa86b35ba5de5f633"}
Sep 10 20:51:51.732503 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:51:51.732449    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7cbb6b7559-r9k87" podStartSLOduration=1.8179231470000001 podStartE2EDuration="2.732433787s" totalImagesPullingTime="914.51064ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:51:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:51:51.729669232 +0000 UTC m=+951.694737428" watchObservedRunningTime="2026-09-10 20:51:51.732433787 +0000 UTC m=+951.697502001"
Sep 10 20:52:16.663750 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:16.663714    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"]
Sep 10 20:52:16.667285 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:16.667257    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:16.669395 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:16.669370    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-80aa25-openshift-default-dockercfg-vrwpj\""
Sep 10 20:52:16.680098 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:16.680076    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"]
Sep 10 20:52:16.833785 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:16.833754    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac33a9c-11a1-4a91-a458-67697f1642ba-workload-certs\") pod \"e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") " pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:16.833938 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:16.833791    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac33a9c-11a1-4a91-a458-67697f1642ba-istiod-ca-cert\") pod \"e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") " pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:16.833938 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:16.833816    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkpxh\" (UniqueName: \"kubernetes.io/projected/3ac33a9c-11a1-4a91-a458-67697f1642ba-kube-api-access-rkpxh\") pod \"e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") " pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:16.833938 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:16.833899    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac33a9c-11a1-4a91-a458-67697f1642ba-istio-envoy\") pod \"e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") " pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:16.834047 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:16.833944    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac33a9c-11a1-4a91-a458-67697f1642ba-istio-data\") pod \"e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") " pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:16.834047 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:16.833980    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac33a9c-11a1-4a91-a458-67697f1642ba-workload-socket\") pod \"e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") " pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:16.834047 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:16.834022    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac33a9c-11a1-4a91-a458-67697f1642ba-credential-socket\") pod \"e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") " pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:16.834199 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:16.834046    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac33a9c-11a1-4a91-a458-67697f1642ba-istio-podinfo\") pod \"e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") " pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:16.834199 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:16.834069    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ac33a9c-11a1-4a91-a458-67697f1642ba-istio-token\") pod \"e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") " pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:16.934533 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:16.934452    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac33a9c-11a1-4a91-a458-67697f1642ba-workload-certs\") pod \"e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") " pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:16.934533 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:16.934489    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac33a9c-11a1-4a91-a458-67697f1642ba-istiod-ca-cert\") pod \"e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") " pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:16.934533 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:16.934511    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkpxh\" (UniqueName: \"kubernetes.io/projected/3ac33a9c-11a1-4a91-a458-67697f1642ba-kube-api-access-rkpxh\") pod \"e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") " pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:16.934804 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:16.934558    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac33a9c-11a1-4a91-a458-67697f1642ba-istio-envoy\") pod \"e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") " pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:16.934804 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:16.934596    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac33a9c-11a1-4a91-a458-67697f1642ba-istio-data\") pod \"e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") " pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:16.934804 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:16.934638    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac33a9c-11a1-4a91-a458-67697f1642ba-workload-socket\") pod \"e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") " pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:16.934804 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:16.934698    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac33a9c-11a1-4a91-a458-67697f1642ba-credential-socket\") pod \"e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") " pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:16.934804 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:16.934725    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac33a9c-11a1-4a91-a458-67697f1642ba-istio-podinfo\") pod \"e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") " pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:16.934804 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:16.934757    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ac33a9c-11a1-4a91-a458-67697f1642ba-istio-token\") pod \"e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") " pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:16.935108 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:16.934903    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac33a9c-11a1-4a91-a458-67697f1642ba-workload-certs\") pod \"e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") " pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:16.935264 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:16.935206    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac33a9c-11a1-4a91-a458-67697f1642ba-credential-socket\") pod \"e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") " pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:16.935374 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:16.935258    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac33a9c-11a1-4a91-a458-67697f1642ba-istio-data\") pod \"e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") " pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:16.935671 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:16.935648    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac33a9c-11a1-4a91-a458-67697f1642ba-workload-socket\") pod \"e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") " pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:16.935842 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:16.935817    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac33a9c-11a1-4a91-a458-67697f1642ba-istiod-ca-cert\") pod \"e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") " pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:16.937830 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:16.937804    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac33a9c-11a1-4a91-a458-67697f1642ba-istio-envoy\") pod \"e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") " pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:16.937994 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:16.937977    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac33a9c-11a1-4a91-a458-67697f1642ba-istio-podinfo\") pod \"e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") " pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:16.942283 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:16.942263    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ac33a9c-11a1-4a91-a458-67697f1642ba-istio-token\") pod \"e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") " pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:16.942441 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:16.942419    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkpxh\" (UniqueName: \"kubernetes.io/projected/3ac33a9c-11a1-4a91-a458-67697f1642ba-kube-api-access-rkpxh\") pod \"e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") " pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:16.977382 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:16.977361    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:17.252120 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:17.252095    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"]
Sep 10 20:52:17.254103 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:52:17.254067    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac33a9c_11a1_4a91_a458_67697f1642ba.slice/crio-7ff7aac2627b7553eb5d416fab32a8171bc4c03a6882bb28d178a744ea6a0392 WatchSource:0}: Error finding container 7ff7aac2627b7553eb5d416fab32a8171bc4c03a6882bb28d178a744ea6a0392: Status 404 returned error can't find the container with id 7ff7aac2627b7553eb5d416fab32a8171bc4c03a6882bb28d178a744ea6a0392
Sep 10 20:52:17.256301 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:17.256255    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 20:52:17.256365 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:17.256348    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 20:52:17.256407 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:17.256389    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 20:52:17.835361 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:17.835322    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww" event={"ID":"3ac33a9c-11a1-4a91-a458-67697f1642ba","Type":"ContainerStarted","Data":"7428e9dde8f934a68076e08d5a74e369edf0a9d54f4f38ba9f6d196fdac8d613"}
Sep 10 20:52:17.835799 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:17.835375    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww" event={"ID":"3ac33a9c-11a1-4a91-a458-67697f1642ba","Type":"ContainerStarted","Data":"7ff7aac2627b7553eb5d416fab32a8171bc4c03a6882bb28d178a744ea6a0392"}
Sep 10 20:52:17.876320 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:17.876277    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww" podStartSLOduration=1.8762547459999999 podStartE2EDuration="1.876254746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:52:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:52:17.875740199 +0000 UTC m=+977.840808395" watchObservedRunningTime="2026-09-10 20:52:17.876254746 +0000 UTC m=+977.841322942"
Sep 10 20:52:17.978393 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:17.978367    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:18.982502 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:18.982470    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:19.844909 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:19.844871    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:19.845760 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:19.845743    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:21.798559 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:21.798531    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-80aa25-f5c49c754-7kwng"]
Sep 10 20:52:21.801748 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:21.801731    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80aa25-f5c49c754-7kwng"
Sep 10 20:52:21.804031 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:21.804008    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-80aa25-dockercfg-rnjfh\""
Sep 10 20:52:21.804206 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:21.804188    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-80aa25\""
Sep 10 20:52:21.813953 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:21.813925    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-80aa25-f5c49c754-7kwng"]
Sep 10 20:52:21.977230 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:21.977195    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmb9c\" (UniqueName: \"kubernetes.io/projected/f0e77222-8445-4c91-8999-8686eb8dabb4-kube-api-access-mmb9c\") pod \"payload-pre-processing-e2e-inf-a-80aa25-f5c49c754-7kwng\" (UID: \"f0e77222-8445-4c91-8999-8686eb8dabb4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80aa25-f5c49c754-7kwng"
Sep 10 20:52:21.977405 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:21.977378    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e77222-8445-4c91-8999-8686eb8dabb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-80aa25-f5c49c754-7kwng\" (UID: \"f0e77222-8445-4c91-8999-8686eb8dabb4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80aa25-f5c49c754-7kwng"
Sep 10 20:52:22.078380 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:22.078351    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmb9c\" (UniqueName: \"kubernetes.io/projected/f0e77222-8445-4c91-8999-8686eb8dabb4-kube-api-access-mmb9c\") pod \"payload-pre-processing-e2e-inf-a-80aa25-f5c49c754-7kwng\" (UID: \"f0e77222-8445-4c91-8999-8686eb8dabb4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80aa25-f5c49c754-7kwng"
Sep 10 20:52:22.078513 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:22.078420    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e77222-8445-4c91-8999-8686eb8dabb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-80aa25-f5c49c754-7kwng\" (UID: \"f0e77222-8445-4c91-8999-8686eb8dabb4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80aa25-f5c49c754-7kwng"
Sep 10 20:52:22.079094 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:22.079073    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e77222-8445-4c91-8999-8686eb8dabb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-80aa25-f5c49c754-7kwng\" (UID: \"f0e77222-8445-4c91-8999-8686eb8dabb4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80aa25-f5c49c754-7kwng"
Sep 10 20:52:22.085979 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:22.085957    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmb9c\" (UniqueName: \"kubernetes.io/projected/f0e77222-8445-4c91-8999-8686eb8dabb4-kube-api-access-mmb9c\") pod \"payload-pre-processing-e2e-inf-a-80aa25-f5c49c754-7kwng\" (UID: \"f0e77222-8445-4c91-8999-8686eb8dabb4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80aa25-f5c49c754-7kwng"
Sep 10 20:52:22.113812 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:22.113788    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80aa25-f5c49c754-7kwng"
Sep 10 20:52:22.384371 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:22.384348    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-80aa25-f5c49c754-7kwng"]
Sep 10 20:52:22.386980 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:52:22.386945    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0e77222_8445_4c91_8999_8686eb8dabb4.slice/crio-a90e32e3e1d4e07864d0b8261631eb74226a571376bebbb9e433ccd3452b6a6b WatchSource:0}: Error finding container a90e32e3e1d4e07864d0b8261631eb74226a571376bebbb9e433ccd3452b6a6b: Status 404 returned error can't find the container with id a90e32e3e1d4e07864d0b8261631eb74226a571376bebbb9e433ccd3452b6a6b
Sep 10 20:52:22.864403 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:22.864362    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80aa25-f5c49c754-7kwng" event={"ID":"f0e77222-8445-4c91-8999-8686eb8dabb4","Type":"ContainerStarted","Data":"76e57605e83c03c79d55c24fabd1ee9207595a2fa2348c343a321c4ba5ef6df1"}
Sep 10 20:52:22.864861 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:22.864462    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80aa25-f5c49c754-7kwng" event={"ID":"f0e77222-8445-4c91-8999-8686eb8dabb4","Type":"ContainerStarted","Data":"a90e32e3e1d4e07864d0b8261631eb74226a571376bebbb9e433ccd3452b6a6b"}
Sep 10 20:52:22.864861 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:22.864496    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80aa25-f5c49c754-7kwng"
Sep 10 20:52:22.880064 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:22.880013    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80aa25-f5c49c754-7kwng" podStartSLOduration=1.8799683790000001 podStartE2EDuration="1.879968379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:52:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:52:22.879793144 +0000 UTC m=+982.844861339" watchObservedRunningTime="2026-09-10 20:52:22.879968379 +0000 UTC m=+982.845036575"
Sep 10 20:52:25.912418 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:25.912385    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"]
Sep 10 20:52:25.915794 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:25.915772    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:25.918357 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:25.918335    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-5d9b44-openshift-default-dockercfg-hlpcv\""
Sep 10 20:52:25.928994 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:25.928970    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"]
Sep 10 20:52:26.008602 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.008572    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c5a180c-786f-4556-8a89-ca0714d01bba-istiod-ca-cert\") pod \"e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") " pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:26.008764 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.008612    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c5a180c-786f-4556-8a89-ca0714d01bba-istio-envoy\") pod \"e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") " pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:26.008764 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.008631    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c5a180c-786f-4556-8a89-ca0714d01bba-istio-token\") pod \"e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") " pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:26.008764 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.008713    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c5a180c-786f-4556-8a89-ca0714d01bba-credential-socket\") pod \"e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") " pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:26.008764 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.008745    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c5a180c-786f-4556-8a89-ca0714d01bba-workload-socket\") pod \"e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") " pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:26.008913 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.008766    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c5a180c-786f-4556-8a89-ca0714d01bba-istio-podinfo\") pod \"e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") " pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:26.008913 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.008790    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c5a180c-786f-4556-8a89-ca0714d01bba-workload-certs\") pod \"e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") " pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:26.008913 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.008809    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncz5v\" (UniqueName: \"kubernetes.io/projected/3c5a180c-786f-4556-8a89-ca0714d01bba-kube-api-access-ncz5v\") pod \"e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") " pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:26.008913 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.008839    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c5a180c-786f-4556-8a89-ca0714d01bba-istio-data\") pod \"e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") " pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:26.109296 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.109263    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c5a180c-786f-4556-8a89-ca0714d01bba-istio-envoy\") pod \"e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") " pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:26.109296 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.109298    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c5a180c-786f-4556-8a89-ca0714d01bba-istio-token\") pod \"e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") " pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:26.109542 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.109330    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c5a180c-786f-4556-8a89-ca0714d01bba-credential-socket\") pod \"e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") " pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:26.109542 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.109360    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c5a180c-786f-4556-8a89-ca0714d01bba-workload-socket\") pod \"e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") " pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:26.109542 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.109383    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c5a180c-786f-4556-8a89-ca0714d01bba-istio-podinfo\") pod \"e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") " pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:26.109542 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.109403    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c5a180c-786f-4556-8a89-ca0714d01bba-workload-certs\") pod \"e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") " pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:26.109542 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.109425    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncz5v\" (UniqueName: \"kubernetes.io/projected/3c5a180c-786f-4556-8a89-ca0714d01bba-kube-api-access-ncz5v\") pod \"e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") " pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:26.109877 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.109621    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c5a180c-786f-4556-8a89-ca0714d01bba-istio-data\") pod \"e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") " pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:26.109877 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.109661    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c5a180c-786f-4556-8a89-ca0714d01bba-istiod-ca-cert\") pod \"e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") " pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:26.109877 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.109798    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c5a180c-786f-4556-8a89-ca0714d01bba-credential-socket\") pod \"e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") " pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:26.110076 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.110043    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c5a180c-786f-4556-8a89-ca0714d01bba-workload-certs\") pod \"e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") " pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:26.110132 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.110081    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c5a180c-786f-4556-8a89-ca0714d01bba-istio-data\") pod \"e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") " pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:26.110290 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.110264    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c5a180c-786f-4556-8a89-ca0714d01bba-workload-socket\") pod \"e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") " pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:26.110718 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.110686    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c5a180c-786f-4556-8a89-ca0714d01bba-istiod-ca-cert\") pod \"e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") " pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:26.112014 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.111956    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c5a180c-786f-4556-8a89-ca0714d01bba-istio-envoy\") pod \"e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") " pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:26.112014 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.112001    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c5a180c-786f-4556-8a89-ca0714d01bba-istio-podinfo\") pod \"e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") " pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:26.127461 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.127439    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncz5v\" (UniqueName: \"kubernetes.io/projected/3c5a180c-786f-4556-8a89-ca0714d01bba-kube-api-access-ncz5v\") pod \"e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") " pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:26.131752 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.131729    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c5a180c-786f-4556-8a89-ca0714d01bba-istio-token\") pod \"e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") " pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:26.226945 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.226851    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:26.503266 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.503242    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"]
Sep 10 20:52:26.505018 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:52:26.504995    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c5a180c_786f_4556_8a89_ca0714d01bba.slice/crio-b8f653795f0b55eeda89eb9e3ade8d5fa59240d78310a562af9a836774b51618 WatchSource:0}: Error finding container b8f653795f0b55eeda89eb9e3ade8d5fa59240d78310a562af9a836774b51618: Status 404 returned error can't find the container with id b8f653795f0b55eeda89eb9e3ade8d5fa59240d78310a562af9a836774b51618
Sep 10 20:52:26.507135 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.507099    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 20:52:26.507242 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.507179    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 20:52:26.507242 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.507212    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 20:52:26.886183 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.886134    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl" event={"ID":"3c5a180c-786f-4556-8a89-ca0714d01bba","Type":"ContainerStarted","Data":"b0a511b28ef37f474bc70b7400e4972581e8e66547e9341cd8629b9c702d1602"}
Sep 10 20:52:26.886372 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.886207    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl" event={"ID":"3c5a180c-786f-4556-8a89-ca0714d01bba","Type":"ContainerStarted","Data":"b8f653795f0b55eeda89eb9e3ade8d5fa59240d78310a562af9a836774b51618"}
Sep 10 20:52:26.908499 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:26.908440    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl" podStartSLOduration=1.908416648 podStartE2EDuration="1.908416648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:52:26.906625983 +0000 UTC m=+986.871694188" watchObservedRunningTime="2026-09-10 20:52:26.908416648 +0000 UTC m=+986.873484844"
Sep 10 20:52:27.227351 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:27.227265    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:27.233263 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:27.233237    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:27.892450 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:27.892417    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:27.893681 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:27.893652    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:29.956403 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:29.956362    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c62a32a3-cc6999685-rbkd4"]
Sep 10 20:52:29.960365 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:29.960330    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c62a32a3-cc6999685-rbkd4"
Sep 10 20:52:29.967411 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:29.967380    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c62a32a3\""
Sep 10 20:52:29.968371 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:29.968344    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c62a32a3-dockercfg-sj6mk\""
Sep 10 20:52:29.972987 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:29.972954    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c62a32a3-cc6999685-rbkd4"]
Sep 10 20:52:30.046315 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:30.046279    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536f3210-bfa7-4bbb-83b2-fc4909e3938c-plugins-config-volume\") pod \"payload-processing-e2e-ait-c62a32a3-cc6999685-rbkd4\" (UID: \"536f3210-bfa7-4bbb-83b2-fc4909e3938c\") " pod="openshift-ingress/payload-processing-e2e-ait-c62a32a3-cc6999685-rbkd4"
Sep 10 20:52:30.046482 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:30.046366    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbnlq\" (UniqueName: \"kubernetes.io/projected/536f3210-bfa7-4bbb-83b2-fc4909e3938c-kube-api-access-hbnlq\") pod \"payload-processing-e2e-ait-c62a32a3-cc6999685-rbkd4\" (UID: \"536f3210-bfa7-4bbb-83b2-fc4909e3938c\") " pod="openshift-ingress/payload-processing-e2e-ait-c62a32a3-cc6999685-rbkd4"
Sep 10 20:52:30.146922 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:30.146872    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536f3210-bfa7-4bbb-83b2-fc4909e3938c-plugins-config-volume\") pod \"payload-processing-e2e-ait-c62a32a3-cc6999685-rbkd4\" (UID: \"536f3210-bfa7-4bbb-83b2-fc4909e3938c\") " pod="openshift-ingress/payload-processing-e2e-ait-c62a32a3-cc6999685-rbkd4"
Sep 10 20:52:30.147090 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:30.146980    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbnlq\" (UniqueName: \"kubernetes.io/projected/536f3210-bfa7-4bbb-83b2-fc4909e3938c-kube-api-access-hbnlq\") pod \"payload-processing-e2e-ait-c62a32a3-cc6999685-rbkd4\" (UID: \"536f3210-bfa7-4bbb-83b2-fc4909e3938c\") " pod="openshift-ingress/payload-processing-e2e-ait-c62a32a3-cc6999685-rbkd4"
Sep 10 20:52:30.147591 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:30.147571    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536f3210-bfa7-4bbb-83b2-fc4909e3938c-plugins-config-volume\") pod \"payload-processing-e2e-ait-c62a32a3-cc6999685-rbkd4\" (UID: \"536f3210-bfa7-4bbb-83b2-fc4909e3938c\") " pod="openshift-ingress/payload-processing-e2e-ait-c62a32a3-cc6999685-rbkd4"
Sep 10 20:52:30.170193 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:30.169696    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbnlq\" (UniqueName: \"kubernetes.io/projected/536f3210-bfa7-4bbb-83b2-fc4909e3938c-kube-api-access-hbnlq\") pod \"payload-processing-e2e-ait-c62a32a3-cc6999685-rbkd4\" (UID: \"536f3210-bfa7-4bbb-83b2-fc4909e3938c\") " pod="openshift-ingress/payload-processing-e2e-ait-c62a32a3-cc6999685-rbkd4"
Sep 10 20:52:30.282318 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:30.282234    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c62a32a3-cc6999685-rbkd4"
Sep 10 20:52:30.579269 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:30.578233    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c62a32a3-cc6999685-rbkd4"]
Sep 10 20:52:30.910864 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:30.910775    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c62a32a3-cc6999685-rbkd4" event={"ID":"536f3210-bfa7-4bbb-83b2-fc4909e3938c","Type":"ContainerStarted","Data":"947b99a45b8e10aa9b8e48ed6bee23474a019723af2e6492c5c595a4c0fb838f"}
Sep 10 20:52:30.910864 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:30.910829    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c62a32a3-cc6999685-rbkd4" event={"ID":"536f3210-bfa7-4bbb-83b2-fc4909e3938c","Type":"ContainerStarted","Data":"1747e1d696065405181cb3e3f34f072d1cbcfa73f1f82264b0403146d855f80a"}
Sep 10 20:52:30.911072 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:30.910873    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c62a32a3-cc6999685-rbkd4"
Sep 10 20:52:30.927649 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:30.927595    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c62a32a3-cc6999685-rbkd4" podStartSLOduration=1.927551331 podStartE2EDuration="1.927551331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:52:30.926523573 +0000 UTC m=+990.891591767" watchObservedRunningTime="2026-09-10 20:52:30.927551331 +0000 UTC m=+990.892619528"
Sep 10 20:52:33.872377 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:33.872337    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80aa25-f5c49c754-7kwng"
Sep 10 20:52:35.887352 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:35.887307    2564 patch_prober.go:28] interesting pod/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 20:52:35.887768 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:35.887387    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww" podUID="3ac33a9c-11a1-4a91-a458-67697f1642ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 20:52:36.944685 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:36.944650    2564 generic.go:356] "Generic (PLEG): container finished" podID="3ac33a9c-11a1-4a91-a458-67697f1642ba" containerID="7428e9dde8f934a68076e08d5a74e369edf0a9d54f4f38ba9f6d196fdac8d613" exitCode=0
Sep 10 20:52:36.945086 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:36.944722    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww" event={"ID":"3ac33a9c-11a1-4a91-a458-67697f1642ba","Type":"ContainerDied","Data":"7428e9dde8f934a68076e08d5a74e369edf0a9d54f4f38ba9f6d196fdac8d613"}
Sep 10 20:52:36.945137 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:36.945122    2564 scope.go:117] "RemoveContainer" containerID="7428e9dde8f934a68076e08d5a74e369edf0a9d54f4f38ba9f6d196fdac8d613"
Sep 10 20:52:36.946108 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:36.946079    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 20:52:36.946237 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:36.946146    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 20:52:36.946237 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:36.946197    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 20:52:37.951421 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:37.951386    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww" event={"ID":"3ac33a9c-11a1-4a91-a458-67697f1642ba","Type":"ContainerStarted","Data":"90bea3965e202b4ae504fa4e099448d82ce00262b8df5728c1277b96b2e78ef3"}
Sep 10 20:52:37.977670 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:37.977625    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:38.978673 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:38.978622    2564 patch_prober.go:28] interesting pod/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 10 20:52:38.979232 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:38.978696    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww" podUID="3ac33a9c-11a1-4a91-a458-67697f1642ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 10 20:52:39.977767 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:39.977721    2564 patch_prober.go:28] interesting pod/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 10 20:52:39.977971 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:39.977792    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww" podUID="3ac33a9c-11a1-4a91-a458-67697f1642ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 10 20:52:40.978643 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:40.978607    2564 patch_prober.go:28] interesting pod/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 10 20:52:40.979160 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:40.979130    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww" podUID="3ac33a9c-11a1-4a91-a458-67697f1642ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 10 20:52:41.917871 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:41.917835    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c62a32a3-cc6999685-rbkd4"
Sep 10 20:52:41.983688 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:41.983650    2564 patch_prober.go:28] interesting pod/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 10 20:52:41.984276 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:41.984235    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww" podUID="3ac33a9c-11a1-4a91-a458-67697f1642ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 10 20:52:42.582296 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:42.582251    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4"]
Sep 10 20:52:42.586031 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:42.585998    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4"
Sep 10 20:52:42.588092 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:42.588060    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-711c424e-kserve-self-signed-certs\""
Sep 10 20:52:42.596357 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:42.596327    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4"]
Sep 10 20:52:42.676192 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:42.673217    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/660816da-324e-45e0-8fbb-45737c13a97a-tmp-dir\") pod \"e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4\" (UID: \"660816da-324e-45e0-8fbb-45737c13a97a\") " pod="llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4"
Sep 10 20:52:42.676192 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:42.673288    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/660816da-324e-45e0-8fbb-45737c13a97a-dshm\") pod \"e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4\" (UID: \"660816da-324e-45e0-8fbb-45737c13a97a\") " pod="llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4"
Sep 10 20:52:42.676192 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:42.673331    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/660816da-324e-45e0-8fbb-45737c13a97a-home\") pod \"e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4\" (UID: \"660816da-324e-45e0-8fbb-45737c13a97a\") " pod="llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4"
Sep 10 20:52:42.676192 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:42.673356    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/660816da-324e-45e0-8fbb-45737c13a97a-model-cache\") pod \"e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4\" (UID: \"660816da-324e-45e0-8fbb-45737c13a97a\") " pod="llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4"
Sep 10 20:52:42.676192 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:42.673446    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/660816da-324e-45e0-8fbb-45737c13a97a-tls-certs\") pod \"e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4\" (UID: \"660816da-324e-45e0-8fbb-45737c13a97a\") " pod="llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4"
Sep 10 20:52:42.676192 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:42.673473    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pkj5\" (UniqueName: \"kubernetes.io/projected/660816da-324e-45e0-8fbb-45737c13a97a-kube-api-access-5pkj5\") pod \"e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4\" (UID: \"660816da-324e-45e0-8fbb-45737c13a97a\") " pod="llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4"
Sep 10 20:52:42.776674 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:42.776634    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/660816da-324e-45e0-8fbb-45737c13a97a-tls-certs\") pod \"e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4\" (UID: \"660816da-324e-45e0-8fbb-45737c13a97a\") " pod="llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4"
Sep 10 20:52:42.777603 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:42.777580    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pkj5\" (UniqueName: \"kubernetes.io/projected/660816da-324e-45e0-8fbb-45737c13a97a-kube-api-access-5pkj5\") pod \"e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4\" (UID: \"660816da-324e-45e0-8fbb-45737c13a97a\") " pod="llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4"
Sep 10 20:52:42.778200 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:42.778159    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/660816da-324e-45e0-8fbb-45737c13a97a-tmp-dir\") pod \"e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4\" (UID: \"660816da-324e-45e0-8fbb-45737c13a97a\") " pod="llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4"
Sep 10 20:52:42.778819 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:42.778801    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/660816da-324e-45e0-8fbb-45737c13a97a-dshm\") pod \"e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4\" (UID: \"660816da-324e-45e0-8fbb-45737c13a97a\") " pod="llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4"
Sep 10 20:52:42.779317 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:42.779298    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/660816da-324e-45e0-8fbb-45737c13a97a-home\") pod \"e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4\" (UID: \"660816da-324e-45e0-8fbb-45737c13a97a\") " pod="llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4"
Sep 10 20:52:42.779772 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:42.779745    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/660816da-324e-45e0-8fbb-45737c13a97a-model-cache\") pod \"e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4\" (UID: \"660816da-324e-45e0-8fbb-45737c13a97a\") " pod="llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4"
Sep 10 20:52:42.782578 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:42.779679    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/660816da-324e-45e0-8fbb-45737c13a97a-home\") pod \"e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4\" (UID: \"660816da-324e-45e0-8fbb-45737c13a97a\") " pod="llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4"
Sep 10 20:52:42.782772 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:42.778672    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/660816da-324e-45e0-8fbb-45737c13a97a-tmp-dir\") pod \"e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4\" (UID: \"660816da-324e-45e0-8fbb-45737c13a97a\") " pod="llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4"
Sep 10 20:52:42.782884 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:42.781926    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/660816da-324e-45e0-8fbb-45737c13a97a-model-cache\") pod \"e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4\" (UID: \"660816da-324e-45e0-8fbb-45737c13a97a\") " pod="llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4"
Sep 10 20:52:42.782980 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:42.782523    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/660816da-324e-45e0-8fbb-45737c13a97a-tls-certs\") pod \"e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4\" (UID: \"660816da-324e-45e0-8fbb-45737c13a97a\") " pod="llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4"
Sep 10 20:52:42.799849 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:42.799824    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pkj5\" (UniqueName: \"kubernetes.io/projected/660816da-324e-45e0-8fbb-45737c13a97a-kube-api-access-5pkj5\") pod \"e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4\" (UID: \"660816da-324e-45e0-8fbb-45737c13a97a\") " pod="llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4"
Sep 10 20:52:42.809471 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:42.809443    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/660816da-324e-45e0-8fbb-45737c13a97a-dshm\") pod \"e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4\" (UID: \"660816da-324e-45e0-8fbb-45737c13a97a\") " pod="llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4"
Sep 10 20:52:42.902675 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:42.902532    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4"
Sep 10 20:52:42.981192 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:42.978424    2564 patch_prober.go:28] interesting pod/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Sep 10 20:52:42.981192 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:42.978488    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww" podUID="3ac33a9c-11a1-4a91-a458-67697f1642ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Sep 10 20:52:43.201704 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:43.201678    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4"]
Sep 10 20:52:43.204485 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:52:43.204455    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod660816da_324e_45e0_8fbb_45737c13a97a.slice/crio-bbef2c2f6ffaced879dd423c4b911d053ad7c135e845cefa2b5eb05eb9bfc86b WatchSource:0}: Error finding container bbef2c2f6ffaced879dd423c4b911d053ad7c135e845cefa2b5eb05eb9bfc86b: Status 404 returned error can't find the container with id bbef2c2f6ffaced879dd423c4b911d053ad7c135e845cefa2b5eb05eb9bfc86b
Sep 10 20:52:43.982016 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:43.981977    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:43.982404 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:43.982380    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:43.983202 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:43.983181    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:52:44.020271 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:44.020225    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4" event={"ID":"660816da-324e-45e0-8fbb-45737c13a97a","Type":"ContainerStarted","Data":"1327511be4deae9fef8ec022fe53a799a4fd2999a69bbcd7f68bf9c55a39956c"}
Sep 10 20:52:44.020271 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:44.020284    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4" event={"ID":"660816da-324e-45e0-8fbb-45737c13a97a","Type":"ContainerStarted","Data":"bbef2c2f6ffaced879dd423c4b911d053ad7c135e845cefa2b5eb05eb9bfc86b"}
Sep 10 20:52:44.038394 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:44.038338    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4" podStartSLOduration=2.038317034 podStartE2EDuration="2.038317034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:52:44.035645703 +0000 UTC m=+1004.000713899" watchObservedRunningTime="2026-09-10 20:52:44.038317034 +0000 UTC m=+1004.003385228"
Sep 10 20:52:47.035864 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:47.035831    2564 generic.go:356] "Generic (PLEG): container finished" podID="3c5a180c-786f-4556-8a89-ca0714d01bba" containerID="b0a511b28ef37f474bc70b7400e4972581e8e66547e9341cd8629b9c702d1602" exitCode=0
Sep 10 20:52:47.036455 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:47.035908    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl" event={"ID":"3c5a180c-786f-4556-8a89-ca0714d01bba","Type":"ContainerDied","Data":"b0a511b28ef37f474bc70b7400e4972581e8e66547e9341cd8629b9c702d1602"}
Sep 10 20:52:47.036455 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:47.036444    2564 scope.go:117] "RemoveContainer" containerID="b0a511b28ef37f474bc70b7400e4972581e8e66547e9341cd8629b9c702d1602"
Sep 10 20:52:47.037521 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:47.037496    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 20:52:47.037582 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:47.037564    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 20:52:47.037617 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:47.037601    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 20:52:48.042749 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:48.042712    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl" event={"ID":"3c5a180c-786f-4556-8a89-ca0714d01bba","Type":"ContainerStarted","Data":"40f6fd61590b53ef20f75570c0232b1ce6a73808f3c25cf7baf151e142daf78e"}
Sep 10 20:52:48.227192 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:48.227140    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:48.227434 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:48.227412    2564 patch_prober.go:28] interesting pod/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 10 20:52:48.227480 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:48.227462    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl" podUID="3c5a180c-786f-4556-8a89-ca0714d01bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 10 20:52:49.227787 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:49.227748    2564 patch_prober.go:28] interesting pod/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 10 20:52:49.228214 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:49.227805    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl" podUID="3c5a180c-786f-4556-8a89-ca0714d01bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 10 20:52:50.228078 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:50.228041    2564 patch_prober.go:28] interesting pod/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 10 20:52:50.228579 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:50.228113    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl" podUID="3c5a180c-786f-4556-8a89-ca0714d01bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 10 20:52:50.454789 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:50.454746    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-467fc3-7ddc48679c-dg28p"]
Sep 10 20:52:50.459121 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:50.459095    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-467fc3-7ddc48679c-dg28p"
Sep 10 20:52:50.463384 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:50.463361    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-467fc3\""
Sep 10 20:52:50.463699 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:50.463436    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-467fc3-dockercfg-ghjmr\""
Sep 10 20:52:50.487058 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:50.486987    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-467fc3-7ddc48679c-dg28p"]
Sep 10 20:52:50.567667 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:50.567627    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnqcr\" (UniqueName: \"kubernetes.io/projected/6af4ce55-b9fc-4633-a9e3-5b85dd8ad789-kube-api-access-cnqcr\") pod \"payload-processing-e2e-inf-b-467fc3-7ddc48679c-dg28p\" (UID: \"6af4ce55-b9fc-4633-a9e3-5b85dd8ad789\") " pod="openshift-ingress/payload-processing-e2e-inf-b-467fc3-7ddc48679c-dg28p"
Sep 10 20:52:50.567970 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:50.567947    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6af4ce55-b9fc-4633-a9e3-5b85dd8ad789-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-467fc3-7ddc48679c-dg28p\" (UID: \"6af4ce55-b9fc-4633-a9e3-5b85dd8ad789\") " pod="openshift-ingress/payload-processing-e2e-inf-b-467fc3-7ddc48679c-dg28p"
Sep 10 20:52:50.669345 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:50.669308    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnqcr\" (UniqueName: \"kubernetes.io/projected/6af4ce55-b9fc-4633-a9e3-5b85dd8ad789-kube-api-access-cnqcr\") pod \"payload-processing-e2e-inf-b-467fc3-7ddc48679c-dg28p\" (UID: \"6af4ce55-b9fc-4633-a9e3-5b85dd8ad789\") " pod="openshift-ingress/payload-processing-e2e-inf-b-467fc3-7ddc48679c-dg28p"
Sep 10 20:52:50.670109 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:50.670079    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6af4ce55-b9fc-4633-a9e3-5b85dd8ad789-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-467fc3-7ddc48679c-dg28p\" (UID: \"6af4ce55-b9fc-4633-a9e3-5b85dd8ad789\") " pod="openshift-ingress/payload-processing-e2e-inf-b-467fc3-7ddc48679c-dg28p"
Sep 10 20:52:50.671785 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:50.671689    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6af4ce55-b9fc-4633-a9e3-5b85dd8ad789-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-467fc3-7ddc48679c-dg28p\" (UID: \"6af4ce55-b9fc-4633-a9e3-5b85dd8ad789\") " pod="openshift-ingress/payload-processing-e2e-inf-b-467fc3-7ddc48679c-dg28p"
Sep 10 20:52:50.680810 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:50.680780    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnqcr\" (UniqueName: \"kubernetes.io/projected/6af4ce55-b9fc-4633-a9e3-5b85dd8ad789-kube-api-access-cnqcr\") pod \"payload-processing-e2e-inf-b-467fc3-7ddc48679c-dg28p\" (UID: \"6af4ce55-b9fc-4633-a9e3-5b85dd8ad789\") " pod="openshift-ingress/payload-processing-e2e-inf-b-467fc3-7ddc48679c-dg28p"
Sep 10 20:52:50.776337 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:50.776237    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-467fc3-7ddc48679c-dg28p"
Sep 10 20:52:51.052386 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:51.052276    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-467fc3-7ddc48679c-dg28p"]
Sep 10 20:52:51.053687 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:52:51.053656    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af4ce55_b9fc_4633_a9e3_5b85dd8ad789.slice/crio-c08a0791dfb12561f6fab7033029231004d530e8fdd66e1fc8cd53f9cdcd4e02 WatchSource:0}: Error finding container c08a0791dfb12561f6fab7033029231004d530e8fdd66e1fc8cd53f9cdcd4e02: Status 404 returned error can't find the container with id c08a0791dfb12561f6fab7033029231004d530e8fdd66e1fc8cd53f9cdcd4e02
Sep 10 20:52:51.076027 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:51.075930    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-467fc3-7ddc48679c-dg28p" event={"ID":"6af4ce55-b9fc-4633-a9e3-5b85dd8ad789","Type":"ContainerStarted","Data":"c08a0791dfb12561f6fab7033029231004d530e8fdd66e1fc8cd53f9cdcd4e02"}
Sep 10 20:52:51.227268 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:51.227231    2564 patch_prober.go:28] interesting pod/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 10 20:52:51.227460 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:51.227295    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl" podUID="3c5a180c-786f-4556-8a89-ca0714d01bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 10 20:52:52.086803 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:52.086758    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-467fc3-7ddc48679c-dg28p" event={"ID":"6af4ce55-b9fc-4633-a9e3-5b85dd8ad789","Type":"ContainerStarted","Data":"34da3b5b8491cb7f890fb17f3d70c8c18dc7325cd331e79d8816dfcbbb2a1c71"}
Sep 10 20:52:52.087307 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:52.087030    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-467fc3-7ddc48679c-dg28p"
Sep 10 20:52:52.106160 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:52.106104    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-467fc3-7ddc48679c-dg28p" podStartSLOduration=2.106090737 podStartE2EDuration="2.106090737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:52:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:52:52.104595219 +0000 UTC m=+1012.069663413" watchObservedRunningTime="2026-09-10 20:52:52.106090737 +0000 UTC m=+1012.071158932"
Sep 10 20:52:52.227576 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:52.227540    2564 patch_prober.go:28] interesting pod/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 10 20:52:52.227840 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:52.227627    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl" podUID="3c5a180c-786f-4556-8a89-ca0714d01bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 10 20:52:52.902756 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:52.902708    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4"
Sep 10 20:52:52.902952 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:52.902772    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4"
Sep 10 20:52:52.915429 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:52.915401    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4"
Sep 10 20:52:53.109131 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:53.109100    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4"
Sep 10 20:52:53.231959 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:53.231867    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:53.232219 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:53.232196    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:52:53.233001 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:52:53.232980    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:53:03.099288 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:03.099256    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-467fc3-7ddc48679c-dg28p"
Sep 10 20:53:10.106254 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:10.106216    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c62a32a3-cc6999685-rbkd4"]
Sep 10 20:53:10.106839 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:10.106445    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c62a32a3-cc6999685-rbkd4" podUID="536f3210-bfa7-4bbb-83b2-fc4909e3938c" containerName="payload-processing" containerID="cri-o://947b99a45b8e10aa9b8e48ed6bee23474a019723af2e6492c5c595a4c0fb838f" gracePeriod=30
Sep 10 20:53:10.375336 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:10.375252    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c62a32a3-cc6999685-rbkd4"
Sep 10 20:53:10.425072 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:10.425039    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbnlq\" (UniqueName: \"kubernetes.io/projected/536f3210-bfa7-4bbb-83b2-fc4909e3938c-kube-api-access-hbnlq\") pod \"536f3210-bfa7-4bbb-83b2-fc4909e3938c\" (UID: \"536f3210-bfa7-4bbb-83b2-fc4909e3938c\") "
Sep 10 20:53:10.425294 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:10.425109    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536f3210-bfa7-4bbb-83b2-fc4909e3938c-plugins-config-volume\") pod \"536f3210-bfa7-4bbb-83b2-fc4909e3938c\" (UID: \"536f3210-bfa7-4bbb-83b2-fc4909e3938c\") "
Sep 10 20:53:10.425616 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:10.425588    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536f3210-bfa7-4bbb-83b2-fc4909e3938c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "536f3210-bfa7-4bbb-83b2-fc4909e3938c" (UID: "536f3210-bfa7-4bbb-83b2-fc4909e3938c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:53:10.427625 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:10.427592    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536f3210-bfa7-4bbb-83b2-fc4909e3938c-kube-api-access-hbnlq" (OuterVolumeSpecName: "kube-api-access-hbnlq") pod "536f3210-bfa7-4bbb-83b2-fc4909e3938c" (UID: "536f3210-bfa7-4bbb-83b2-fc4909e3938c"). InnerVolumeSpecName "kube-api-access-hbnlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:53:10.526608 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:10.526570    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536f3210-bfa7-4bbb-83b2-fc4909e3938c-plugins-config-volume\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:53:10.526608 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:10.526600    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbnlq\" (UniqueName: \"kubernetes.io/projected/536f3210-bfa7-4bbb-83b2-fc4909e3938c-kube-api-access-hbnlq\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:53:11.193600 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.193566    2564 generic.go:356] "Generic (PLEG): container finished" podID="536f3210-bfa7-4bbb-83b2-fc4909e3938c" containerID="947b99a45b8e10aa9b8e48ed6bee23474a019723af2e6492c5c595a4c0fb838f" exitCode=0
Sep 10 20:53:11.194127 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.193616    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c62a32a3-cc6999685-rbkd4"
Sep 10 20:53:11.194127 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.193644    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c62a32a3-cc6999685-rbkd4" event={"ID":"536f3210-bfa7-4bbb-83b2-fc4909e3938c","Type":"ContainerDied","Data":"947b99a45b8e10aa9b8e48ed6bee23474a019723af2e6492c5c595a4c0fb838f"}
Sep 10 20:53:11.194127 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.193699    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c62a32a3-cc6999685-rbkd4" event={"ID":"536f3210-bfa7-4bbb-83b2-fc4909e3938c","Type":"ContainerDied","Data":"1747e1d696065405181cb3e3f34f072d1cbcfa73f1f82264b0403146d855f80a"}
Sep 10 20:53:11.194127 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.193723    2564 scope.go:117] "RemoveContainer" containerID="947b99a45b8e10aa9b8e48ed6bee23474a019723af2e6492c5c595a4c0fb838f"
Sep 10 20:53:11.202658 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.202639    2564 scope.go:117] "RemoveContainer" containerID="947b99a45b8e10aa9b8e48ed6bee23474a019723af2e6492c5c595a4c0fb838f"
Sep 10 20:53:11.202927 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:53:11.202908    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"947b99a45b8e10aa9b8e48ed6bee23474a019723af2e6492c5c595a4c0fb838f\": container with ID starting with 947b99a45b8e10aa9b8e48ed6bee23474a019723af2e6492c5c595a4c0fb838f not found: ID does not exist" containerID="947b99a45b8e10aa9b8e48ed6bee23474a019723af2e6492c5c595a4c0fb838f"
Sep 10 20:53:11.202972 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.202937    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"947b99a45b8e10aa9b8e48ed6bee23474a019723af2e6492c5c595a4c0fb838f"} err="failed to get container status \"947b99a45b8e10aa9b8e48ed6bee23474a019723af2e6492c5c595a4c0fb838f\": rpc error: code = NotFound desc = could not find container \"947b99a45b8e10aa9b8e48ed6bee23474a019723af2e6492c5c595a4c0fb838f\": container with ID starting with 947b99a45b8e10aa9b8e48ed6bee23474a019723af2e6492c5c595a4c0fb838f not found: ID does not exist"
Sep 10 20:53:11.230662 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.230643    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c62a32a3-cc6999685-rbkd4"]
Sep 10 20:53:11.237352 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.237333    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c62a32a3-cc6999685-rbkd4"]
Sep 10 20:53:11.397067 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.397028    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"]
Sep 10 20:53:11.397408 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.397393    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="536f3210-bfa7-4bbb-83b2-fc4909e3938c" containerName="payload-processing"
Sep 10 20:53:11.397485 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.397409    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="536f3210-bfa7-4bbb-83b2-fc4909e3938c" containerName="payload-processing"
Sep 10 20:53:11.397485 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.397477    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="536f3210-bfa7-4bbb-83b2-fc4909e3938c" containerName="payload-processing"
Sep 10 20:53:11.401634 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.401614    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:11.403835 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.403814    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-090c5b75-gw-openshift-default-dockercfg-fwrng\""
Sep 10 20:53:11.416837 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.416801    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"]
Sep 10 20:53:11.434766 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.434740    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17b95594-bfd6-4b6a-b820-9d191f981a32-istio-token\") pod \"e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") " pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:11.434895 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.434782    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17b95594-bfd6-4b6a-b820-9d191f981a32-istiod-ca-cert\") pod \"e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") " pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:11.434895 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.434810    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17b95594-bfd6-4b6a-b820-9d191f981a32-workload-certs\") pod \"e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") " pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:11.434895 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.434863    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17b95594-bfd6-4b6a-b820-9d191f981a32-istio-data\") pod \"e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") " pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:11.435012 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.434948    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b95594-bfd6-4b6a-b820-9d191f981a32-workload-socket\") pod \"e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") " pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:11.435012 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.435000    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17b95594-bfd6-4b6a-b820-9d191f981a32-istio-envoy\") pod \"e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") " pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:11.435083 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.435033    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17b95594-bfd6-4b6a-b820-9d191f981a32-istio-podinfo\") pod \"e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") " pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:11.435130 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.435091    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b95594-bfd6-4b6a-b820-9d191f981a32-credential-socket\") pod \"e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") " pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:11.435207 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.435184    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7hvb\" (UniqueName: \"kubernetes.io/projected/17b95594-bfd6-4b6a-b820-9d191f981a32-kube-api-access-b7hvb\") pod \"e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") " pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:11.536343 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.536238    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b95594-bfd6-4b6a-b820-9d191f981a32-workload-socket\") pod \"e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") " pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:11.536343 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.536322    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17b95594-bfd6-4b6a-b820-9d191f981a32-istio-envoy\") pod \"e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") " pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:11.536576 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.536357    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17b95594-bfd6-4b6a-b820-9d191f981a32-istio-podinfo\") pod \"e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") " pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:11.536576 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.536414    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b95594-bfd6-4b6a-b820-9d191f981a32-credential-socket\") pod \"e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") " pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:11.536576 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.536469    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7hvb\" (UniqueName: \"kubernetes.io/projected/17b95594-bfd6-4b6a-b820-9d191f981a32-kube-api-access-b7hvb\") pod \"e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") " pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:11.536576 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.536499    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17b95594-bfd6-4b6a-b820-9d191f981a32-istio-token\") pod \"e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") " pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:11.536576 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.536533    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17b95594-bfd6-4b6a-b820-9d191f981a32-istiod-ca-cert\") pod \"e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") " pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:11.536576 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.536561    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17b95594-bfd6-4b6a-b820-9d191f981a32-workload-certs\") pod \"e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") " pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:11.536869 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.536592    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17b95594-bfd6-4b6a-b820-9d191f981a32-istio-data\") pod \"e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") " pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:11.536869 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.536651    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b95594-bfd6-4b6a-b820-9d191f981a32-workload-socket\") pod \"e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") " pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:11.537007 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.536980    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b95594-bfd6-4b6a-b820-9d191f981a32-credential-socket\") pod \"e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") " pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:11.537343 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.537261    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17b95594-bfd6-4b6a-b820-9d191f981a32-workload-certs\") pod \"e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") " pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:11.537461 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.537421    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17b95594-bfd6-4b6a-b820-9d191f981a32-istiod-ca-cert\") pod \"e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") " pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:11.537615 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.537595    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17b95594-bfd6-4b6a-b820-9d191f981a32-istio-data\") pod \"e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") " pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:11.539552 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.539533    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17b95594-bfd6-4b6a-b820-9d191f981a32-istio-podinfo\") pod \"e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") " pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:11.539632 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.539583    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17b95594-bfd6-4b6a-b820-9d191f981a32-istio-envoy\") pod \"e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") " pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:11.546625 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.546549    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17b95594-bfd6-4b6a-b820-9d191f981a32-istio-token\") pod \"e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") " pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:11.546765 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.546742    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7hvb\" (UniqueName: \"kubernetes.io/projected/17b95594-bfd6-4b6a-b820-9d191f981a32-kube-api-access-b7hvb\") pod \"e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") " pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:11.713045 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.712994    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:11.989954 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.989926    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"]
Sep 10 20:53:11.990893 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:53:11.990868    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b95594_bfd6_4b6a_b820_9d191f981a32.slice/crio-05dfa623cf7fe7b5c7321e3930fb1059eb8ffe2b09b86c2f93cd6610ba0db524 WatchSource:0}: Error finding container 05dfa623cf7fe7b5c7321e3930fb1059eb8ffe2b09b86c2f93cd6610ba0db524: Status 404 returned error can't find the container with id 05dfa623cf7fe7b5c7321e3930fb1059eb8ffe2b09b86c2f93cd6610ba0db524
Sep 10 20:53:11.993042 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.993012    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 20:53:11.993114 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.993076    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 20:53:11.993114 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:11.993104    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 20:53:12.208462 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:12.208424    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45" event={"ID":"17b95594-bfd6-4b6a-b820-9d191f981a32","Type":"ContainerStarted","Data":"0e96aa74d5435b5dd9bb776c15c53a6bcc44ee51d60a48ed83701378cecd39ca"}
Sep 10 20:53:12.208888 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:12.208514    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45" event={"ID":"17b95594-bfd6-4b6a-b820-9d191f981a32","Type":"ContainerStarted","Data":"05dfa623cf7fe7b5c7321e3930fb1059eb8ffe2b09b86c2f93cd6610ba0db524"}
Sep 10 20:53:12.228982 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:12.228931    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45" podStartSLOduration=1.2289124359999999 podStartE2EDuration="1.228912436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:53:12.226595677 +0000 UTC m=+1032.191663895" watchObservedRunningTime="2026-09-10 20:53:12.228912436 +0000 UTC m=+1032.193980634"
Sep 10 20:53:12.542444 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:12.542401    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="536f3210-bfa7-4bbb-83b2-fc4909e3938c" path="/var/lib/kubelet/pods/536f3210-bfa7-4bbb-83b2-fc4909e3938c/volumes"
Sep 10 20:53:12.713329 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:12.713296    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:13.718962 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:13.718929    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:14.221862 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:14.221824    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:14.222650 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:14.222632    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:33.327492 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:33.327459    2564 generic.go:356] "Generic (PLEG): container finished" podID="17b95594-bfd6-4b6a-b820-9d191f981a32" containerID="0e96aa74d5435b5dd9bb776c15c53a6bcc44ee51d60a48ed83701378cecd39ca" exitCode=0
Sep 10 20:53:33.327884 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:33.327535    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45" event={"ID":"17b95594-bfd6-4b6a-b820-9d191f981a32","Type":"ContainerDied","Data":"0e96aa74d5435b5dd9bb776c15c53a6bcc44ee51d60a48ed83701378cecd39ca"}
Sep 10 20:53:33.327986 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:33.327972    2564 scope.go:117] "RemoveContainer" containerID="0e96aa74d5435b5dd9bb776c15c53a6bcc44ee51d60a48ed83701378cecd39ca"
Sep 10 20:53:33.329026 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:33.328994    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 20:53:33.329111 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:33.329066    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 20:53:33.329111 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:33.329097    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 20:53:34.335134 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:34.335098    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45" event={"ID":"17b95594-bfd6-4b6a-b820-9d191f981a32","Type":"ContainerStarted","Data":"151a2fa6f3d26b18b0d6f1893b309ad5a37ba3c791039cdeb29e7574f4d8453b"}
Sep 10 20:53:34.713549 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:34.713446    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:34.713741 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:34.713719    2564 patch_prober.go:28] interesting pod/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Sep 10 20:53:34.713785 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:34.713765    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45" podUID="17b95594-bfd6-4b6a-b820-9d191f981a32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Sep 10 20:53:35.713930 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:35.713895    2564 patch_prober.go:28] interesting pod/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Sep 10 20:53:35.714315 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:35.713950    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45" podUID="17b95594-bfd6-4b6a-b820-9d191f981a32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Sep 10 20:53:36.714151 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:36.714117    2564 patch_prober.go:28] interesting pod/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Sep 10 20:53:36.714555 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:36.714188    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45" podUID="17b95594-bfd6-4b6a-b820-9d191f981a32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Sep 10 20:53:37.714086 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:37.714051    2564 patch_prober.go:28] interesting pod/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Sep 10 20:53:37.714263 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:37.714101    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45" podUID="17b95594-bfd6-4b6a-b820-9d191f981a32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Sep 10 20:53:38.713289 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:38.713251    2564 patch_prober.go:28] interesting pod/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Sep 10 20:53:38.713454 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:38.713321    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45" podUID="17b95594-bfd6-4b6a-b820-9d191f981a32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Sep 10 20:53:39.717889 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:39.717857    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:39.718366 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:39.718252    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:39.718953 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:39.718931    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:44.903626 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:44.903586    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"]
Sep 10 20:53:44.904108 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:44.903851    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45" podUID="17b95594-bfd6-4b6a-b820-9d191f981a32" containerName="istio-proxy" containerID="cri-o://151a2fa6f3d26b18b0d6f1893b309ad5a37ba3c791039cdeb29e7574f4d8453b" gracePeriod=30
Sep 10 20:53:50.157507 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.157481    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:50.283928 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.283845    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17b95594-bfd6-4b6a-b820-9d191f981a32-workload-certs\") pod \"17b95594-bfd6-4b6a-b820-9d191f981a32\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") "
Sep 10 20:53:50.283928 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.283882    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17b95594-bfd6-4b6a-b820-9d191f981a32-istio-envoy\") pod \"17b95594-bfd6-4b6a-b820-9d191f981a32\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") "
Sep 10 20:53:50.283928 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.283920    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b95594-bfd6-4b6a-b820-9d191f981a32-credential-socket\") pod \"17b95594-bfd6-4b6a-b820-9d191f981a32\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") "
Sep 10 20:53:50.284203 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.283947    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7hvb\" (UniqueName: \"kubernetes.io/projected/17b95594-bfd6-4b6a-b820-9d191f981a32-kube-api-access-b7hvb\") pod \"17b95594-bfd6-4b6a-b820-9d191f981a32\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") "
Sep 10 20:53:50.284203 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.284052    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17b95594-bfd6-4b6a-b820-9d191f981a32-istiod-ca-cert\") pod \"17b95594-bfd6-4b6a-b820-9d191f981a32\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") "
Sep 10 20:53:50.284203 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.284080    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17b95594-bfd6-4b6a-b820-9d191f981a32-istio-token\") pod \"17b95594-bfd6-4b6a-b820-9d191f981a32\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") "
Sep 10 20:53:50.284203 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.284108    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17b95594-bfd6-4b6a-b820-9d191f981a32-istio-data\") pod \"17b95594-bfd6-4b6a-b820-9d191f981a32\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") "
Sep 10 20:53:50.284203 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.284130    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b95594-bfd6-4b6a-b820-9d191f981a32-workload-socket\") pod \"17b95594-bfd6-4b6a-b820-9d191f981a32\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") "
Sep 10 20:53:50.284203 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.284155    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17b95594-bfd6-4b6a-b820-9d191f981a32-istio-podinfo\") pod \"17b95594-bfd6-4b6a-b820-9d191f981a32\" (UID: \"17b95594-bfd6-4b6a-b820-9d191f981a32\") "
Sep 10 20:53:50.284203 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.284152    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b95594-bfd6-4b6a-b820-9d191f981a32-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "17b95594-bfd6-4b6a-b820-9d191f981a32" (UID: "17b95594-bfd6-4b6a-b820-9d191f981a32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:53:50.284577 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.284301    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b95594-bfd6-4b6a-b820-9d191f981a32-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "17b95594-bfd6-4b6a-b820-9d191f981a32" (UID: "17b95594-bfd6-4b6a-b820-9d191f981a32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:53:50.284577 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.284409    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b95594-bfd6-4b6a-b820-9d191f981a32-istio-data" (OuterVolumeSpecName: "istio-data") pod "17b95594-bfd6-4b6a-b820-9d191f981a32" (UID: "17b95594-bfd6-4b6a-b820-9d191f981a32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:53:50.284577 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.284428    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b95594-bfd6-4b6a-b820-9d191f981a32-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "17b95594-bfd6-4b6a-b820-9d191f981a32" (UID: "17b95594-bfd6-4b6a-b820-9d191f981a32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:53:50.284577 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.284533    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17b95594-bfd6-4b6a-b820-9d191f981a32-workload-certs\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:53:50.284577 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.284548    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b95594-bfd6-4b6a-b820-9d191f981a32-credential-socket\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:53:50.284577 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.284557    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17b95594-bfd6-4b6a-b820-9d191f981a32-istio-data\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:53:50.284577 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.284566    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b95594-bfd6-4b6a-b820-9d191f981a32-workload-socket\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:53:50.284983 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.284791    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17b95594-bfd6-4b6a-b820-9d191f981a32-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "17b95594-bfd6-4b6a-b820-9d191f981a32" (UID: "17b95594-bfd6-4b6a-b820-9d191f981a32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:53:50.286366 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.286341    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17b95594-bfd6-4b6a-b820-9d191f981a32-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "17b95594-bfd6-4b6a-b820-9d191f981a32" (UID: "17b95594-bfd6-4b6a-b820-9d191f981a32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 20:53:50.286366 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.286353    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17b95594-bfd6-4b6a-b820-9d191f981a32-kube-api-access-b7hvb" (OuterVolumeSpecName: "kube-api-access-b7hvb") pod "17b95594-bfd6-4b6a-b820-9d191f981a32" (UID: "17b95594-bfd6-4b6a-b820-9d191f981a32"). InnerVolumeSpecName "kube-api-access-b7hvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:53:50.286538 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.286519    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b95594-bfd6-4b6a-b820-9d191f981a32-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "17b95594-bfd6-4b6a-b820-9d191f981a32" (UID: "17b95594-bfd6-4b6a-b820-9d191f981a32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:53:50.286593 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.286556    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17b95594-bfd6-4b6a-b820-9d191f981a32-istio-token" (OuterVolumeSpecName: "istio-token") pod "17b95594-bfd6-4b6a-b820-9d191f981a32" (UID: "17b95594-bfd6-4b6a-b820-9d191f981a32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:53:50.385611 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.385576    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17b95594-bfd6-4b6a-b820-9d191f981a32-istiod-ca-cert\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:53:50.385611 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.385606    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17b95594-bfd6-4b6a-b820-9d191f981a32-istio-token\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:53:50.385611 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.385615    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17b95594-bfd6-4b6a-b820-9d191f981a32-istio-podinfo\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:53:50.385834 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.385623    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17b95594-bfd6-4b6a-b820-9d191f981a32-istio-envoy\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:53:50.385834 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.385632    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7hvb\" (UniqueName: \"kubernetes.io/projected/17b95594-bfd6-4b6a-b820-9d191f981a32-kube-api-access-b7hvb\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:53:50.424660 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.424628    2564 generic.go:356] "Generic (PLEG): container finished" podID="17b95594-bfd6-4b6a-b820-9d191f981a32" containerID="151a2fa6f3d26b18b0d6f1893b309ad5a37ba3c791039cdeb29e7574f4d8453b" exitCode=0
Sep 10 20:53:50.424841 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.424676    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"
Sep 10 20:53:50.424841 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.424717    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45" event={"ID":"17b95594-bfd6-4b6a-b820-9d191f981a32","Type":"ContainerDied","Data":"151a2fa6f3d26b18b0d6f1893b309ad5a37ba3c791039cdeb29e7574f4d8453b"}
Sep 10 20:53:50.424841 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.424780    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45" event={"ID":"17b95594-bfd6-4b6a-b820-9d191f981a32","Type":"ContainerDied","Data":"05dfa623cf7fe7b5c7321e3930fb1059eb8ffe2b09b86c2f93cd6610ba0db524"}
Sep 10 20:53:50.424841 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.424810    2564 scope.go:117] "RemoveContainer" containerID="151a2fa6f3d26b18b0d6f1893b309ad5a37ba3c791039cdeb29e7574f4d8453b"
Sep 10 20:53:50.434693 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.434674    2564 scope.go:117] "RemoveContainer" containerID="0e96aa74d5435b5dd9bb776c15c53a6bcc44ee51d60a48ed83701378cecd39ca"
Sep 10 20:53:50.442890 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.442867    2564 scope.go:117] "RemoveContainer" containerID="151a2fa6f3d26b18b0d6f1893b309ad5a37ba3c791039cdeb29e7574f4d8453b"
Sep 10 20:53:50.443150 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:53:50.443132    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"151a2fa6f3d26b18b0d6f1893b309ad5a37ba3c791039cdeb29e7574f4d8453b\": container with ID starting with 151a2fa6f3d26b18b0d6f1893b309ad5a37ba3c791039cdeb29e7574f4d8453b not found: ID does not exist" containerID="151a2fa6f3d26b18b0d6f1893b309ad5a37ba3c791039cdeb29e7574f4d8453b"
Sep 10 20:53:50.443222 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.443158    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"151a2fa6f3d26b18b0d6f1893b309ad5a37ba3c791039cdeb29e7574f4d8453b"} err="failed to get container status \"151a2fa6f3d26b18b0d6f1893b309ad5a37ba3c791039cdeb29e7574f4d8453b\": rpc error: code = NotFound desc = could not find container \"151a2fa6f3d26b18b0d6f1893b309ad5a37ba3c791039cdeb29e7574f4d8453b\": container with ID starting with 151a2fa6f3d26b18b0d6f1893b309ad5a37ba3c791039cdeb29e7574f4d8453b not found: ID does not exist"
Sep 10 20:53:50.443222 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.443195    2564 scope.go:117] "RemoveContainer" containerID="0e96aa74d5435b5dd9bb776c15c53a6bcc44ee51d60a48ed83701378cecd39ca"
Sep 10 20:53:50.443472 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:53:50.443453    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e96aa74d5435b5dd9bb776c15c53a6bcc44ee51d60a48ed83701378cecd39ca\": container with ID starting with 0e96aa74d5435b5dd9bb776c15c53a6bcc44ee51d60a48ed83701378cecd39ca not found: ID does not exist" containerID="0e96aa74d5435b5dd9bb776c15c53a6bcc44ee51d60a48ed83701378cecd39ca"
Sep 10 20:53:50.443518 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.443478    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e96aa74d5435b5dd9bb776c15c53a6bcc44ee51d60a48ed83701378cecd39ca"} err="failed to get container status \"0e96aa74d5435b5dd9bb776c15c53a6bcc44ee51d60a48ed83701378cecd39ca\": rpc error: code = NotFound desc = could not find container \"0e96aa74d5435b5dd9bb776c15c53a6bcc44ee51d60a48ed83701378cecd39ca\": container with ID starting with 0e96aa74d5435b5dd9bb776c15c53a6bcc44ee51d60a48ed83701378cecd39ca not found: ID does not exist"
Sep 10 20:53:50.457193 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.454530    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"]
Sep 10 20:53:50.461724 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.461695    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-090c5b75-gw-openshift-default-c9df95596-g6g45"]
Sep 10 20:53:50.543497 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.543404    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17b95594-bfd6-4b6a-b820-9d191f981a32" path="/var/lib/kubelet/pods/17b95594-bfd6-4b6a-b820-9d191f981a32/volumes"
Sep 10 20:53:50.586115 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.586077    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-50a00efc-5b5f999b6c-mvhtr"]
Sep 10 20:53:50.586507 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.586489    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17b95594-bfd6-4b6a-b820-9d191f981a32" containerName="istio-proxy"
Sep 10 20:53:50.586605 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.586509    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="17b95594-bfd6-4b6a-b820-9d191f981a32" containerName="istio-proxy"
Sep 10 20:53:50.586605 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.586529    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17b95594-bfd6-4b6a-b820-9d191f981a32" containerName="istio-proxy"
Sep 10 20:53:50.586605 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.586537    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="17b95594-bfd6-4b6a-b820-9d191f981a32" containerName="istio-proxy"
Sep 10 20:53:50.586757 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.586644    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="17b95594-bfd6-4b6a-b820-9d191f981a32" containerName="istio-proxy"
Sep 10 20:53:50.586757 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.586657    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="17b95594-bfd6-4b6a-b820-9d191f981a32" containerName="istio-proxy"
Sep 10 20:53:50.591199 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.591161    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-50a00efc-5b5f999b6c-mvhtr"
Sep 10 20:53:50.593758 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.593737    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-50a00efc\""
Sep 10 20:53:50.593887 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.593807    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-50a00efc-dockercfg-bzr2p\""
Sep 10 20:53:50.601744 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.601718    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-50a00efc-5b5f999b6c-mvhtr"]
Sep 10 20:53:50.688578 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.688532    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9x2q\" (UniqueName: \"kubernetes.io/projected/d8c4a0a6-ba90-4a90-9b81-7555237a6b60-kube-api-access-n9x2q\") pod \"payload-processing-e2e-ait-50a00efc-5b5f999b6c-mvhtr\" (UID: \"d8c4a0a6-ba90-4a90-9b81-7555237a6b60\") " pod="openshift-ingress/payload-processing-e2e-ait-50a00efc-5b5f999b6c-mvhtr"
Sep 10 20:53:50.688578 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.688583    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8c4a0a6-ba90-4a90-9b81-7555237a6b60-plugins-config-volume\") pod \"payload-processing-e2e-ait-50a00efc-5b5f999b6c-mvhtr\" (UID: \"d8c4a0a6-ba90-4a90-9b81-7555237a6b60\") " pod="openshift-ingress/payload-processing-e2e-ait-50a00efc-5b5f999b6c-mvhtr"
Sep 10 20:53:50.789977 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.789938    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9x2q\" (UniqueName: \"kubernetes.io/projected/d8c4a0a6-ba90-4a90-9b81-7555237a6b60-kube-api-access-n9x2q\") pod \"payload-processing-e2e-ait-50a00efc-5b5f999b6c-mvhtr\" (UID: \"d8c4a0a6-ba90-4a90-9b81-7555237a6b60\") " pod="openshift-ingress/payload-processing-e2e-ait-50a00efc-5b5f999b6c-mvhtr"
Sep 10 20:53:50.790215 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.789989    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8c4a0a6-ba90-4a90-9b81-7555237a6b60-plugins-config-volume\") pod \"payload-processing-e2e-ait-50a00efc-5b5f999b6c-mvhtr\" (UID: \"d8c4a0a6-ba90-4a90-9b81-7555237a6b60\") " pod="openshift-ingress/payload-processing-e2e-ait-50a00efc-5b5f999b6c-mvhtr"
Sep 10 20:53:50.790757 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.790735    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8c4a0a6-ba90-4a90-9b81-7555237a6b60-plugins-config-volume\") pod \"payload-processing-e2e-ait-50a00efc-5b5f999b6c-mvhtr\" (UID: \"d8c4a0a6-ba90-4a90-9b81-7555237a6b60\") " pod="openshift-ingress/payload-processing-e2e-ait-50a00efc-5b5f999b6c-mvhtr"
Sep 10 20:53:50.800026 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.799971    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9x2q\" (UniqueName: \"kubernetes.io/projected/d8c4a0a6-ba90-4a90-9b81-7555237a6b60-kube-api-access-n9x2q\") pod \"payload-processing-e2e-ait-50a00efc-5b5f999b6c-mvhtr\" (UID: \"d8c4a0a6-ba90-4a90-9b81-7555237a6b60\") " pod="openshift-ingress/payload-processing-e2e-ait-50a00efc-5b5f999b6c-mvhtr"
Sep 10 20:53:50.902246 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:50.902214    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-50a00efc-5b5f999b6c-mvhtr"
Sep 10 20:53:51.177087 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:51.177058    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-50a00efc-5b5f999b6c-mvhtr"]
Sep 10 20:53:51.178011 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:53:51.177988    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c4a0a6_ba90_4a90_9b81_7555237a6b60.slice/crio-7caf7525ef09782371abb92496232566647fff1359c22930b4fdc3589d05a91e WatchSource:0}: Error finding container 7caf7525ef09782371abb92496232566647fff1359c22930b4fdc3589d05a91e: Status 404 returned error can't find the container with id 7caf7525ef09782371abb92496232566647fff1359c22930b4fdc3589d05a91e
Sep 10 20:53:51.432566 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:51.432465    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-50a00efc-5b5f999b6c-mvhtr" event={"ID":"d8c4a0a6-ba90-4a90-9b81-7555237a6b60","Type":"ContainerStarted","Data":"9a3299317503862ec935bed7ae10589d0f547a74a30c735e7643e42c967744d8"}
Sep 10 20:53:51.432566 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:51.432532    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-50a00efc-5b5f999b6c-mvhtr"
Sep 10 20:53:51.432566 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:51.432555    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-50a00efc-5b5f999b6c-mvhtr" event={"ID":"d8c4a0a6-ba90-4a90-9b81-7555237a6b60","Type":"ContainerStarted","Data":"7caf7525ef09782371abb92496232566647fff1359c22930b4fdc3589d05a91e"}
Sep 10 20:53:51.454993 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:51.454940    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-50a00efc-5b5f999b6c-mvhtr" podStartSLOduration=1.4549229750000001 podStartE2EDuration="1.454922975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:53:51.453247238 +0000 UTC m=+1071.418315433" watchObservedRunningTime="2026-09-10 20:53:51.454922975 +0000 UTC m=+1071.419991170"
Sep 10 20:53:55.294243 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:55.294130    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4"]
Sep 10 20:53:55.297951 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:55.297926    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4"
Sep 10 20:53:55.299932 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:55.299911    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-711c424e-kserve-self-signed-certs\""
Sep 10 20:53:55.307620 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:55.307593    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4"]
Sep 10 20:53:55.429164 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:55.429136    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8510cbc4-66ae-40db-bacb-a1ef98b158bb-tls-certs\") pod \"e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4\" (UID: \"8510cbc4-66ae-40db-bacb-a1ef98b158bb\") " pod="llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4"
Sep 10 20:53:55.429328 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:55.429199    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8510cbc4-66ae-40db-bacb-a1ef98b158bb-model-cache\") pod \"e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4\" (UID: \"8510cbc4-66ae-40db-bacb-a1ef98b158bb\") " pod="llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4"
Sep 10 20:53:55.429328 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:55.429242    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8510cbc4-66ae-40db-bacb-a1ef98b158bb-dshm\") pod \"e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4\" (UID: \"8510cbc4-66ae-40db-bacb-a1ef98b158bb\") " pod="llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4"
Sep 10 20:53:55.429328 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:55.429259    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzlj5\" (UniqueName: \"kubernetes.io/projected/8510cbc4-66ae-40db-bacb-a1ef98b158bb-kube-api-access-lzlj5\") pod \"e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4\" (UID: \"8510cbc4-66ae-40db-bacb-a1ef98b158bb\") " pod="llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4"
Sep 10 20:53:55.429328 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:55.429325    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8510cbc4-66ae-40db-bacb-a1ef98b158bb-home\") pod \"e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4\" (UID: \"8510cbc4-66ae-40db-bacb-a1ef98b158bb\") " pod="llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4"
Sep 10 20:53:55.429491 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:55.429358    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8510cbc4-66ae-40db-bacb-a1ef98b158bb-tmp-dir\") pod \"e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4\" (UID: \"8510cbc4-66ae-40db-bacb-a1ef98b158bb\") " pod="llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4"
Sep 10 20:53:55.530392 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:55.530365    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8510cbc4-66ae-40db-bacb-a1ef98b158bb-tls-certs\") pod \"e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4\" (UID: \"8510cbc4-66ae-40db-bacb-a1ef98b158bb\") " pod="llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4"
Sep 10 20:53:55.530506 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:55.530405    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8510cbc4-66ae-40db-bacb-a1ef98b158bb-model-cache\") pod \"e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4\" (UID: \"8510cbc4-66ae-40db-bacb-a1ef98b158bb\") " pod="llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4"
Sep 10 20:53:55.530506 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:55.530426    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8510cbc4-66ae-40db-bacb-a1ef98b158bb-dshm\") pod \"e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4\" (UID: \"8510cbc4-66ae-40db-bacb-a1ef98b158bb\") " pod="llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4"
Sep 10 20:53:55.530506 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:55.530443    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzlj5\" (UniqueName: \"kubernetes.io/projected/8510cbc4-66ae-40db-bacb-a1ef98b158bb-kube-api-access-lzlj5\") pod \"e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4\" (UID: \"8510cbc4-66ae-40db-bacb-a1ef98b158bb\") " pod="llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4"
Sep 10 20:53:55.530506 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:55.530482    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8510cbc4-66ae-40db-bacb-a1ef98b158bb-home\") pod \"e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4\" (UID: \"8510cbc4-66ae-40db-bacb-a1ef98b158bb\") " pod="llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4"
Sep 10 20:53:55.530666 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:55.530514    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8510cbc4-66ae-40db-bacb-a1ef98b158bb-tmp-dir\") pod \"e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4\" (UID: \"8510cbc4-66ae-40db-bacb-a1ef98b158bb\") " pod="llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4"
Sep 10 20:53:55.530822 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:55.530795    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8510cbc4-66ae-40db-bacb-a1ef98b158bb-model-cache\") pod \"e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4\" (UID: \"8510cbc4-66ae-40db-bacb-a1ef98b158bb\") " pod="llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4"
Sep 10 20:53:55.530891 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:55.530871    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8510cbc4-66ae-40db-bacb-a1ef98b158bb-home\") pod \"e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4\" (UID: \"8510cbc4-66ae-40db-bacb-a1ef98b158bb\") " pod="llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4"
Sep 10 20:53:55.530956 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:55.530939    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8510cbc4-66ae-40db-bacb-a1ef98b158bb-tmp-dir\") pod \"e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4\" (UID: \"8510cbc4-66ae-40db-bacb-a1ef98b158bb\") " pod="llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4"
Sep 10 20:53:55.532686 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:55.532663    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8510cbc4-66ae-40db-bacb-a1ef98b158bb-dshm\") pod \"e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4\" (UID: \"8510cbc4-66ae-40db-bacb-a1ef98b158bb\") " pod="llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4"
Sep 10 20:53:55.532940 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:55.532923    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8510cbc4-66ae-40db-bacb-a1ef98b158bb-tls-certs\") pod \"e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4\" (UID: \"8510cbc4-66ae-40db-bacb-a1ef98b158bb\") " pod="llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4"
Sep 10 20:53:55.538459 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:55.538439    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzlj5\" (UniqueName: \"kubernetes.io/projected/8510cbc4-66ae-40db-bacb-a1ef98b158bb-kube-api-access-lzlj5\") pod \"e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4\" (UID: \"8510cbc4-66ae-40db-bacb-a1ef98b158bb\") " pod="llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4"
Sep 10 20:53:55.607985 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:55.607955    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4"
Sep 10 20:53:55.880343 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:55.880307    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4"]
Sep 10 20:53:55.881989 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:53:55.881967    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8510cbc4_66ae_40db_bacb_a1ef98b158bb.slice/crio-aa237b2ff8d98634056f8861637eec5f65ca75d7dd8fd43bb88e499299dc977b WatchSource:0}: Error finding container aa237b2ff8d98634056f8861637eec5f65ca75d7dd8fd43bb88e499299dc977b: Status 404 returned error can't find the container with id aa237b2ff8d98634056f8861637eec5f65ca75d7dd8fd43bb88e499299dc977b
Sep 10 20:53:56.461685 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:56.461645    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4" event={"ID":"8510cbc4-66ae-40db-bacb-a1ef98b158bb","Type":"ContainerStarted","Data":"c4d8734ef98e302741a3a47b7bf1b07c130d7abeaa29b91e849510b4f98df003"}
Sep 10 20:53:56.462057 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:56.461697    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4" event={"ID":"8510cbc4-66ae-40db-bacb-a1ef98b158bb","Type":"ContainerStarted","Data":"aa237b2ff8d98634056f8861637eec5f65ca75d7dd8fd43bb88e499299dc977b"}
Sep 10 20:53:56.481220 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:53:56.481147    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4" podStartSLOduration=1.481128791 podStartE2EDuration="1.481128791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:53:56.479453596 +0000 UTC m=+1076.444521821" watchObservedRunningTime="2026-09-10 20:53:56.481128791 +0000 UTC m=+1076.446196987"
Sep 10 20:54:02.441099 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:02.441059    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-50a00efc-5b5f999b6c-mvhtr"
Sep 10 20:54:05.608923 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:05.608883    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4"
Sep 10 20:54:05.608923 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:05.608928    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4"
Sep 10 20:54:05.621272 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:05.621249    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4"
Sep 10 20:54:06.529760 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:06.529732    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4"
Sep 10 20:54:20.709341 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:20.709303    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-50a00efc-5b5f999b6c-mvhtr"]
Sep 10 20:54:20.709889 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:20.709570    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-50a00efc-5b5f999b6c-mvhtr" podUID="d8c4a0a6-ba90-4a90-9b81-7555237a6b60" containerName="payload-processing" containerID="cri-o://9a3299317503862ec935bed7ae10589d0f547a74a30c735e7643e42c967744d8" gracePeriod=30
Sep 10 20:54:20.976323 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:20.976285    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-50a00efc-5b5f999b6c-mvhtr"
Sep 10 20:54:21.016826 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:21.016795    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9x2q\" (UniqueName: \"kubernetes.io/projected/d8c4a0a6-ba90-4a90-9b81-7555237a6b60-kube-api-access-n9x2q\") pod \"d8c4a0a6-ba90-4a90-9b81-7555237a6b60\" (UID: \"d8c4a0a6-ba90-4a90-9b81-7555237a6b60\") "
Sep 10 20:54:21.016981 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:21.016857    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8c4a0a6-ba90-4a90-9b81-7555237a6b60-plugins-config-volume\") pod \"d8c4a0a6-ba90-4a90-9b81-7555237a6b60\" (UID: \"d8c4a0a6-ba90-4a90-9b81-7555237a6b60\") "
Sep 10 20:54:21.017342 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:21.017312    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8c4a0a6-ba90-4a90-9b81-7555237a6b60-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d8c4a0a6-ba90-4a90-9b81-7555237a6b60" (UID: "d8c4a0a6-ba90-4a90-9b81-7555237a6b60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:54:21.018820 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:21.018792    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c4a0a6-ba90-4a90-9b81-7555237a6b60-kube-api-access-n9x2q" (OuterVolumeSpecName: "kube-api-access-n9x2q") pod "d8c4a0a6-ba90-4a90-9b81-7555237a6b60" (UID: "d8c4a0a6-ba90-4a90-9b81-7555237a6b60"). InnerVolumeSpecName "kube-api-access-n9x2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:54:21.117868 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:21.117831    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9x2q\" (UniqueName: \"kubernetes.io/projected/d8c4a0a6-ba90-4a90-9b81-7555237a6b60-kube-api-access-n9x2q\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:54:21.117868 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:21.117863    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8c4a0a6-ba90-4a90-9b81-7555237a6b60-plugins-config-volume\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:54:21.598644 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:21.598611    2564 generic.go:356] "Generic (PLEG): container finished" podID="d8c4a0a6-ba90-4a90-9b81-7555237a6b60" containerID="9a3299317503862ec935bed7ae10589d0f547a74a30c735e7643e42c967744d8" exitCode=0
Sep 10 20:54:21.598850 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:21.598674    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-50a00efc-5b5f999b6c-mvhtr"
Sep 10 20:54:21.598850 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:21.598678    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-50a00efc-5b5f999b6c-mvhtr" event={"ID":"d8c4a0a6-ba90-4a90-9b81-7555237a6b60","Type":"ContainerDied","Data":"9a3299317503862ec935bed7ae10589d0f547a74a30c735e7643e42c967744d8"}
Sep 10 20:54:21.598850 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:21.598794    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-50a00efc-5b5f999b6c-mvhtr" event={"ID":"d8c4a0a6-ba90-4a90-9b81-7555237a6b60","Type":"ContainerDied","Data":"7caf7525ef09782371abb92496232566647fff1359c22930b4fdc3589d05a91e"}
Sep 10 20:54:21.598850 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:21.598815    2564 scope.go:117] "RemoveContainer" containerID="9a3299317503862ec935bed7ae10589d0f547a74a30c735e7643e42c967744d8"
Sep 10 20:54:21.609840 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:21.609822    2564 scope.go:117] "RemoveContainer" containerID="9a3299317503862ec935bed7ae10589d0f547a74a30c735e7643e42c967744d8"
Sep 10 20:54:21.610132 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:54:21.610115    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a3299317503862ec935bed7ae10589d0f547a74a30c735e7643e42c967744d8\": container with ID starting with 9a3299317503862ec935bed7ae10589d0f547a74a30c735e7643e42c967744d8 not found: ID does not exist" containerID="9a3299317503862ec935bed7ae10589d0f547a74a30c735e7643e42c967744d8"
Sep 10 20:54:21.610229 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:21.610139    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a3299317503862ec935bed7ae10589d0f547a74a30c735e7643e42c967744d8"} err="failed to get container status \"9a3299317503862ec935bed7ae10589d0f547a74a30c735e7643e42c967744d8\": rpc error: code = NotFound desc = could not find container \"9a3299317503862ec935bed7ae10589d0f547a74a30c735e7643e42c967744d8\": container with ID starting with 9a3299317503862ec935bed7ae10589d0f547a74a30c735e7643e42c967744d8 not found: ID does not exist"
Sep 10 20:54:21.635961 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:21.635933    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-50a00efc-5b5f999b6c-mvhtr"]
Sep 10 20:54:21.651242 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:21.651219    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-50a00efc-5b5f999b6c-mvhtr"]
Sep 10 20:54:22.541064 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:22.541030    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8c4a0a6-ba90-4a90-9b81-7555237a6b60" path="/var/lib/kubelet/pods/d8c4a0a6-ba90-4a90-9b81-7555237a6b60/volumes"
Sep 10 20:54:59.783210 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:59.783158    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t"]
Sep 10 20:54:59.783664 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:59.783520    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8c4a0a6-ba90-4a90-9b81-7555237a6b60" containerName="payload-processing"
Sep 10 20:54:59.783664 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:59.783531    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8c4a0a6-ba90-4a90-9b81-7555237a6b60" containerName="payload-processing"
Sep 10 20:54:59.783664 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:59.783589    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8c4a0a6-ba90-4a90-9b81-7555237a6b60" containerName="payload-processing"
Sep 10 20:54:59.786552 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:59.786534    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t"
Sep 10 20:54:59.789677 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:59.789652    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-80aa25\"/\"test-model-80aa25-kserve-self-signed-certs\""
Sep 10 20:54:59.789677 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:59.789664    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-80aa25\"/\"openshift-service-ca.crt\""
Sep 10 20:54:59.789862 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:59.789708    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-80aa25\"/\"default-dockercfg-6jlrg\""
Sep 10 20:54:59.789862 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:59.789765    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-80aa25\"/\"kube-root-ca.crt\""
Sep 10 20:54:59.798437 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:59.798407    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t"]
Sep 10 20:54:59.935255 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:59.935221    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b3755d8-15be-417c-8661-1381d26afd0d-dshm\") pod \"test-model-80aa25-kserve-7c4c7df979-fkv2t\" (UID: \"7b3755d8-15be-417c-8661-1381d26afd0d\") " pod="ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t"
Sep 10 20:54:59.935427 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:59.935297    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b3755d8-15be-417c-8661-1381d26afd0d-home\") pod \"test-model-80aa25-kserve-7c4c7df979-fkv2t\" (UID: \"7b3755d8-15be-417c-8661-1381d26afd0d\") " pod="ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t"
Sep 10 20:54:59.935427 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:59.935325    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3755d8-15be-417c-8661-1381d26afd0d-tls-certs\") pod \"test-model-80aa25-kserve-7c4c7df979-fkv2t\" (UID: \"7b3755d8-15be-417c-8661-1381d26afd0d\") " pod="ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t"
Sep 10 20:54:59.935427 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:59.935341    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z242p\" (UniqueName: \"kubernetes.io/projected/7b3755d8-15be-417c-8661-1381d26afd0d-kube-api-access-z242p\") pod \"test-model-80aa25-kserve-7c4c7df979-fkv2t\" (UID: \"7b3755d8-15be-417c-8661-1381d26afd0d\") " pod="ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t"
Sep 10 20:54:59.935427 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:59.935365    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b3755d8-15be-417c-8661-1381d26afd0d-model-cache\") pod \"test-model-80aa25-kserve-7c4c7df979-fkv2t\" (UID: \"7b3755d8-15be-417c-8661-1381d26afd0d\") " pod="ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t"
Sep 10 20:54:59.935427 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:54:59.935412    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b3755d8-15be-417c-8661-1381d26afd0d-tmp-dir\") pod \"test-model-80aa25-kserve-7c4c7df979-fkv2t\" (UID: \"7b3755d8-15be-417c-8661-1381d26afd0d\") " pod="ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t"
Sep 10 20:55:00.036271 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:00.036149    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b3755d8-15be-417c-8661-1381d26afd0d-home\") pod \"test-model-80aa25-kserve-7c4c7df979-fkv2t\" (UID: \"7b3755d8-15be-417c-8661-1381d26afd0d\") " pod="ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t"
Sep 10 20:55:00.036271 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:00.036220    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3755d8-15be-417c-8661-1381d26afd0d-tls-certs\") pod \"test-model-80aa25-kserve-7c4c7df979-fkv2t\" (UID: \"7b3755d8-15be-417c-8661-1381d26afd0d\") " pod="ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t"
Sep 10 20:55:00.036271 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:00.036246    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z242p\" (UniqueName: \"kubernetes.io/projected/7b3755d8-15be-417c-8661-1381d26afd0d-kube-api-access-z242p\") pod \"test-model-80aa25-kserve-7c4c7df979-fkv2t\" (UID: \"7b3755d8-15be-417c-8661-1381d26afd0d\") " pod="ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t"
Sep 10 20:55:00.036573 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:00.036277    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b3755d8-15be-417c-8661-1381d26afd0d-model-cache\") pod \"test-model-80aa25-kserve-7c4c7df979-fkv2t\" (UID: \"7b3755d8-15be-417c-8661-1381d26afd0d\") " pod="ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t"
Sep 10 20:55:00.036573 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:00.036316    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b3755d8-15be-417c-8661-1381d26afd0d-tmp-dir\") pod \"test-model-80aa25-kserve-7c4c7df979-fkv2t\" (UID: \"7b3755d8-15be-417c-8661-1381d26afd0d\") " pod="ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t"
Sep 10 20:55:00.036573 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:00.036412    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b3755d8-15be-417c-8661-1381d26afd0d-dshm\") pod \"test-model-80aa25-kserve-7c4c7df979-fkv2t\" (UID: \"7b3755d8-15be-417c-8661-1381d26afd0d\") " pod="ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t"
Sep 10 20:55:00.036741 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:00.036616    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b3755d8-15be-417c-8661-1381d26afd0d-home\") pod \"test-model-80aa25-kserve-7c4c7df979-fkv2t\" (UID: \"7b3755d8-15be-417c-8661-1381d26afd0d\") " pod="ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t"
Sep 10 20:55:00.036741 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:00.036673    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b3755d8-15be-417c-8661-1381d26afd0d-model-cache\") pod \"test-model-80aa25-kserve-7c4c7df979-fkv2t\" (UID: \"7b3755d8-15be-417c-8661-1381d26afd0d\") " pod="ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t"
Sep 10 20:55:00.036848 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:00.036785    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b3755d8-15be-417c-8661-1381d26afd0d-tmp-dir\") pod \"test-model-80aa25-kserve-7c4c7df979-fkv2t\" (UID: \"7b3755d8-15be-417c-8661-1381d26afd0d\") " pod="ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t"
Sep 10 20:55:00.038683 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:00.038657    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b3755d8-15be-417c-8661-1381d26afd0d-dshm\") pod \"test-model-80aa25-kserve-7c4c7df979-fkv2t\" (UID: \"7b3755d8-15be-417c-8661-1381d26afd0d\") " pod="ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t"
Sep 10 20:55:00.038790 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:00.038774    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3755d8-15be-417c-8661-1381d26afd0d-tls-certs\") pod \"test-model-80aa25-kserve-7c4c7df979-fkv2t\" (UID: \"7b3755d8-15be-417c-8661-1381d26afd0d\") " pod="ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t"
Sep 10 20:55:00.048138 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:00.048117    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z242p\" (UniqueName: \"kubernetes.io/projected/7b3755d8-15be-417c-8661-1381d26afd0d-kube-api-access-z242p\") pod \"test-model-80aa25-kserve-7c4c7df979-fkv2t\" (UID: \"7b3755d8-15be-417c-8661-1381d26afd0d\") " pod="ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t"
Sep 10 20:55:00.096491 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:00.096454    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t"
Sep 10 20:55:00.371093 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:00.371060    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t"]
Sep 10 20:55:00.372818 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:55:00.372790    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b3755d8_15be_417c_8661_1381d26afd0d.slice/crio-b5219688f54c4aa4990e045a4e9e7ec321ca77e264ca09ec1d50873e80bef491 WatchSource:0}: Error finding container b5219688f54c4aa4990e045a4e9e7ec321ca77e264ca09ec1d50873e80bef491: Status 404 returned error can't find the container with id b5219688f54c4aa4990e045a4e9e7ec321ca77e264ca09ec1d50873e80bef491
Sep 10 20:55:00.813445 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:00.813362    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t" event={"ID":"7b3755d8-15be-417c-8661-1381d26afd0d","Type":"ContainerStarted","Data":"f3ad9a26dad2c203cdcf635db60cb7dfd8e221f08249bac68208fe35253ed343"}
Sep 10 20:55:00.813445 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:00.813401    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t" event={"ID":"7b3755d8-15be-417c-8661-1381d26afd0d","Type":"ContainerStarted","Data":"b5219688f54c4aa4990e045a4e9e7ec321ca77e264ca09ec1d50873e80bef491"}
Sep 10 20:55:00.842981 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:00.842932    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t" podStartSLOduration=1.842915847 podStartE2EDuration="1.842915847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:55:00.841203169 +0000 UTC m=+1140.806271418" watchObservedRunningTime="2026-09-10 20:55:00.842915847 +0000 UTC m=+1140.807984041"
Sep 10 20:55:03.682608 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:03.682570    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4"]
Sep 10 20:55:04.774219 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:04.774185    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4"]
Sep 10 20:55:10.097329 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:10.097286    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t"
Sep 10 20:55:10.097748 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:10.097369    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t"
Sep 10 20:55:10.109845 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:10.109820    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t"
Sep 10 20:55:10.877112 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:10.877077    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t"
Sep 10 20:55:18.701527 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:18.701483    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4" podUID="660816da-324e-45e0-8fbb-45737c13a97a" containerName="main" containerID="cri-o://1327511be4deae9fef8ec022fe53a799a4fd2999a69bbcd7f68bf9c55a39956c" gracePeriod=45
Sep 10 20:55:18.907891 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:18.907861    2564 generic.go:356] "Generic (PLEG): container finished" podID="660816da-324e-45e0-8fbb-45737c13a97a" containerID="1327511be4deae9fef8ec022fe53a799a4fd2999a69bbcd7f68bf9c55a39956c" exitCode=0
Sep 10 20:55:18.908035 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:18.907940    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4" event={"ID":"660816da-324e-45e0-8fbb-45737c13a97a","Type":"ContainerDied","Data":"1327511be4deae9fef8ec022fe53a799a4fd2999a69bbcd7f68bf9c55a39956c"}
Sep 10 20:55:18.940660 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:18.940640    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4"
Sep 10 20:55:19.013577 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:19.013521    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/660816da-324e-45e0-8fbb-45737c13a97a-model-cache\") pod \"660816da-324e-45e0-8fbb-45737c13a97a\" (UID: \"660816da-324e-45e0-8fbb-45737c13a97a\") "
Sep 10 20:55:19.013577 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:19.013557    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/660816da-324e-45e0-8fbb-45737c13a97a-dshm\") pod \"660816da-324e-45e0-8fbb-45737c13a97a\" (UID: \"660816da-324e-45e0-8fbb-45737c13a97a\") "
Sep 10 20:55:19.013722 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:19.013600    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/660816da-324e-45e0-8fbb-45737c13a97a-tls-certs\") pod \"660816da-324e-45e0-8fbb-45737c13a97a\" (UID: \"660816da-324e-45e0-8fbb-45737c13a97a\") "
Sep 10 20:55:19.013722 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:19.013628    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/660816da-324e-45e0-8fbb-45737c13a97a-home\") pod \"660816da-324e-45e0-8fbb-45737c13a97a\" (UID: \"660816da-324e-45e0-8fbb-45737c13a97a\") "
Sep 10 20:55:19.013722 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:19.013652    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/660816da-324e-45e0-8fbb-45737c13a97a-tmp-dir\") pod \"660816da-324e-45e0-8fbb-45737c13a97a\" (UID: \"660816da-324e-45e0-8fbb-45737c13a97a\") "
Sep 10 20:55:19.013722 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:19.013680    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pkj5\" (UniqueName: \"kubernetes.io/projected/660816da-324e-45e0-8fbb-45737c13a97a-kube-api-access-5pkj5\") pod \"660816da-324e-45e0-8fbb-45737c13a97a\" (UID: \"660816da-324e-45e0-8fbb-45737c13a97a\") "
Sep 10 20:55:19.013924 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:19.013754    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660816da-324e-45e0-8fbb-45737c13a97a-model-cache" (OuterVolumeSpecName: "model-cache") pod "660816da-324e-45e0-8fbb-45737c13a97a" (UID: "660816da-324e-45e0-8fbb-45737c13a97a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:55:19.013982 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:19.013932    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660816da-324e-45e0-8fbb-45737c13a97a-home" (OuterVolumeSpecName: "home") pod "660816da-324e-45e0-8fbb-45737c13a97a" (UID: "660816da-324e-45e0-8fbb-45737c13a97a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:55:19.013982 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:19.013958    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/660816da-324e-45e0-8fbb-45737c13a97a-model-cache\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:55:19.014057 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:19.013970    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660816da-324e-45e0-8fbb-45737c13a97a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "660816da-324e-45e0-8fbb-45737c13a97a" (UID: "660816da-324e-45e0-8fbb-45737c13a97a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:55:19.015858 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:19.015834    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660816da-324e-45e0-8fbb-45737c13a97a-dshm" (OuterVolumeSpecName: "dshm") pod "660816da-324e-45e0-8fbb-45737c13a97a" (UID: "660816da-324e-45e0-8fbb-45737c13a97a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:55:19.015951 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:19.015831    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/660816da-324e-45e0-8fbb-45737c13a97a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "660816da-324e-45e0-8fbb-45737c13a97a" (UID: "660816da-324e-45e0-8fbb-45737c13a97a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:55:19.015951 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:19.015881    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660816da-324e-45e0-8fbb-45737c13a97a-kube-api-access-5pkj5" (OuterVolumeSpecName: "kube-api-access-5pkj5") pod "660816da-324e-45e0-8fbb-45737c13a97a" (UID: "660816da-324e-45e0-8fbb-45737c13a97a"). InnerVolumeSpecName "kube-api-access-5pkj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:55:19.115130 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:19.115109    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/660816da-324e-45e0-8fbb-45737c13a97a-tls-certs\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:55:19.115130 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:19.115127    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/660816da-324e-45e0-8fbb-45737c13a97a-home\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:55:19.115281 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:19.115135    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/660816da-324e-45e0-8fbb-45737c13a97a-tmp-dir\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:55:19.115281 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:19.115144    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pkj5\" (UniqueName: \"kubernetes.io/projected/660816da-324e-45e0-8fbb-45737c13a97a-kube-api-access-5pkj5\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:55:19.115281 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:19.115152    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/660816da-324e-45e0-8fbb-45737c13a97a-dshm\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:55:19.793410 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:19.793367    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4" podUID="8510cbc4-66ae-40db-bacb-a1ef98b158bb" containerName="main" containerID="cri-o://c4d8734ef98e302741a3a47b7bf1b07c130d7abeaa29b91e849510b4f98df003" gracePeriod=45
Sep 10 20:55:19.915077 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:19.915050    2564 generic.go:356] "Generic (PLEG): container finished" podID="8510cbc4-66ae-40db-bacb-a1ef98b158bb" containerID="c4d8734ef98e302741a3a47b7bf1b07c130d7abeaa29b91e849510b4f98df003" exitCode=0
Sep 10 20:55:19.915217 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:19.915080    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4" event={"ID":"8510cbc4-66ae-40db-bacb-a1ef98b158bb","Type":"ContainerDied","Data":"c4d8734ef98e302741a3a47b7bf1b07c130d7abeaa29b91e849510b4f98df003"}
Sep 10 20:55:19.916571 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:19.916547    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4" event={"ID":"660816da-324e-45e0-8fbb-45737c13a97a","Type":"ContainerDied","Data":"bbef2c2f6ffaced879dd423c4b911d053ad7c135e845cefa2b5eb05eb9bfc86b"}
Sep 10 20:55:19.916685 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:19.916580    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4"
Sep 10 20:55:19.916685 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:19.916590    2564 scope.go:117] "RemoveContainer" containerID="1327511be4deae9fef8ec022fe53a799a4fd2999a69bbcd7f68bf9c55a39956c"
Sep 10 20:55:19.943523 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:19.943495    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4"]
Sep 10 20:55:19.945713 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:19.945690    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-711c424e-kserve-554d88c69d-f47m4"]
Sep 10 20:55:20.045600 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:20.045549    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4"
Sep 10 20:55:20.121742 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:20.121716    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8510cbc4-66ae-40db-bacb-a1ef98b158bb-tmp-dir\") pod \"8510cbc4-66ae-40db-bacb-a1ef98b158bb\" (UID: \"8510cbc4-66ae-40db-bacb-a1ef98b158bb\") "
Sep 10 20:55:20.121844 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:20.121756    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8510cbc4-66ae-40db-bacb-a1ef98b158bb-dshm\") pod \"8510cbc4-66ae-40db-bacb-a1ef98b158bb\" (UID: \"8510cbc4-66ae-40db-bacb-a1ef98b158bb\") "
Sep 10 20:55:20.121844 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:20.121777    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8510cbc4-66ae-40db-bacb-a1ef98b158bb-tls-certs\") pod \"8510cbc4-66ae-40db-bacb-a1ef98b158bb\" (UID: \"8510cbc4-66ae-40db-bacb-a1ef98b158bb\") "
Sep 10 20:55:20.121844 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:20.121815    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8510cbc4-66ae-40db-bacb-a1ef98b158bb-home\") pod \"8510cbc4-66ae-40db-bacb-a1ef98b158bb\" (UID: \"8510cbc4-66ae-40db-bacb-a1ef98b158bb\") "
Sep 10 20:55:20.122002 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:20.121870    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8510cbc4-66ae-40db-bacb-a1ef98b158bb-model-cache\") pod \"8510cbc4-66ae-40db-bacb-a1ef98b158bb\" (UID: \"8510cbc4-66ae-40db-bacb-a1ef98b158bb\") "
Sep 10 20:55:20.122002 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:20.121890    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzlj5\" (UniqueName: \"kubernetes.io/projected/8510cbc4-66ae-40db-bacb-a1ef98b158bb-kube-api-access-lzlj5\") pod \"8510cbc4-66ae-40db-bacb-a1ef98b158bb\" (UID: \"8510cbc4-66ae-40db-bacb-a1ef98b158bb\") "
Sep 10 20:55:20.122120 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:20.121995    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8510cbc4-66ae-40db-bacb-a1ef98b158bb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8510cbc4-66ae-40db-bacb-a1ef98b158bb" (UID: "8510cbc4-66ae-40db-bacb-a1ef98b158bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:55:20.122202 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:20.122120    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8510cbc4-66ae-40db-bacb-a1ef98b158bb-model-cache" (OuterVolumeSpecName: "model-cache") pod "8510cbc4-66ae-40db-bacb-a1ef98b158bb" (UID: "8510cbc4-66ae-40db-bacb-a1ef98b158bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:55:20.122202 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:20.122137    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8510cbc4-66ae-40db-bacb-a1ef98b158bb-home" (OuterVolumeSpecName: "home") pod "8510cbc4-66ae-40db-bacb-a1ef98b158bb" (UID: "8510cbc4-66ae-40db-bacb-a1ef98b158bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:55:20.122299 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:20.122238    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8510cbc4-66ae-40db-bacb-a1ef98b158bb-tmp-dir\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:55:20.122299 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:20.122257    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8510cbc4-66ae-40db-bacb-a1ef98b158bb-home\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:55:20.122299 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:20.122270    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8510cbc4-66ae-40db-bacb-a1ef98b158bb-model-cache\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:55:20.123751 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:20.123729    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8510cbc4-66ae-40db-bacb-a1ef98b158bb-dshm" (OuterVolumeSpecName: "dshm") pod "8510cbc4-66ae-40db-bacb-a1ef98b158bb" (UID: "8510cbc4-66ae-40db-bacb-a1ef98b158bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:55:20.123942 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:20.123923    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8510cbc4-66ae-40db-bacb-a1ef98b158bb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8510cbc4-66ae-40db-bacb-a1ef98b158bb" (UID: "8510cbc4-66ae-40db-bacb-a1ef98b158bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:55:20.124065 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:20.124041    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8510cbc4-66ae-40db-bacb-a1ef98b158bb-kube-api-access-lzlj5" (OuterVolumeSpecName: "kube-api-access-lzlj5") pod "8510cbc4-66ae-40db-bacb-a1ef98b158bb" (UID: "8510cbc4-66ae-40db-bacb-a1ef98b158bb"). InnerVolumeSpecName "kube-api-access-lzlj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:55:20.222877 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:20.222857    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8510cbc4-66ae-40db-bacb-a1ef98b158bb-tls-certs\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:55:20.222877 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:20.222878    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzlj5\" (UniqueName: \"kubernetes.io/projected/8510cbc4-66ae-40db-bacb-a1ef98b158bb-kube-api-access-lzlj5\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:55:20.223004 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:20.222887    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8510cbc4-66ae-40db-bacb-a1ef98b158bb-dshm\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:55:20.541076 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:20.541042    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="660816da-324e-45e0-8fbb-45737c13a97a" path="/var/lib/kubelet/pods/660816da-324e-45e0-8fbb-45737c13a97a/volumes"
Sep 10 20:55:20.923152 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:20.923124    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4"
Sep 10 20:55:20.923637 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:20.923150    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4" event={"ID":"8510cbc4-66ae-40db-bacb-a1ef98b158bb","Type":"ContainerDied","Data":"aa237b2ff8d98634056f8861637eec5f65ca75d7dd8fd43bb88e499299dc977b"}
Sep 10 20:55:20.923637 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:20.923219    2564 scope.go:117] "RemoveContainer" containerID="c4d8734ef98e302741a3a47b7bf1b07c130d7abeaa29b91e849510b4f98df003"
Sep 10 20:55:20.939982 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:20.939958    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4"]
Sep 10 20:55:20.946540 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:20.946517    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-711c424e-kserve-5f7685cd64-xkvk4"]
Sep 10 20:55:22.540881 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:22.540850    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8510cbc4-66ae-40db-bacb-a1ef98b158bb" path="/var/lib/kubelet/pods/8510cbc4-66ae-40db-bacb-a1ef98b158bb/volumes"
Sep 10 20:55:36.895364 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:36.895287    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk"]
Sep 10 20:55:36.895771 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:36.895639    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="660816da-324e-45e0-8fbb-45737c13a97a" containerName="main"
Sep 10 20:55:36.895771 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:36.895650    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="660816da-324e-45e0-8fbb-45737c13a97a" containerName="main"
Sep 10 20:55:36.895771 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:36.895660    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8510cbc4-66ae-40db-bacb-a1ef98b158bb" containerName="main"
Sep 10 20:55:36.895771 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:36.895666    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8510cbc4-66ae-40db-bacb-a1ef98b158bb" containerName="main"
Sep 10 20:55:36.895771 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:36.895724    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8510cbc4-66ae-40db-bacb-a1ef98b158bb" containerName="main"
Sep 10 20:55:36.895771 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:36.895731    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="660816da-324e-45e0-8fbb-45737c13a97a" containerName="main"
Sep 10 20:55:36.898362 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:36.898341    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk"
Sep 10 20:55:36.900985 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:36.900959    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-467fc3\"/\"test-model-467fc3-kserve-self-signed-certs\""
Sep 10 20:55:36.900985 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:36.900981    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-467fc3\"/\"openshift-service-ca.crt\""
Sep 10 20:55:36.901161 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:36.900965    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-467fc3\"/\"kube-root-ca.crt\""
Sep 10 20:55:36.901161 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:36.901071    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-467fc3\"/\"default-dockercfg-f9c9s\""
Sep 10 20:55:36.909800 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:36.909780    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk"]
Sep 10 20:55:36.958640 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:36.958614    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/020714b8-832a-441b-84e0-ce38db14f968-tmp-dir\") pod \"test-model-467fc3-kserve-6b67c59b75-4tqrk\" (UID: \"020714b8-832a-441b-84e0-ce38db14f968\") " pod="ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk"
Sep 10 20:55:36.958738 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:36.958650    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/020714b8-832a-441b-84e0-ce38db14f968-home\") pod \"test-model-467fc3-kserve-6b67c59b75-4tqrk\" (UID: \"020714b8-832a-441b-84e0-ce38db14f968\") " pod="ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk"
Sep 10 20:55:36.958738 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:36.958682    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/020714b8-832a-441b-84e0-ce38db14f968-tls-certs\") pod \"test-model-467fc3-kserve-6b67c59b75-4tqrk\" (UID: \"020714b8-832a-441b-84e0-ce38db14f968\") " pod="ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk"
Sep 10 20:55:36.958842 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:36.958747    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/020714b8-832a-441b-84e0-ce38db14f968-model-cache\") pod \"test-model-467fc3-kserve-6b67c59b75-4tqrk\" (UID: \"020714b8-832a-441b-84e0-ce38db14f968\") " pod="ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk"
Sep 10 20:55:36.958842 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:36.958780    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/020714b8-832a-441b-84e0-ce38db14f968-dshm\") pod \"test-model-467fc3-kserve-6b67c59b75-4tqrk\" (UID: \"020714b8-832a-441b-84e0-ce38db14f968\") " pod="ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk"
Sep 10 20:55:36.958842 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:36.958811    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7cls\" (UniqueName: \"kubernetes.io/projected/020714b8-832a-441b-84e0-ce38db14f968-kube-api-access-x7cls\") pod \"test-model-467fc3-kserve-6b67c59b75-4tqrk\" (UID: \"020714b8-832a-441b-84e0-ce38db14f968\") " pod="ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk"
Sep 10 20:55:37.059454 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:37.059424    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/020714b8-832a-441b-84e0-ce38db14f968-tls-certs\") pod \"test-model-467fc3-kserve-6b67c59b75-4tqrk\" (UID: \"020714b8-832a-441b-84e0-ce38db14f968\") " pod="ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk"
Sep 10 20:55:37.059454 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:37.059459    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/020714b8-832a-441b-84e0-ce38db14f968-model-cache\") pod \"test-model-467fc3-kserve-6b67c59b75-4tqrk\" (UID: \"020714b8-832a-441b-84e0-ce38db14f968\") " pod="ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk"
Sep 10 20:55:37.059607 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:37.059478    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/020714b8-832a-441b-84e0-ce38db14f968-dshm\") pod \"test-model-467fc3-kserve-6b67c59b75-4tqrk\" (UID: \"020714b8-832a-441b-84e0-ce38db14f968\") " pod="ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk"
Sep 10 20:55:37.059657 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:37.059604    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7cls\" (UniqueName: \"kubernetes.io/projected/020714b8-832a-441b-84e0-ce38db14f968-kube-api-access-x7cls\") pod \"test-model-467fc3-kserve-6b67c59b75-4tqrk\" (UID: \"020714b8-832a-441b-84e0-ce38db14f968\") " pod="ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk"
Sep 10 20:55:37.059701 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:37.059673    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/020714b8-832a-441b-84e0-ce38db14f968-tmp-dir\") pod \"test-model-467fc3-kserve-6b67c59b75-4tqrk\" (UID: \"020714b8-832a-441b-84e0-ce38db14f968\") " pod="ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk"
Sep 10 20:55:37.059801 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:37.059724    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/020714b8-832a-441b-84e0-ce38db14f968-home\") pod \"test-model-467fc3-kserve-6b67c59b75-4tqrk\" (UID: \"020714b8-832a-441b-84e0-ce38db14f968\") " pod="ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk"
Sep 10 20:55:37.059876 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:37.059856    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/020714b8-832a-441b-84e0-ce38db14f968-model-cache\") pod \"test-model-467fc3-kserve-6b67c59b75-4tqrk\" (UID: \"020714b8-832a-441b-84e0-ce38db14f968\") " pod="ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk"
Sep 10 20:55:37.060011 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:37.059993    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/020714b8-832a-441b-84e0-ce38db14f968-tmp-dir\") pod \"test-model-467fc3-kserve-6b67c59b75-4tqrk\" (UID: \"020714b8-832a-441b-84e0-ce38db14f968\") " pod="ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk"
Sep 10 20:55:37.060074 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:37.060003    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/020714b8-832a-441b-84e0-ce38db14f968-home\") pod \"test-model-467fc3-kserve-6b67c59b75-4tqrk\" (UID: \"020714b8-832a-441b-84e0-ce38db14f968\") " pod="ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk"
Sep 10 20:55:37.061693 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:37.061673    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/020714b8-832a-441b-84e0-ce38db14f968-dshm\") pod \"test-model-467fc3-kserve-6b67c59b75-4tqrk\" (UID: \"020714b8-832a-441b-84e0-ce38db14f968\") " pod="ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk"
Sep 10 20:55:37.062027 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:37.062010    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/020714b8-832a-441b-84e0-ce38db14f968-tls-certs\") pod \"test-model-467fc3-kserve-6b67c59b75-4tqrk\" (UID: \"020714b8-832a-441b-84e0-ce38db14f968\") " pod="ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk"
Sep 10 20:55:37.066917 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:37.066893    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7cls\" (UniqueName: \"kubernetes.io/projected/020714b8-832a-441b-84e0-ce38db14f968-kube-api-access-x7cls\") pod \"test-model-467fc3-kserve-6b67c59b75-4tqrk\" (UID: \"020714b8-832a-441b-84e0-ce38db14f968\") " pod="ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk"
Sep 10 20:55:37.207878 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:37.207786    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk"
Sep 10 20:55:37.481984 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:37.481955    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk"]
Sep 10 20:55:37.482656 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:55:37.482632    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod020714b8_832a_441b_84e0_ce38db14f968.slice/crio-26b90558caedc250a2ebc0f00bcbfaca12aab129869d9f14e3072a82fd3b09fa WatchSource:0}: Error finding container 26b90558caedc250a2ebc0f00bcbfaca12aab129869d9f14e3072a82fd3b09fa: Status 404 returned error can't find the container with id 26b90558caedc250a2ebc0f00bcbfaca12aab129869d9f14e3072a82fd3b09fa
Sep 10 20:55:38.022884 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:38.022844    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk" event={"ID":"020714b8-832a-441b-84e0-ce38db14f968","Type":"ContainerStarted","Data":"0139b9daf5a5972f1995ee58bc5094e8241fb2ee8736915d9c278239e0a17e1b"}
Sep 10 20:55:38.023299 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:38.022890    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk" event={"ID":"020714b8-832a-441b-84e0-ce38db14f968","Type":"ContainerStarted","Data":"26b90558caedc250a2ebc0f00bcbfaca12aab129869d9f14e3072a82fd3b09fa"}
Sep 10 20:55:38.040981 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:38.040927    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk" podStartSLOduration=2.040912158 podStartE2EDuration="2.040912158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:55:38.039375326 +0000 UTC m=+1178.004443522" watchObservedRunningTime="2026-09-10 20:55:38.040912158 +0000 UTC m=+1178.005980352"
Sep 10 20:55:42.279401 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:42.279368    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"]
Sep 10 20:55:42.279819 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:42.279549    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl" podUID="3c5a180c-786f-4556-8a89-ca0714d01bba" containerName="istio-proxy" containerID="cri-o://40f6fd61590b53ef20f75570c0232b1ce6a73808f3c25cf7baf151e142daf78e" gracePeriod=30
Sep 10 20:55:47.208049 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:47.208018    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk"
Sep 10 20:55:47.208559 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:47.208063    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk"
Sep 10 20:55:47.220600 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:47.220574    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk"
Sep 10 20:55:47.523647 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:47.523624    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:55:47.647624 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:47.647592    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c5a180c-786f-4556-8a89-ca0714d01bba-istio-envoy\") pod \"3c5a180c-786f-4556-8a89-ca0714d01bba\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") "
Sep 10 20:55:47.647624 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:47.647627    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncz5v\" (UniqueName: \"kubernetes.io/projected/3c5a180c-786f-4556-8a89-ca0714d01bba-kube-api-access-ncz5v\") pod \"3c5a180c-786f-4556-8a89-ca0714d01bba\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") "
Sep 10 20:55:47.647821 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:47.647665    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c5a180c-786f-4556-8a89-ca0714d01bba-istio-token\") pod \"3c5a180c-786f-4556-8a89-ca0714d01bba\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") "
Sep 10 20:55:47.647821 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:47.647688    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c5a180c-786f-4556-8a89-ca0714d01bba-istio-podinfo\") pod \"3c5a180c-786f-4556-8a89-ca0714d01bba\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") "
Sep 10 20:55:47.647821 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:47.647703    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c5a180c-786f-4556-8a89-ca0714d01bba-workload-certs\") pod \"3c5a180c-786f-4556-8a89-ca0714d01bba\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") "
Sep 10 20:55:47.647821 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:47.647733    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c5a180c-786f-4556-8a89-ca0714d01bba-istio-data\") pod \"3c5a180c-786f-4556-8a89-ca0714d01bba\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") "
Sep 10 20:55:47.647821 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:47.647769    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c5a180c-786f-4556-8a89-ca0714d01bba-istiod-ca-cert\") pod \"3c5a180c-786f-4556-8a89-ca0714d01bba\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") "
Sep 10 20:55:47.647821 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:47.647801    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c5a180c-786f-4556-8a89-ca0714d01bba-credential-socket\") pod \"3c5a180c-786f-4556-8a89-ca0714d01bba\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") "
Sep 10 20:55:47.648130 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:47.647830    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c5a180c-786f-4556-8a89-ca0714d01bba-workload-socket\") pod \"3c5a180c-786f-4556-8a89-ca0714d01bba\" (UID: \"3c5a180c-786f-4556-8a89-ca0714d01bba\") "
Sep 10 20:55:47.648130 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:47.648049    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c5a180c-786f-4556-8a89-ca0714d01bba-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3c5a180c-786f-4556-8a89-ca0714d01bba" (UID: "3c5a180c-786f-4556-8a89-ca0714d01bba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:55:47.648130 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:47.648065    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c5a180c-786f-4556-8a89-ca0714d01bba-istio-data" (OuterVolumeSpecName: "istio-data") pod "3c5a180c-786f-4556-8a89-ca0714d01bba" (UID: "3c5a180c-786f-4556-8a89-ca0714d01bba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:55:47.648335 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:47.648241    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c5a180c-786f-4556-8a89-ca0714d01bba-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3c5a180c-786f-4556-8a89-ca0714d01bba" (UID: "3c5a180c-786f-4556-8a89-ca0714d01bba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:55:47.648335 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:47.648258    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c5a180c-786f-4556-8a89-ca0714d01bba-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3c5a180c-786f-4556-8a89-ca0714d01bba" (UID: "3c5a180c-786f-4556-8a89-ca0714d01bba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:55:47.648436 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:47.648347    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c5a180c-786f-4556-8a89-ca0714d01bba-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3c5a180c-786f-4556-8a89-ca0714d01bba" (UID: "3c5a180c-786f-4556-8a89-ca0714d01bba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:55:47.649936 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:47.649904    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c5a180c-786f-4556-8a89-ca0714d01bba-kube-api-access-ncz5v" (OuterVolumeSpecName: "kube-api-access-ncz5v") pod "3c5a180c-786f-4556-8a89-ca0714d01bba" (UID: "3c5a180c-786f-4556-8a89-ca0714d01bba"). InnerVolumeSpecName "kube-api-access-ncz5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:55:47.650049 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:47.649952    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c5a180c-786f-4556-8a89-ca0714d01bba-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3c5a180c-786f-4556-8a89-ca0714d01bba" (UID: "3c5a180c-786f-4556-8a89-ca0714d01bba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:55:47.650049 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:47.649961    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c5a180c-786f-4556-8a89-ca0714d01bba-istio-token" (OuterVolumeSpecName: "istio-token") pod "3c5a180c-786f-4556-8a89-ca0714d01bba" (UID: "3c5a180c-786f-4556-8a89-ca0714d01bba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:55:47.650049 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:47.650000    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c5a180c-786f-4556-8a89-ca0714d01bba-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3c5a180c-786f-4556-8a89-ca0714d01bba" (UID: "3c5a180c-786f-4556-8a89-ca0714d01bba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 20:55:47.749292 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:47.749264    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c5a180c-786f-4556-8a89-ca0714d01bba-istio-envoy\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:55:47.749292 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:47.749287    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncz5v\" (UniqueName: \"kubernetes.io/projected/3c5a180c-786f-4556-8a89-ca0714d01bba-kube-api-access-ncz5v\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:55:47.749292 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:47.749298    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c5a180c-786f-4556-8a89-ca0714d01bba-istio-token\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:55:47.749484 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:47.749307    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c5a180c-786f-4556-8a89-ca0714d01bba-istio-podinfo\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:55:47.749484 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:47.749315    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c5a180c-786f-4556-8a89-ca0714d01bba-workload-certs\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:55:47.749484 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:47.749323    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c5a180c-786f-4556-8a89-ca0714d01bba-istio-data\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:55:47.749484 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:47.749331    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c5a180c-786f-4556-8a89-ca0714d01bba-istiod-ca-cert\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:55:47.749484 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:47.749338    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c5a180c-786f-4556-8a89-ca0714d01bba-credential-socket\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:55:47.749484 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:47.749347    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c5a180c-786f-4556-8a89-ca0714d01bba-workload-socket\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:55:48.080976 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:48.080943    2564 generic.go:356] "Generic (PLEG): container finished" podID="3c5a180c-786f-4556-8a89-ca0714d01bba" containerID="40f6fd61590b53ef20f75570c0232b1ce6a73808f3c25cf7baf151e142daf78e" exitCode=0
Sep 10 20:55:48.081137 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:48.080991    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"
Sep 10 20:55:48.081137 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:48.081030    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl" event={"ID":"3c5a180c-786f-4556-8a89-ca0714d01bba","Type":"ContainerDied","Data":"40f6fd61590b53ef20f75570c0232b1ce6a73808f3c25cf7baf151e142daf78e"}
Sep 10 20:55:48.081137 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:48.081093    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl" event={"ID":"3c5a180c-786f-4556-8a89-ca0714d01bba","Type":"ContainerDied","Data":"b8f653795f0b55eeda89eb9e3ade8d5fa59240d78310a562af9a836774b51618"}
Sep 10 20:55:48.081137 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:48.081122    2564 scope.go:117] "RemoveContainer" containerID="40f6fd61590b53ef20f75570c0232b1ce6a73808f3c25cf7baf151e142daf78e"
Sep 10 20:55:48.091083 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:48.091069    2564 scope.go:117] "RemoveContainer" containerID="b0a511b28ef37f474bc70b7400e4972581e8e66547e9341cd8629b9c702d1602"
Sep 10 20:55:48.093822 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:48.093802    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk"
Sep 10 20:55:48.099404 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:48.099388    2564 scope.go:117] "RemoveContainer" containerID="40f6fd61590b53ef20f75570c0232b1ce6a73808f3c25cf7baf151e142daf78e"
Sep 10 20:55:48.099646 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:55:48.099625    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40f6fd61590b53ef20f75570c0232b1ce6a73808f3c25cf7baf151e142daf78e\": container with ID starting with 40f6fd61590b53ef20f75570c0232b1ce6a73808f3c25cf7baf151e142daf78e not found: ID does not exist" containerID="40f6fd61590b53ef20f75570c0232b1ce6a73808f3c25cf7baf151e142daf78e"
Sep 10 20:55:48.099695 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:48.099654    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40f6fd61590b53ef20f75570c0232b1ce6a73808f3c25cf7baf151e142daf78e"} err="failed to get container status \"40f6fd61590b53ef20f75570c0232b1ce6a73808f3c25cf7baf151e142daf78e\": rpc error: code = NotFound desc = could not find container \"40f6fd61590b53ef20f75570c0232b1ce6a73808f3c25cf7baf151e142daf78e\": container with ID starting with 40f6fd61590b53ef20f75570c0232b1ce6a73808f3c25cf7baf151e142daf78e not found: ID does not exist"
Sep 10 20:55:48.099695 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:48.099671    2564 scope.go:117] "RemoveContainer" containerID="b0a511b28ef37f474bc70b7400e4972581e8e66547e9341cd8629b9c702d1602"
Sep 10 20:55:48.099901 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:55:48.099883    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0a511b28ef37f474bc70b7400e4972581e8e66547e9341cd8629b9c702d1602\": container with ID starting with b0a511b28ef37f474bc70b7400e4972581e8e66547e9341cd8629b9c702d1602 not found: ID does not exist" containerID="b0a511b28ef37f474bc70b7400e4972581e8e66547e9341cd8629b9c702d1602"
Sep 10 20:55:48.099951 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:48.099911    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0a511b28ef37f474bc70b7400e4972581e8e66547e9341cd8629b9c702d1602"} err="failed to get container status \"b0a511b28ef37f474bc70b7400e4972581e8e66547e9341cd8629b9c702d1602\": rpc error: code = NotFound desc = could not find container \"b0a511b28ef37f474bc70b7400e4972581e8e66547e9341cd8629b9c702d1602\": container with ID starting with b0a511b28ef37f474bc70b7400e4972581e8e66547e9341cd8629b9c702d1602 not found: ID does not exist"
Sep 10 20:55:48.103021 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:48.103001    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"]
Sep 10 20:55:48.108281 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:48.108262    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-5d9b44-openshift-default-795c55bf46-rgzzl"]
Sep 10 20:55:48.540707 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:55:48.540632    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c5a180c-786f-4556-8a89-ca0714d01bba" path="/var/lib/kubelet/pods/3c5a180c-786f-4556-8a89-ca0714d01bba/volumes"
Sep 10 20:56:23.610081 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:23.610033    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz"]
Sep 10 20:56:23.610693 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:23.610580    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c5a180c-786f-4556-8a89-ca0714d01bba" containerName="istio-proxy"
Sep 10 20:56:23.610693 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:23.610603    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c5a180c-786f-4556-8a89-ca0714d01bba" containerName="istio-proxy"
Sep 10 20:56:23.610821 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:23.610717    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c5a180c-786f-4556-8a89-ca0714d01bba" containerName="istio-proxy"
Sep 10 20:56:23.610821 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:23.610728    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c5a180c-786f-4556-8a89-ca0714d01bba" containerName="istio-proxy"
Sep 10 20:56:23.615314 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:23.615290    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz"
Sep 10 20:56:23.617973 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:23.617930    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-75dcbb\"/\"kube-root-ca.crt\""
Sep 10 20:56:23.618680 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:23.618662    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-75dcbb\"/\"openshift-service-ca.crt\""
Sep 10 20:56:23.618794 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:23.618736    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-75dcbb\"/\"default-dockercfg-4tbdn\""
Sep 10 20:56:23.618898 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:23.618799    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-75dcbb\"/\"explicit-model-75dcbb-kserve-self-signed-certs\""
Sep 10 20:56:23.623465 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:23.623443    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz"]
Sep 10 20:56:23.720738 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:23.720713    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d33e09b-fc12-4dff-8c20-1cd98243db6a-dshm\") pod \"explicit-model-75dcbb-kserve-6c5649d587-k6bxz\" (UID: \"2d33e09b-fc12-4dff-8c20-1cd98243db6a\") " pod="ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz"
Sep 10 20:56:23.720859 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:23.720748    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d33e09b-fc12-4dff-8c20-1cd98243db6a-home\") pod \"explicit-model-75dcbb-kserve-6c5649d587-k6bxz\" (UID: \"2d33e09b-fc12-4dff-8c20-1cd98243db6a\") " pod="ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz"
Sep 10 20:56:23.720859 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:23.720768    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d33e09b-fc12-4dff-8c20-1cd98243db6a-tmp-dir\") pod \"explicit-model-75dcbb-kserve-6c5649d587-k6bxz\" (UID: \"2d33e09b-fc12-4dff-8c20-1cd98243db6a\") " pod="ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz"
Sep 10 20:56:23.720859 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:23.720787    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9br9\" (UniqueName: \"kubernetes.io/projected/2d33e09b-fc12-4dff-8c20-1cd98243db6a-kube-api-access-n9br9\") pod \"explicit-model-75dcbb-kserve-6c5649d587-k6bxz\" (UID: \"2d33e09b-fc12-4dff-8c20-1cd98243db6a\") " pod="ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz"
Sep 10 20:56:23.720859 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:23.720810    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d33e09b-fc12-4dff-8c20-1cd98243db6a-tls-certs\") pod \"explicit-model-75dcbb-kserve-6c5649d587-k6bxz\" (UID: \"2d33e09b-fc12-4dff-8c20-1cd98243db6a\") " pod="ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz"
Sep 10 20:56:23.720859 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:23.720858    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d33e09b-fc12-4dff-8c20-1cd98243db6a-model-cache\") pod \"explicit-model-75dcbb-kserve-6c5649d587-k6bxz\" (UID: \"2d33e09b-fc12-4dff-8c20-1cd98243db6a\") " pod="ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz"
Sep 10 20:56:23.821440 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:23.821412    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d33e09b-fc12-4dff-8c20-1cd98243db6a-dshm\") pod \"explicit-model-75dcbb-kserve-6c5649d587-k6bxz\" (UID: \"2d33e09b-fc12-4dff-8c20-1cd98243db6a\") " pod="ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz"
Sep 10 20:56:23.821570 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:23.821457    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d33e09b-fc12-4dff-8c20-1cd98243db6a-home\") pod \"explicit-model-75dcbb-kserve-6c5649d587-k6bxz\" (UID: \"2d33e09b-fc12-4dff-8c20-1cd98243db6a\") " pod="ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz"
Sep 10 20:56:23.821570 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:23.821485    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d33e09b-fc12-4dff-8c20-1cd98243db6a-tmp-dir\") pod \"explicit-model-75dcbb-kserve-6c5649d587-k6bxz\" (UID: \"2d33e09b-fc12-4dff-8c20-1cd98243db6a\") " pod="ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz"
Sep 10 20:56:23.821570 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:23.821509    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9br9\" (UniqueName: \"kubernetes.io/projected/2d33e09b-fc12-4dff-8c20-1cd98243db6a-kube-api-access-n9br9\") pod \"explicit-model-75dcbb-kserve-6c5649d587-k6bxz\" (UID: \"2d33e09b-fc12-4dff-8c20-1cd98243db6a\") " pod="ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz"
Sep 10 20:56:23.821570 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:23.821535    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d33e09b-fc12-4dff-8c20-1cd98243db6a-tls-certs\") pod \"explicit-model-75dcbb-kserve-6c5649d587-k6bxz\" (UID: \"2d33e09b-fc12-4dff-8c20-1cd98243db6a\") " pod="ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz"
Sep 10 20:56:23.821743 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:23.821592    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d33e09b-fc12-4dff-8c20-1cd98243db6a-model-cache\") pod \"explicit-model-75dcbb-kserve-6c5649d587-k6bxz\" (UID: \"2d33e09b-fc12-4dff-8c20-1cd98243db6a\") " pod="ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz"
Sep 10 20:56:23.821893 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:23.821870    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d33e09b-fc12-4dff-8c20-1cd98243db6a-tmp-dir\") pod \"explicit-model-75dcbb-kserve-6c5649d587-k6bxz\" (UID: \"2d33e09b-fc12-4dff-8c20-1cd98243db6a\") " pod="ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz"
Sep 10 20:56:23.821953 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:23.821879    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d33e09b-fc12-4dff-8c20-1cd98243db6a-home\") pod \"explicit-model-75dcbb-kserve-6c5649d587-k6bxz\" (UID: \"2d33e09b-fc12-4dff-8c20-1cd98243db6a\") " pod="ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz"
Sep 10 20:56:23.821953 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:23.821939    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d33e09b-fc12-4dff-8c20-1cd98243db6a-model-cache\") pod \"explicit-model-75dcbb-kserve-6c5649d587-k6bxz\" (UID: \"2d33e09b-fc12-4dff-8c20-1cd98243db6a\") " pod="ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz"
Sep 10 20:56:23.823689 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:23.823670    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d33e09b-fc12-4dff-8c20-1cd98243db6a-dshm\") pod \"explicit-model-75dcbb-kserve-6c5649d587-k6bxz\" (UID: \"2d33e09b-fc12-4dff-8c20-1cd98243db6a\") " pod="ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz"
Sep 10 20:56:23.823990 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:23.823974    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d33e09b-fc12-4dff-8c20-1cd98243db6a-tls-certs\") pod \"explicit-model-75dcbb-kserve-6c5649d587-k6bxz\" (UID: \"2d33e09b-fc12-4dff-8c20-1cd98243db6a\") " pod="ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz"
Sep 10 20:56:23.838207 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:23.838151    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9br9\" (UniqueName: \"kubernetes.io/projected/2d33e09b-fc12-4dff-8c20-1cd98243db6a-kube-api-access-n9br9\") pod \"explicit-model-75dcbb-kserve-6c5649d587-k6bxz\" (UID: \"2d33e09b-fc12-4dff-8c20-1cd98243db6a\") " pod="ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz"
Sep 10 20:56:23.927755 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:23.927699    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz"
Sep 10 20:56:24.203696 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:24.203667    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz"]
Sep 10 20:56:24.205514 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:56:24.205489    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d33e09b_fc12_4dff_8c20_1cd98243db6a.slice/crio-385f92155b7b57d4fba343d2a9111dacfee3783d79c197903032a48b9bb0a164 WatchSource:0}: Error finding container 385f92155b7b57d4fba343d2a9111dacfee3783d79c197903032a48b9bb0a164: Status 404 returned error can't find the container with id 385f92155b7b57d4fba343d2a9111dacfee3783d79c197903032a48b9bb0a164
Sep 10 20:56:24.280923 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:24.280899    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz" event={"ID":"2d33e09b-fc12-4dff-8c20-1cd98243db6a","Type":"ContainerStarted","Data":"c3c61b61bd93b069bdc5ab2d4877b0fa54a6279f471b473526f5948cd016f526"}
Sep 10 20:56:24.281028 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:24.280939    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz" event={"ID":"2d33e09b-fc12-4dff-8c20-1cd98243db6a","Type":"ContainerStarted","Data":"385f92155b7b57d4fba343d2a9111dacfee3783d79c197903032a48b9bb0a164"}
Sep 10 20:56:24.302401 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:24.302317    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz" podStartSLOduration=1.302267608 podStartE2EDuration="1.302267608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:56:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:56:24.30087768 +0000 UTC m=+1224.265945875" watchObservedRunningTime="2026-09-10 20:56:24.302267608 +0000 UTC m=+1224.267335805"
Sep 10 20:56:33.380493 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:33.380457    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz"]
Sep 10 20:56:33.517419 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:33.517384    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-75dcbb-48knz"]
Sep 10 20:56:33.517803 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:33.517785    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c5a180c-786f-4556-8a89-ca0714d01bba" containerName="istio-proxy"
Sep 10 20:56:33.517897 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:33.517806    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c5a180c-786f-4556-8a89-ca0714d01bba" containerName="istio-proxy"
Sep 10 20:56:33.520918 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:33.520897    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-75dcbb-48knz"
Sep 10 20:56:33.524422 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:33.524405    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plv2h\""
Sep 10 20:56:33.525214 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:33.525200    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 20:56:33.534393 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:33.534370    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-75dcbb-48knz"]
Sep 10 20:56:33.593174 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:33.593149    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac99faea-f500-4168-a4db-d96a5ed1f074-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-75dcbb-48knz\" (UID: \"ac99faea-f500-4168-a4db-d96a5ed1f074\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-75dcbb-48knz"
Sep 10 20:56:33.694325 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:33.694259    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac99faea-f500-4168-a4db-d96a5ed1f074-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-75dcbb-48knz\" (UID: \"ac99faea-f500-4168-a4db-d96a5ed1f074\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-75dcbb-48knz"
Sep 10 20:56:33.694868 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:33.694847    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac99faea-f500-4168-a4db-d96a5ed1f074-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-75dcbb-48knz\" (UID: \"ac99faea-f500-4168-a4db-d96a5ed1f074\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-75dcbb-48knz"
Sep 10 20:56:33.831080 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:33.831060    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-75dcbb-48knz"
Sep 10 20:56:33.928202 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:33.928162    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz"
Sep 10 20:56:34.107390 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:34.107367    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-75dcbb-48knz"]
Sep 10 20:56:34.109224 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:56:34.109196    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac99faea_f500_4168_a4db_d96a5ed1f074.slice/crio-98877914f79f1974e50cc684f031a7257557eaa1df2153ffc6718af40af7781b WatchSource:0}: Error finding container 98877914f79f1974e50cc684f031a7257557eaa1df2153ffc6718af40af7781b: Status 404 returned error can't find the container with id 98877914f79f1974e50cc684f031a7257557eaa1df2153ffc6718af40af7781b
Sep 10 20:56:34.111143 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:34.111122    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 20:56:34.341928 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:34.341897    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-75dcbb-48knz" event={"ID":"ac99faea-f500-4168-a4db-d96a5ed1f074","Type":"ContainerStarted","Data":"98877914f79f1974e50cc684f031a7257557eaa1df2153ffc6718af40af7781b"}
Sep 10 20:56:39.380152 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:39.380119    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-75dcbb-48knz" event={"ID":"ac99faea-f500-4168-a4db-d96a5ed1f074","Type":"ContainerStarted","Data":"db80012f62c206e66ddc909d84157bc948cd3e62a4d49e74198f00b6b201a158"}
Sep 10 20:56:39.401007 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:39.400958    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-75dcbb-48knz" podStartSLOduration=1.548378342 podStartE2EDuration="6.400941789s" totalImagesPullingTime="4.852563447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:56:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:56:39.399561437 +0000 UTC m=+1239.364629631" watchObservedRunningTime="2026-09-10 20:56:39.400941789 +0000 UTC m=+1239.366009985"
Sep 10 20:56:40.387766 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:40.387728    2564 generic.go:356] "Generic (PLEG): container finished" podID="ac99faea-f500-4168-a4db-d96a5ed1f074" containerID="db80012f62c206e66ddc909d84157bc948cd3e62a4d49e74198f00b6b201a158" exitCode=0
Sep 10 20:56:40.388143 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:40.387821    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-75dcbb-48knz" event={"ID":"ac99faea-f500-4168-a4db-d96a5ed1f074","Type":"ContainerDied","Data":"db80012f62c206e66ddc909d84157bc948cd3e62a4d49e74198f00b6b201a158"}
Sep 10 20:56:41.524982 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:41.524956    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-75dcbb-48knz"
Sep 10 20:56:41.561462 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:41.561436    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac99faea-f500-4168-a4db-d96a5ed1f074-maas-api-service-ca\") pod \"ac99faea-f500-4168-a4db-d96a5ed1f074\" (UID: \"ac99faea-f500-4168-a4db-d96a5ed1f074\") "
Sep 10 20:56:41.561737 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:41.561717    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac99faea-f500-4168-a4db-d96a5ed1f074-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ac99faea-f500-4168-a4db-d96a5ed1f074" (UID: "ac99faea-f500-4168-a4db-d96a5ed1f074"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:56:41.662134 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:41.662066    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac99faea-f500-4168-a4db-d96a5ed1f074-maas-api-service-ca\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:56:42.401320 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:42.401284    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-75dcbb-48knz"
Sep 10 20:56:42.401320 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:42.401302    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-75dcbb-48knz" event={"ID":"ac99faea-f500-4168-a4db-d96a5ed1f074","Type":"ContainerDied","Data":"98877914f79f1974e50cc684f031a7257557eaa1df2153ffc6718af40af7781b"}
Sep 10 20:56:42.401541 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:42.401349    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98877914f79f1974e50cc684f031a7257557eaa1df2153ffc6718af40af7781b"
Sep 10 20:56:44.546112 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:44.546079    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-75dcbb-48knz"]
Sep 10 20:56:44.549249 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:44.549225    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-75dcbb-48knz"]
Sep 10 20:56:45.905710 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:45.905678    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fc6e011f-57b64c9b57-hwj9x"]
Sep 10 20:56:45.906047 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:45.905999    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac99faea-f500-4168-a4db-d96a5ed1f074" containerName="revoke-keys"
Sep 10 20:56:45.906047 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:45.906009    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac99faea-f500-4168-a4db-d96a5ed1f074" containerName="revoke-keys"
Sep 10 20:56:45.906642 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:45.906609    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac99faea-f500-4168-a4db-d96a5ed1f074" containerName="revoke-keys"
Sep 10 20:56:45.910768 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:45.910741    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fc6e011f-57b64c9b57-hwj9x"
Sep 10 20:56:45.924028 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:45.924003    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-fc6e011f-dockercfg-jvmvl\""
Sep 10 20:56:45.924182 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:45.924004    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-fc6e011f\""
Sep 10 20:56:45.931645 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:45.931622    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fc6e011f-57b64c9b57-hwj9x"]
Sep 10 20:56:45.994261 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:45.994227    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78291f5f-f180-4c2f-b36e-24181627b9e8-plugins-config-volume\") pod \"payload-processing-e2e-mt-fc6e011f-57b64c9b57-hwj9x\" (UID: \"78291f5f-f180-4c2f-b36e-24181627b9e8\") " pod="openshift-ingress/payload-processing-e2e-mt-fc6e011f-57b64c9b57-hwj9x"
Sep 10 20:56:45.994398 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:45.994318    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vltbr\" (UniqueName: \"kubernetes.io/projected/78291f5f-f180-4c2f-b36e-24181627b9e8-kube-api-access-vltbr\") pod \"payload-processing-e2e-mt-fc6e011f-57b64c9b57-hwj9x\" (UID: \"78291f5f-f180-4c2f-b36e-24181627b9e8\") " pod="openshift-ingress/payload-processing-e2e-mt-fc6e011f-57b64c9b57-hwj9x"
Sep 10 20:56:46.095512 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:46.095485    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vltbr\" (UniqueName: \"kubernetes.io/projected/78291f5f-f180-4c2f-b36e-24181627b9e8-kube-api-access-vltbr\") pod \"payload-processing-e2e-mt-fc6e011f-57b64c9b57-hwj9x\" (UID: \"78291f5f-f180-4c2f-b36e-24181627b9e8\") " pod="openshift-ingress/payload-processing-e2e-mt-fc6e011f-57b64c9b57-hwj9x"
Sep 10 20:56:46.095633 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:46.095544    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78291f5f-f180-4c2f-b36e-24181627b9e8-plugins-config-volume\") pod \"payload-processing-e2e-mt-fc6e011f-57b64c9b57-hwj9x\" (UID: \"78291f5f-f180-4c2f-b36e-24181627b9e8\") " pod="openshift-ingress/payload-processing-e2e-mt-fc6e011f-57b64c9b57-hwj9x"
Sep 10 20:56:46.096242 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:46.096224    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78291f5f-f180-4c2f-b36e-24181627b9e8-plugins-config-volume\") pod \"payload-processing-e2e-mt-fc6e011f-57b64c9b57-hwj9x\" (UID: \"78291f5f-f180-4c2f-b36e-24181627b9e8\") " pod="openshift-ingress/payload-processing-e2e-mt-fc6e011f-57b64c9b57-hwj9x"
Sep 10 20:56:46.104826 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:46.104804    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vltbr\" (UniqueName: \"kubernetes.io/projected/78291f5f-f180-4c2f-b36e-24181627b9e8-kube-api-access-vltbr\") pod \"payload-processing-e2e-mt-fc6e011f-57b64c9b57-hwj9x\" (UID: \"78291f5f-f180-4c2f-b36e-24181627b9e8\") " pod="openshift-ingress/payload-processing-e2e-mt-fc6e011f-57b64c9b57-hwj9x"
Sep 10 20:56:46.223888 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:46.223826    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fc6e011f-57b64c9b57-hwj9x"
Sep 10 20:56:46.503698 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:46.503676    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fc6e011f-57b64c9b57-hwj9x"]
Sep 10 20:56:46.506000 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:56:46.505966    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78291f5f_f180_4c2f_b36e_24181627b9e8.slice/crio-0bb127aa167e5564bdcbd1b13a6d8bf464d77bc71b52fa3a1e0688dadc22708f WatchSource:0}: Error finding container 0bb127aa167e5564bdcbd1b13a6d8bf464d77bc71b52fa3a1e0688dadc22708f: Status 404 returned error can't find the container with id 0bb127aa167e5564bdcbd1b13a6d8bf464d77bc71b52fa3a1e0688dadc22708f
Sep 10 20:56:46.540660 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:46.540633    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac99faea-f500-4168-a4db-d96a5ed1f074" path="/var/lib/kubelet/pods/ac99faea-f500-4168-a4db-d96a5ed1f074/volumes"
Sep 10 20:56:47.429309 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:47.429274    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fc6e011f-57b64c9b57-hwj9x" event={"ID":"78291f5f-f180-4c2f-b36e-24181627b9e8","Type":"ContainerStarted","Data":"e4707936bfd353073f6c734fb6c7706715ead8971e61cca7079ae8b60f6418f8"}
Sep 10 20:56:47.429671 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:47.429327    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fc6e011f-57b64c9b57-hwj9x" event={"ID":"78291f5f-f180-4c2f-b36e-24181627b9e8","Type":"ContainerStarted","Data":"0bb127aa167e5564bdcbd1b13a6d8bf464d77bc71b52fa3a1e0688dadc22708f"}
Sep 10 20:56:47.429671 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:47.429406    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-fc6e011f-57b64c9b57-hwj9x"
Sep 10 20:56:47.452519 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:47.452467    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-fc6e011f-57b64c9b57-hwj9x" podStartSLOduration=2.452453404 podStartE2EDuration="2.452453404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:56:47.449955928 +0000 UTC m=+1247.415024123" watchObservedRunningTime="2026-09-10 20:56:47.452453404 +0000 UTC m=+1247.417521633"
Sep 10 20:56:48.398801 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:48.398760    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz" podUID="2d33e09b-fc12-4dff-8c20-1cd98243db6a" containerName="main" containerID="cri-o://c3c61b61bd93b069bdc5ab2d4877b0fa54a6279f471b473526f5948cd016f526" gracePeriod=45
Sep 10 20:56:48.665605 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:48.665524    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz"
Sep 10 20:56:48.716948 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:48.716918    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d33e09b-fc12-4dff-8c20-1cd98243db6a-model-cache\") pod \"2d33e09b-fc12-4dff-8c20-1cd98243db6a\" (UID: \"2d33e09b-fc12-4dff-8c20-1cd98243db6a\") "
Sep 10 20:56:48.717131 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:48.716980    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d33e09b-fc12-4dff-8c20-1cd98243db6a-dshm\") pod \"2d33e09b-fc12-4dff-8c20-1cd98243db6a\" (UID: \"2d33e09b-fc12-4dff-8c20-1cd98243db6a\") "
Sep 10 20:56:48.717131 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:48.717006    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d33e09b-fc12-4dff-8c20-1cd98243db6a-tmp-dir\") pod \"2d33e09b-fc12-4dff-8c20-1cd98243db6a\" (UID: \"2d33e09b-fc12-4dff-8c20-1cd98243db6a\") "
Sep 10 20:56:48.717131 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:48.717039    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d33e09b-fc12-4dff-8c20-1cd98243db6a-home\") pod \"2d33e09b-fc12-4dff-8c20-1cd98243db6a\" (UID: \"2d33e09b-fc12-4dff-8c20-1cd98243db6a\") "
Sep 10 20:56:48.717131 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:48.717063    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9br9\" (UniqueName: \"kubernetes.io/projected/2d33e09b-fc12-4dff-8c20-1cd98243db6a-kube-api-access-n9br9\") pod \"2d33e09b-fc12-4dff-8c20-1cd98243db6a\" (UID: \"2d33e09b-fc12-4dff-8c20-1cd98243db6a\") "
Sep 10 20:56:48.717131 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:48.717108    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d33e09b-fc12-4dff-8c20-1cd98243db6a-tls-certs\") pod \"2d33e09b-fc12-4dff-8c20-1cd98243db6a\" (UID: \"2d33e09b-fc12-4dff-8c20-1cd98243db6a\") "
Sep 10 20:56:48.717462 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:48.717161    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d33e09b-fc12-4dff-8c20-1cd98243db6a-model-cache" (OuterVolumeSpecName: "model-cache") pod "2d33e09b-fc12-4dff-8c20-1cd98243db6a" (UID: "2d33e09b-fc12-4dff-8c20-1cd98243db6a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:56:48.717462 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:48.717277    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d33e09b-fc12-4dff-8c20-1cd98243db6a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2d33e09b-fc12-4dff-8c20-1cd98243db6a" (UID: "2d33e09b-fc12-4dff-8c20-1cd98243db6a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:56:48.717462 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:48.717313    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d33e09b-fc12-4dff-8c20-1cd98243db6a-home" (OuterVolumeSpecName: "home") pod "2d33e09b-fc12-4dff-8c20-1cd98243db6a" (UID: "2d33e09b-fc12-4dff-8c20-1cd98243db6a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:56:48.717610 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:48.717467    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d33e09b-fc12-4dff-8c20-1cd98243db6a-tmp-dir\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:56:48.717610 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:48.717486    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d33e09b-fc12-4dff-8c20-1cd98243db6a-home\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:56:48.717610 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:48.717501    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d33e09b-fc12-4dff-8c20-1cd98243db6a-model-cache\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:56:48.719379 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:48.719351    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d33e09b-fc12-4dff-8c20-1cd98243db6a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2d33e09b-fc12-4dff-8c20-1cd98243db6a" (UID: "2d33e09b-fc12-4dff-8c20-1cd98243db6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:56:48.719379 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:48.719368    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d33e09b-fc12-4dff-8c20-1cd98243db6a-dshm" (OuterVolumeSpecName: "dshm") pod "2d33e09b-fc12-4dff-8c20-1cd98243db6a" (UID: "2d33e09b-fc12-4dff-8c20-1cd98243db6a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:56:48.719518 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:48.719379    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d33e09b-fc12-4dff-8c20-1cd98243db6a-kube-api-access-n9br9" (OuterVolumeSpecName: "kube-api-access-n9br9") pod "2d33e09b-fc12-4dff-8c20-1cd98243db6a" (UID: "2d33e09b-fc12-4dff-8c20-1cd98243db6a"). InnerVolumeSpecName "kube-api-access-n9br9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:56:48.818138 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:48.818109    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d33e09b-fc12-4dff-8c20-1cd98243db6a-dshm\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:56:48.818138 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:48.818135    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9br9\" (UniqueName: \"kubernetes.io/projected/2d33e09b-fc12-4dff-8c20-1cd98243db6a-kube-api-access-n9br9\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:56:48.818138 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:48.818145    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d33e09b-fc12-4dff-8c20-1cd98243db6a-tls-certs\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:56:49.439748 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:49.439711    2564 generic.go:356] "Generic (PLEG): container finished" podID="2d33e09b-fc12-4dff-8c20-1cd98243db6a" containerID="c3c61b61bd93b069bdc5ab2d4877b0fa54a6279f471b473526f5948cd016f526" exitCode=0
Sep 10 20:56:49.439902 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:49.439759    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz"
Sep 10 20:56:49.439902 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:49.439756    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz" event={"ID":"2d33e09b-fc12-4dff-8c20-1cd98243db6a","Type":"ContainerDied","Data":"c3c61b61bd93b069bdc5ab2d4877b0fa54a6279f471b473526f5948cd016f526"}
Sep 10 20:56:49.439902 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:49.439883    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz" event={"ID":"2d33e09b-fc12-4dff-8c20-1cd98243db6a","Type":"ContainerDied","Data":"385f92155b7b57d4fba343d2a9111dacfee3783d79c197903032a48b9bb0a164"}
Sep 10 20:56:49.440012 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:49.439912    2564 scope.go:117] "RemoveContainer" containerID="c3c61b61bd93b069bdc5ab2d4877b0fa54a6279f471b473526f5948cd016f526"
Sep 10 20:56:49.449046 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:49.449026    2564 scope.go:117] "RemoveContainer" containerID="c3c61b61bd93b069bdc5ab2d4877b0fa54a6279f471b473526f5948cd016f526"
Sep 10 20:56:49.449299 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:56:49.449281    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3c61b61bd93b069bdc5ab2d4877b0fa54a6279f471b473526f5948cd016f526\": container with ID starting with c3c61b61bd93b069bdc5ab2d4877b0fa54a6279f471b473526f5948cd016f526 not found: ID does not exist" containerID="c3c61b61bd93b069bdc5ab2d4877b0fa54a6279f471b473526f5948cd016f526"
Sep 10 20:56:49.449364 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:49.449310    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3c61b61bd93b069bdc5ab2d4877b0fa54a6279f471b473526f5948cd016f526"} err="failed to get container status \"c3c61b61bd93b069bdc5ab2d4877b0fa54a6279f471b473526f5948cd016f526\": rpc error: code = NotFound desc = could not find container \"c3c61b61bd93b069bdc5ab2d4877b0fa54a6279f471b473526f5948cd016f526\": container with ID starting with c3c61b61bd93b069bdc5ab2d4877b0fa54a6279f471b473526f5948cd016f526 not found: ID does not exist"
Sep 10 20:56:49.474765 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:49.474739    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz"]
Sep 10 20:56:49.490815 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:49.490789    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-75dcbb/explicit-model-75dcbb-kserve-6c5649d587-k6bxz"]
Sep 10 20:56:50.545995 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:50.545962    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d33e09b-fc12-4dff-8c20-1cd98243db6a" path="/var/lib/kubelet/pods/2d33e09b-fc12-4dff-8c20-1cd98243db6a/volumes"
Sep 10 20:56:58.436398 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:56:58.436296    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-fc6e011f-57b64c9b57-hwj9x"
Sep 10 20:57:17.660866 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:17.660827    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc6e011f-hq7pl"]
Sep 10 20:57:17.661438 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:17.661336    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d33e09b-fc12-4dff-8c20-1cd98243db6a" containerName="main"
Sep 10 20:57:17.661438 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:17.661357    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d33e09b-fc12-4dff-8c20-1cd98243db6a" containerName="main"
Sep 10 20:57:17.661561 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:17.661461    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d33e09b-fc12-4dff-8c20-1cd98243db6a" containerName="main"
Sep 10 20:57:17.669554 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:17.669532    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc6e011f-hq7pl"
Sep 10 20:57:17.675253 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:17.675229    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 20:57:17.675445 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:17.675232    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plv2h\""
Sep 10 20:57:17.675920 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:17.675902    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc6e011f-hq7pl"]
Sep 10 20:57:17.740386 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:17.740356    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ca34531-dc05-421e-adb6-f6d62a8ecabe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fc6e011f-hq7pl\" (UID: \"3ca34531-dc05-421e-adb6-f6d62a8ecabe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc6e011f-hq7pl"
Sep 10 20:57:17.841288 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:17.841254    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ca34531-dc05-421e-adb6-f6d62a8ecabe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fc6e011f-hq7pl\" (UID: \"3ca34531-dc05-421e-adb6-f6d62a8ecabe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc6e011f-hq7pl"
Sep 10 20:57:17.841835 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:17.841818    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ca34531-dc05-421e-adb6-f6d62a8ecabe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fc6e011f-hq7pl\" (UID: \"3ca34531-dc05-421e-adb6-f6d62a8ecabe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc6e011f-hq7pl"
Sep 10 20:57:17.981327 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:17.981261    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc6e011f-hq7pl"
Sep 10 20:57:18.257743 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:18.257716    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc6e011f-hq7pl"]
Sep 10 20:57:18.260107 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:57:18.260073    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca34531_dc05_421e_adb6_f6d62a8ecabe.slice/crio-363c326f3cb889891f5769157c14697a113ef54bd45ddfe07f359a3749abedef WatchSource:0}: Error finding container 363c326f3cb889891f5769157c14697a113ef54bd45ddfe07f359a3749abedef: Status 404 returned error can't find the container with id 363c326f3cb889891f5769157c14697a113ef54bd45ddfe07f359a3749abedef
Sep 10 20:57:18.593472 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:18.593440    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc6e011f-hq7pl" event={"ID":"3ca34531-dc05-421e-adb6-f6d62a8ecabe","Type":"ContainerStarted","Data":"a978d9d9a1185f9a114a54b521442c3c6f7797c9c27abe74191d3b09fa49de1d"}
Sep 10 20:57:18.593592 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:18.593496    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc6e011f-hq7pl" event={"ID":"3ca34531-dc05-421e-adb6-f6d62a8ecabe","Type":"ContainerStarted","Data":"363c326f3cb889891f5769157c14697a113ef54bd45ddfe07f359a3749abedef"}
Sep 10 20:57:18.611749 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:18.611706    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc6e011f-hq7pl" podStartSLOduration=1.611693298 podStartE2EDuration="1.611693298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:57:18.610730232 +0000 UTC m=+1278.575798428" watchObservedRunningTime="2026-09-10 20:57:18.611693298 +0000 UTC m=+1278.576761509"
Sep 10 20:57:19.602062 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:19.602032    2564 generic.go:356] "Generic (PLEG): container finished" podID="3ca34531-dc05-421e-adb6-f6d62a8ecabe" containerID="a978d9d9a1185f9a114a54b521442c3c6f7797c9c27abe74191d3b09fa49de1d" exitCode=0
Sep 10 20:57:19.602444 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:19.602110    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc6e011f-hq7pl" event={"ID":"3ca34531-dc05-421e-adb6-f6d62a8ecabe","Type":"ContainerDied","Data":"a978d9d9a1185f9a114a54b521442c3c6f7797c9c27abe74191d3b09fa49de1d"}
Sep 10 20:57:20.735947 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:20.735926    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc6e011f-hq7pl"
Sep 10 20:57:20.867718 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:20.867643    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ca34531-dc05-421e-adb6-f6d62a8ecabe-maas-api-service-ca\") pod \"3ca34531-dc05-421e-adb6-f6d62a8ecabe\" (UID: \"3ca34531-dc05-421e-adb6-f6d62a8ecabe\") "
Sep 10 20:57:20.867949 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:20.867928    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ca34531-dc05-421e-adb6-f6d62a8ecabe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3ca34531-dc05-421e-adb6-f6d62a8ecabe" (UID: "3ca34531-dc05-421e-adb6-f6d62a8ecabe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:57:20.974701 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:20.969896    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ca34531-dc05-421e-adb6-f6d62a8ecabe-maas-api-service-ca\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:57:21.613410 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:21.613384    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc6e011f-hq7pl"
Sep 10 20:57:21.613578 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:21.613427    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc6e011f-hq7pl" event={"ID":"3ca34531-dc05-421e-adb6-f6d62a8ecabe","Type":"ContainerDied","Data":"363c326f3cb889891f5769157c14697a113ef54bd45ddfe07f359a3749abedef"}
Sep 10 20:57:21.613578 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:21.613473    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="363c326f3cb889891f5769157c14697a113ef54bd45ddfe07f359a3749abedef"
Sep 10 20:57:28.351429 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:28.351394    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fc6e011f-57b64c9b57-hwj9x"]
Sep 10 20:57:28.351923 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:28.351675    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-fc6e011f-57b64c9b57-hwj9x" podUID="78291f5f-f180-4c2f-b36e-24181627b9e8" containerName="payload-processing" containerID="cri-o://e4707936bfd353073f6c734fb6c7706715ead8971e61cca7079ae8b60f6418f8" gracePeriod=30
Sep 10 20:57:28.435079 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:28.435039    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-fc6e011f-57b64c9b57-hwj9x" podUID="78291f5f-f180-4c2f-b36e-24181627b9e8" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.51:9004: connect: connection refused"
Sep 10 20:57:28.611050 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:28.610990    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fc6e011f-57b64c9b57-hwj9x"
Sep 10 20:57:28.649744 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:28.649706    2564 generic.go:356] "Generic (PLEG): container finished" podID="78291f5f-f180-4c2f-b36e-24181627b9e8" containerID="e4707936bfd353073f6c734fb6c7706715ead8971e61cca7079ae8b60f6418f8" exitCode=0
Sep 10 20:57:28.649920 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:28.649742    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fc6e011f-57b64c9b57-hwj9x" event={"ID":"78291f5f-f180-4c2f-b36e-24181627b9e8","Type":"ContainerDied","Data":"e4707936bfd353073f6c734fb6c7706715ead8971e61cca7079ae8b60f6418f8"}
Sep 10 20:57:28.649920 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:28.649813    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fc6e011f-57b64c9b57-hwj9x" event={"ID":"78291f5f-f180-4c2f-b36e-24181627b9e8","Type":"ContainerDied","Data":"0bb127aa167e5564bdcbd1b13a6d8bf464d77bc71b52fa3a1e0688dadc22708f"}
Sep 10 20:57:28.649920 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:28.649846    2564 scope.go:117] "RemoveContainer" containerID="e4707936bfd353073f6c734fb6c7706715ead8971e61cca7079ae8b60f6418f8"
Sep 10 20:57:28.650373 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:28.650335    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fc6e011f-57b64c9b57-hwj9x"
Sep 10 20:57:28.659268 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:28.659246    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc6e011f-hq7pl"]
Sep 10 20:57:28.660360 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:28.660338    2564 scope.go:117] "RemoveContainer" containerID="e4707936bfd353073f6c734fb6c7706715ead8971e61cca7079ae8b60f6418f8"
Sep 10 20:57:28.660663 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:57:28.660642    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4707936bfd353073f6c734fb6c7706715ead8971e61cca7079ae8b60f6418f8\": container with ID starting with e4707936bfd353073f6c734fb6c7706715ead8971e61cca7079ae8b60f6418f8 not found: ID does not exist" containerID="e4707936bfd353073f6c734fb6c7706715ead8971e61cca7079ae8b60f6418f8"
Sep 10 20:57:28.660744 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:28.660670    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4707936bfd353073f6c734fb6c7706715ead8971e61cca7079ae8b60f6418f8"} err="failed to get container status \"e4707936bfd353073f6c734fb6c7706715ead8971e61cca7079ae8b60f6418f8\": rpc error: code = NotFound desc = could not find container \"e4707936bfd353073f6c734fb6c7706715ead8971e61cca7079ae8b60f6418f8\": container with ID starting with e4707936bfd353073f6c734fb6c7706715ead8971e61cca7079ae8b60f6418f8 not found: ID does not exist"
Sep 10 20:57:28.665965 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:28.665941    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc6e011f-hq7pl"]
Sep 10 20:57:28.736212 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:28.736183    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vltbr\" (UniqueName: \"kubernetes.io/projected/78291f5f-f180-4c2f-b36e-24181627b9e8-kube-api-access-vltbr\") pod \"78291f5f-f180-4c2f-b36e-24181627b9e8\" (UID: \"78291f5f-f180-4c2f-b36e-24181627b9e8\") "
Sep 10 20:57:28.736396 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:28.736230    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78291f5f-f180-4c2f-b36e-24181627b9e8-plugins-config-volume\") pod \"78291f5f-f180-4c2f-b36e-24181627b9e8\" (UID: \"78291f5f-f180-4c2f-b36e-24181627b9e8\") "
Sep 10 20:57:28.736658 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:28.736630    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78291f5f-f180-4c2f-b36e-24181627b9e8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "78291f5f-f180-4c2f-b36e-24181627b9e8" (UID: "78291f5f-f180-4c2f-b36e-24181627b9e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:57:28.738293 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:28.738273    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78291f5f-f180-4c2f-b36e-24181627b9e8-kube-api-access-vltbr" (OuterVolumeSpecName: "kube-api-access-vltbr") pod "78291f5f-f180-4c2f-b36e-24181627b9e8" (UID: "78291f5f-f180-4c2f-b36e-24181627b9e8"). InnerVolumeSpecName "kube-api-access-vltbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:57:28.837613 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:28.837580    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vltbr\" (UniqueName: \"kubernetes.io/projected/78291f5f-f180-4c2f-b36e-24181627b9e8-kube-api-access-vltbr\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:57:28.837613 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:28.837610    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78291f5f-f180-4c2f-b36e-24181627b9e8-plugins-config-volume\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:57:28.977220 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:28.977182    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fc6e011f-57b64c9b57-hwj9x"]
Sep 10 20:57:28.979092 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:28.979069    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fc6e011f-57b64c9b57-hwj9x"]
Sep 10 20:57:30.541274 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:30.541237    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ca34531-dc05-421e-adb6-f6d62a8ecabe" path="/var/lib/kubelet/pods/3ca34531-dc05-421e-adb6-f6d62a8ecabe/volumes"
Sep 10 20:57:30.541653 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:30.541622    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78291f5f-f180-4c2f-b36e-24181627b9e8" path="/var/lib/kubelet/pods/78291f5f-f180-4c2f-b36e-24181627b9e8/volumes"
Sep 10 20:57:34.174787 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:34.174750    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t"]
Sep 10 20:57:34.704737 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:34.704704    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk"]
Sep 10 20:57:46.198182 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:46.198134    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-80aa25-f5c49c754-7kwng"]
Sep 10 20:57:46.198584 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:46.198398    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80aa25-f5c49c754-7kwng" podUID="f0e77222-8445-4c91-8999-8686eb8dabb4" containerName="payload-pre-processing" containerID="cri-o://76e57605e83c03c79d55c24fabd1ee9207595a2fa2348c343a321c4ba5ef6df1" gracePeriod=30
Sep 10 20:57:46.452913 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:46.452846    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80aa25-f5c49c754-7kwng"
Sep 10 20:57:46.477388 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:46.477357    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmb9c\" (UniqueName: \"kubernetes.io/projected/f0e77222-8445-4c91-8999-8686eb8dabb4-kube-api-access-mmb9c\") pod \"f0e77222-8445-4c91-8999-8686eb8dabb4\" (UID: \"f0e77222-8445-4c91-8999-8686eb8dabb4\") "
Sep 10 20:57:46.477549 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:46.477463    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e77222-8445-4c91-8999-8686eb8dabb4-plugins-config-volume\") pod \"f0e77222-8445-4c91-8999-8686eb8dabb4\" (UID: \"f0e77222-8445-4c91-8999-8686eb8dabb4\") "
Sep 10 20:57:46.477806 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:46.477772    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0e77222-8445-4c91-8999-8686eb8dabb4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f0e77222-8445-4c91-8999-8686eb8dabb4" (UID: "f0e77222-8445-4c91-8999-8686eb8dabb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:57:46.479518 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:46.479497    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0e77222-8445-4c91-8999-8686eb8dabb4-kube-api-access-mmb9c" (OuterVolumeSpecName: "kube-api-access-mmb9c") pod "f0e77222-8445-4c91-8999-8686eb8dabb4" (UID: "f0e77222-8445-4c91-8999-8686eb8dabb4"). InnerVolumeSpecName "kube-api-access-mmb9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:57:46.578388 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:46.578358    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e77222-8445-4c91-8999-8686eb8dabb4-plugins-config-volume\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:57:46.578388 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:46.578383    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmb9c\" (UniqueName: \"kubernetes.io/projected/f0e77222-8445-4c91-8999-8686eb8dabb4-kube-api-access-mmb9c\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:57:46.750693 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:46.750617    2564 generic.go:356] "Generic (PLEG): container finished" podID="f0e77222-8445-4c91-8999-8686eb8dabb4" containerID="76e57605e83c03c79d55c24fabd1ee9207595a2fa2348c343a321c4ba5ef6df1" exitCode=0
Sep 10 20:57:46.750693 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:46.750661    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80aa25-f5c49c754-7kwng"
Sep 10 20:57:46.750930 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:46.750709    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80aa25-f5c49c754-7kwng" event={"ID":"f0e77222-8445-4c91-8999-8686eb8dabb4","Type":"ContainerDied","Data":"76e57605e83c03c79d55c24fabd1ee9207595a2fa2348c343a321c4ba5ef6df1"}
Sep 10 20:57:46.750930 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:46.750768    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80aa25-f5c49c754-7kwng" event={"ID":"f0e77222-8445-4c91-8999-8686eb8dabb4","Type":"ContainerDied","Data":"a90e32e3e1d4e07864d0b8261631eb74226a571376bebbb9e433ccd3452b6a6b"}
Sep 10 20:57:46.750930 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:46.750799    2564 scope.go:117] "RemoveContainer" containerID="76e57605e83c03c79d55c24fabd1ee9207595a2fa2348c343a321c4ba5ef6df1"
Sep 10 20:57:46.759401 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:46.759382    2564 scope.go:117] "RemoveContainer" containerID="76e57605e83c03c79d55c24fabd1ee9207595a2fa2348c343a321c4ba5ef6df1"
Sep 10 20:57:46.759712 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:57:46.759689    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76e57605e83c03c79d55c24fabd1ee9207595a2fa2348c343a321c4ba5ef6df1\": container with ID starting with 76e57605e83c03c79d55c24fabd1ee9207595a2fa2348c343a321c4ba5ef6df1 not found: ID does not exist" containerID="76e57605e83c03c79d55c24fabd1ee9207595a2fa2348c343a321c4ba5ef6df1"
Sep 10 20:57:46.759790 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:46.759718    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76e57605e83c03c79d55c24fabd1ee9207595a2fa2348c343a321c4ba5ef6df1"} err="failed to get container status \"76e57605e83c03c79d55c24fabd1ee9207595a2fa2348c343a321c4ba5ef6df1\": rpc error: code = NotFound desc = could not find container \"76e57605e83c03c79d55c24fabd1ee9207595a2fa2348c343a321c4ba5ef6df1\": container with ID starting with 76e57605e83c03c79d55c24fabd1ee9207595a2fa2348c343a321c4ba5ef6df1 not found: ID does not exist"
Sep 10 20:57:46.782061 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:46.782032    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-80aa25-f5c49c754-7kwng"]
Sep 10 20:57:46.783784 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:46.783762    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-80aa25-f5c49c754-7kwng"]
Sep 10 20:57:48.542890 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:48.542858    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0e77222-8445-4c91-8999-8686eb8dabb4" path="/var/lib/kubelet/pods/f0e77222-8445-4c91-8999-8686eb8dabb4/volumes"
Sep 10 20:57:49.192921 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:49.192882    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t" podUID="7b3755d8-15be-417c-8661-1381d26afd0d" containerName="main" containerID="cri-o://f3ad9a26dad2c203cdcf635db60cb7dfd8e221f08249bac68208fe35253ed343" gracePeriod=45
Sep 10 20:57:49.443603 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:49.443546    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t"
Sep 10 20:57:49.503350 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:49.503312    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b3755d8-15be-417c-8661-1381d26afd0d-home\") pod \"7b3755d8-15be-417c-8661-1381d26afd0d\" (UID: \"7b3755d8-15be-417c-8661-1381d26afd0d\") "
Sep 10 20:57:49.503350 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:49.503345    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3755d8-15be-417c-8661-1381d26afd0d-tls-certs\") pod \"7b3755d8-15be-417c-8661-1381d26afd0d\" (UID: \"7b3755d8-15be-417c-8661-1381d26afd0d\") "
Sep 10 20:57:49.503598 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:49.503366    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b3755d8-15be-417c-8661-1381d26afd0d-model-cache\") pod \"7b3755d8-15be-417c-8661-1381d26afd0d\" (UID: \"7b3755d8-15be-417c-8661-1381d26afd0d\") "
Sep 10 20:57:49.503598 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:49.503400    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b3755d8-15be-417c-8661-1381d26afd0d-dshm\") pod \"7b3755d8-15be-417c-8661-1381d26afd0d\" (UID: \"7b3755d8-15be-417c-8661-1381d26afd0d\") "
Sep 10 20:57:49.503598 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:49.503434    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z242p\" (UniqueName: \"kubernetes.io/projected/7b3755d8-15be-417c-8661-1381d26afd0d-kube-api-access-z242p\") pod \"7b3755d8-15be-417c-8661-1381d26afd0d\" (UID: \"7b3755d8-15be-417c-8661-1381d26afd0d\") "
Sep 10 20:57:49.503598 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:49.503492    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b3755d8-15be-417c-8661-1381d26afd0d-tmp-dir\") pod \"7b3755d8-15be-417c-8661-1381d26afd0d\" (UID: \"7b3755d8-15be-417c-8661-1381d26afd0d\") "
Sep 10 20:57:49.503822 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:49.503590    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b3755d8-15be-417c-8661-1381d26afd0d-model-cache" (OuterVolumeSpecName: "model-cache") pod "7b3755d8-15be-417c-8661-1381d26afd0d" (UID: "7b3755d8-15be-417c-8661-1381d26afd0d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:57:49.503822 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:49.503626    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b3755d8-15be-417c-8661-1381d26afd0d-home" (OuterVolumeSpecName: "home") pod "7b3755d8-15be-417c-8661-1381d26afd0d" (UID: "7b3755d8-15be-417c-8661-1381d26afd0d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:57:49.503822 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:49.503761    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b3755d8-15be-417c-8661-1381d26afd0d-home\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:57:49.503822 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:49.503779    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b3755d8-15be-417c-8661-1381d26afd0d-model-cache\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:57:49.504045 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:49.503816    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b3755d8-15be-417c-8661-1381d26afd0d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7b3755d8-15be-417c-8661-1381d26afd0d" (UID: "7b3755d8-15be-417c-8661-1381d26afd0d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:57:49.505707 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:49.505681    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b3755d8-15be-417c-8661-1381d26afd0d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7b3755d8-15be-417c-8661-1381d26afd0d" (UID: "7b3755d8-15be-417c-8661-1381d26afd0d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:57:49.505997 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:49.505977    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b3755d8-15be-417c-8661-1381d26afd0d-kube-api-access-z242p" (OuterVolumeSpecName: "kube-api-access-z242p") pod "7b3755d8-15be-417c-8661-1381d26afd0d" (UID: "7b3755d8-15be-417c-8661-1381d26afd0d"). InnerVolumeSpecName "kube-api-access-z242p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:57:49.505997 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:49.505981    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b3755d8-15be-417c-8661-1381d26afd0d-dshm" (OuterVolumeSpecName: "dshm") pod "7b3755d8-15be-417c-8661-1381d26afd0d" (UID: "7b3755d8-15be-417c-8661-1381d26afd0d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:57:49.604340 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:49.604293    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3755d8-15be-417c-8661-1381d26afd0d-tls-certs\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:57:49.604340 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:49.604338    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b3755d8-15be-417c-8661-1381d26afd0d-dshm\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:57:49.604764 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:49.604353    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z242p\" (UniqueName: \"kubernetes.io/projected/7b3755d8-15be-417c-8661-1381d26afd0d-kube-api-access-z242p\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:57:49.604764 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:49.604369    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b3755d8-15be-417c-8661-1381d26afd0d-tmp-dir\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:57:49.723681 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:49.723583    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk" podUID="020714b8-832a-441b-84e0-ce38db14f968" containerName="main" containerID="cri-o://0139b9daf5a5972f1995ee58bc5094e8241fb2ee8736915d9c278239e0a17e1b" gracePeriod=45
Sep 10 20:57:49.767067 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:49.767041    2564 generic.go:356] "Generic (PLEG): container finished" podID="7b3755d8-15be-417c-8661-1381d26afd0d" containerID="f3ad9a26dad2c203cdcf635db60cb7dfd8e221f08249bac68208fe35253ed343" exitCode=0
Sep 10 20:57:49.767192 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:49.767080    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t" event={"ID":"7b3755d8-15be-417c-8661-1381d26afd0d","Type":"ContainerDied","Data":"f3ad9a26dad2c203cdcf635db60cb7dfd8e221f08249bac68208fe35253ed343"}
Sep 10 20:57:49.767192 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:49.767087    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t"
Sep 10 20:57:49.767192 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:49.767107    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t" event={"ID":"7b3755d8-15be-417c-8661-1381d26afd0d","Type":"ContainerDied","Data":"b5219688f54c4aa4990e045a4e9e7ec321ca77e264ca09ec1d50873e80bef491"}
Sep 10 20:57:49.767192 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:49.767123    2564 scope.go:117] "RemoveContainer" containerID="f3ad9a26dad2c203cdcf635db60cb7dfd8e221f08249bac68208fe35253ed343"
Sep 10 20:57:49.789888 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:49.789861    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t"]
Sep 10 20:57:49.799767 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:49.799744    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-80aa25/test-model-80aa25-kserve-7c4c7df979-fkv2t"]
Sep 10 20:57:49.834241 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:49.834214    2564 scope.go:117] "RemoveContainer" containerID="f3ad9a26dad2c203cdcf635db60cb7dfd8e221f08249bac68208fe35253ed343"
Sep 10 20:57:49.834612 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:57:49.834581    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3ad9a26dad2c203cdcf635db60cb7dfd8e221f08249bac68208fe35253ed343\": container with ID starting with f3ad9a26dad2c203cdcf635db60cb7dfd8e221f08249bac68208fe35253ed343 not found: ID does not exist" containerID="f3ad9a26dad2c203cdcf635db60cb7dfd8e221f08249bac68208fe35253ed343"
Sep 10 20:57:49.834681 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:49.834622    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3ad9a26dad2c203cdcf635db60cb7dfd8e221f08249bac68208fe35253ed343"} err="failed to get container status \"f3ad9a26dad2c203cdcf635db60cb7dfd8e221f08249bac68208fe35253ed343\": rpc error: code = NotFound desc = could not find container \"f3ad9a26dad2c203cdcf635db60cb7dfd8e221f08249bac68208fe35253ed343\": container with ID starting with f3ad9a26dad2c203cdcf635db60cb7dfd8e221f08249bac68208fe35253ed343 not found: ID does not exist"
Sep 10 20:57:49.957810 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:49.957786    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk"
Sep 10 20:57:50.008566 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:50.008503    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/020714b8-832a-441b-84e0-ce38db14f968-dshm\") pod \"020714b8-832a-441b-84e0-ce38db14f968\" (UID: \"020714b8-832a-441b-84e0-ce38db14f968\") "
Sep 10 20:57:50.008566 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:50.008534    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7cls\" (UniqueName: \"kubernetes.io/projected/020714b8-832a-441b-84e0-ce38db14f968-kube-api-access-x7cls\") pod \"020714b8-832a-441b-84e0-ce38db14f968\" (UID: \"020714b8-832a-441b-84e0-ce38db14f968\") "
Sep 10 20:57:50.008566 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:50.008558    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/020714b8-832a-441b-84e0-ce38db14f968-model-cache\") pod \"020714b8-832a-441b-84e0-ce38db14f968\" (UID: \"020714b8-832a-441b-84e0-ce38db14f968\") "
Sep 10 20:57:50.008832 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:50.008601    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/020714b8-832a-441b-84e0-ce38db14f968-home\") pod \"020714b8-832a-441b-84e0-ce38db14f968\" (UID: \"020714b8-832a-441b-84e0-ce38db14f968\") "
Sep 10 20:57:50.008832 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:50.008619    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/020714b8-832a-441b-84e0-ce38db14f968-tmp-dir\") pod \"020714b8-832a-441b-84e0-ce38db14f968\" (UID: \"020714b8-832a-441b-84e0-ce38db14f968\") "
Sep 10 20:57:50.008832 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:50.008657    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/020714b8-832a-441b-84e0-ce38db14f968-tls-certs\") pod \"020714b8-832a-441b-84e0-ce38db14f968\" (UID: \"020714b8-832a-441b-84e0-ce38db14f968\") "
Sep 10 20:57:50.009003 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:50.008909    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020714b8-832a-441b-84e0-ce38db14f968-home" (OuterVolumeSpecName: "home") pod "020714b8-832a-441b-84e0-ce38db14f968" (UID: "020714b8-832a-441b-84e0-ce38db14f968"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:57:50.009003 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:50.008917    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020714b8-832a-441b-84e0-ce38db14f968-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "020714b8-832a-441b-84e0-ce38db14f968" (UID: "020714b8-832a-441b-84e0-ce38db14f968"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:57:50.009003 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:50.008902    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020714b8-832a-441b-84e0-ce38db14f968-model-cache" (OuterVolumeSpecName: "model-cache") pod "020714b8-832a-441b-84e0-ce38db14f968" (UID: "020714b8-832a-441b-84e0-ce38db14f968"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:57:50.010757 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:50.010734    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/020714b8-832a-441b-84e0-ce38db14f968-kube-api-access-x7cls" (OuterVolumeSpecName: "kube-api-access-x7cls") pod "020714b8-832a-441b-84e0-ce38db14f968" (UID: "020714b8-832a-441b-84e0-ce38db14f968"). InnerVolumeSpecName "kube-api-access-x7cls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:57:50.010822 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:50.010768    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/020714b8-832a-441b-84e0-ce38db14f968-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "020714b8-832a-441b-84e0-ce38db14f968" (UID: "020714b8-832a-441b-84e0-ce38db14f968"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:57:50.010822 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:50.010768    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020714b8-832a-441b-84e0-ce38db14f968-dshm" (OuterVolumeSpecName: "dshm") pod "020714b8-832a-441b-84e0-ce38db14f968" (UID: "020714b8-832a-441b-84e0-ce38db14f968"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:57:50.109266 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:50.109239    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/020714b8-832a-441b-84e0-ce38db14f968-dshm\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:57:50.109266 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:50.109262    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7cls\" (UniqueName: \"kubernetes.io/projected/020714b8-832a-441b-84e0-ce38db14f968-kube-api-access-x7cls\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:57:50.109432 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:50.109273    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/020714b8-832a-441b-84e0-ce38db14f968-model-cache\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:57:50.109432 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:50.109282    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/020714b8-832a-441b-84e0-ce38db14f968-home\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:57:50.109432 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:50.109290    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/020714b8-832a-441b-84e0-ce38db14f968-tmp-dir\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:57:50.109432 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:50.109297    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/020714b8-832a-441b-84e0-ce38db14f968-tls-certs\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:57:50.541422 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:50.541390    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b3755d8-15be-417c-8661-1381d26afd0d" path="/var/lib/kubelet/pods/7b3755d8-15be-417c-8661-1381d26afd0d/volumes"
Sep 10 20:57:50.773287 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:50.773252    2564 generic.go:356] "Generic (PLEG): container finished" podID="020714b8-832a-441b-84e0-ce38db14f968" containerID="0139b9daf5a5972f1995ee58bc5094e8241fb2ee8736915d9c278239e0a17e1b" exitCode=0
Sep 10 20:57:50.773805 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:50.773294    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk"
Sep 10 20:57:50.773805 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:50.773342    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk" event={"ID":"020714b8-832a-441b-84e0-ce38db14f968","Type":"ContainerDied","Data":"0139b9daf5a5972f1995ee58bc5094e8241fb2ee8736915d9c278239e0a17e1b"}
Sep 10 20:57:50.773805 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:50.773390    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk" event={"ID":"020714b8-832a-441b-84e0-ce38db14f968","Type":"ContainerDied","Data":"26b90558caedc250a2ebc0f00bcbfaca12aab129869d9f14e3072a82fd3b09fa"}
Sep 10 20:57:50.773805 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:50.773420    2564 scope.go:117] "RemoveContainer" containerID="0139b9daf5a5972f1995ee58bc5094e8241fb2ee8736915d9c278239e0a17e1b"
Sep 10 20:57:50.782001 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:50.781985    2564 scope.go:117] "RemoveContainer" containerID="0139b9daf5a5972f1995ee58bc5094e8241fb2ee8736915d9c278239e0a17e1b"
Sep 10 20:57:50.782281 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:57:50.782262    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0139b9daf5a5972f1995ee58bc5094e8241fb2ee8736915d9c278239e0a17e1b\": container with ID starting with 0139b9daf5a5972f1995ee58bc5094e8241fb2ee8736915d9c278239e0a17e1b not found: ID does not exist" containerID="0139b9daf5a5972f1995ee58bc5094e8241fb2ee8736915d9c278239e0a17e1b"
Sep 10 20:57:50.782342 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:50.782288    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0139b9daf5a5972f1995ee58bc5094e8241fb2ee8736915d9c278239e0a17e1b"} err="failed to get container status \"0139b9daf5a5972f1995ee58bc5094e8241fb2ee8736915d9c278239e0a17e1b\": rpc error: code = NotFound desc = could not find container \"0139b9daf5a5972f1995ee58bc5094e8241fb2ee8736915d9c278239e0a17e1b\": container with ID starting with 0139b9daf5a5972f1995ee58bc5094e8241fb2ee8736915d9c278239e0a17e1b not found: ID does not exist"
Sep 10 20:57:50.796183 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:50.796112    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk"]
Sep 10 20:57:50.809795 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:50.809775    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-467fc3/test-model-467fc3-kserve-6b67c59b75-4tqrk"]
Sep 10 20:57:52.541219 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:52.541162    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="020714b8-832a-441b-84e0-ce38db14f968" path="/var/lib/kubelet/pods/020714b8-832a-441b-84e0-ce38db14f968/volumes"
Sep 10 20:57:54.279245 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:54.279210    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"]
Sep 10 20:57:54.279617 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:54.279418    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww" podUID="3ac33a9c-11a1-4a91-a458-67697f1642ba" containerName="istio-proxy" containerID="cri-o://90bea3965e202b4ae504fa4e099448d82ce00262b8df5728c1277b96b2e78ef3" gracePeriod=30
Sep 10 20:57:58.985228 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:58.985189    2564 patch_prober.go:28] interesting pod/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 20:57:58.985616 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:58.985280    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww" podUID="3ac33a9c-11a1-4a91-a458-67697f1642ba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 20:57:59.531337 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.531310    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:57:59.588618 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.588586    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac33a9c-11a1-4a91-a458-67697f1642ba-istiod-ca-cert\") pod \"3ac33a9c-11a1-4a91-a458-67697f1642ba\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") "
Sep 10 20:57:59.588618 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.588621    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac33a9c-11a1-4a91-a458-67697f1642ba-credential-socket\") pod \"3ac33a9c-11a1-4a91-a458-67697f1642ba\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") "
Sep 10 20:57:59.588865 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.588655    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac33a9c-11a1-4a91-a458-67697f1642ba-istio-data\") pod \"3ac33a9c-11a1-4a91-a458-67697f1642ba\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") "
Sep 10 20:57:59.588865 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.588676    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac33a9c-11a1-4a91-a458-67697f1642ba-workload-certs\") pod \"3ac33a9c-11a1-4a91-a458-67697f1642ba\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") "
Sep 10 20:57:59.588865 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.588716    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac33a9c-11a1-4a91-a458-67697f1642ba-istio-podinfo\") pod \"3ac33a9c-11a1-4a91-a458-67697f1642ba\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") "
Sep 10 20:57:59.589028 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.588859    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac33a9c-11a1-4a91-a458-67697f1642ba-istio-envoy\") pod \"3ac33a9c-11a1-4a91-a458-67697f1642ba\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") "
Sep 10 20:57:59.589028 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.588925    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkpxh\" (UniqueName: \"kubernetes.io/projected/3ac33a9c-11a1-4a91-a458-67697f1642ba-kube-api-access-rkpxh\") pod \"3ac33a9c-11a1-4a91-a458-67697f1642ba\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") "
Sep 10 20:57:59.589028 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.588955    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac33a9c-11a1-4a91-a458-67697f1642ba-istio-data" (OuterVolumeSpecName: "istio-data") pod "3ac33a9c-11a1-4a91-a458-67697f1642ba" (UID: "3ac33a9c-11a1-4a91-a458-67697f1642ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:57:59.589028 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.588981    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac33a9c-11a1-4a91-a458-67697f1642ba-workload-socket\") pod \"3ac33a9c-11a1-4a91-a458-67697f1642ba\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") "
Sep 10 20:57:59.589028 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.588982    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac33a9c-11a1-4a91-a458-67697f1642ba-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3ac33a9c-11a1-4a91-a458-67697f1642ba" (UID: "3ac33a9c-11a1-4a91-a458-67697f1642ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:57:59.589028 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.589004    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ac33a9c-11a1-4a91-a458-67697f1642ba-istio-token\") pod \"3ac33a9c-11a1-4a91-a458-67697f1642ba\" (UID: \"3ac33a9c-11a1-4a91-a458-67697f1642ba\") "
Sep 10 20:57:59.589362 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.589100    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ac33a9c-11a1-4a91-a458-67697f1642ba-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3ac33a9c-11a1-4a91-a458-67697f1642ba" (UID: "3ac33a9c-11a1-4a91-a458-67697f1642ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:57:59.589362 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.589106    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac33a9c-11a1-4a91-a458-67697f1642ba-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3ac33a9c-11a1-4a91-a458-67697f1642ba" (UID: "3ac33a9c-11a1-4a91-a458-67697f1642ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:57:59.589362 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.589261    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac33a9c-11a1-4a91-a458-67697f1642ba-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3ac33a9c-11a1-4a91-a458-67697f1642ba" (UID: "3ac33a9c-11a1-4a91-a458-67697f1642ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:57:59.589521 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.589386    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac33a9c-11a1-4a91-a458-67697f1642ba-istio-data\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:57:59.589521 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.589406    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac33a9c-11a1-4a91-a458-67697f1642ba-workload-certs\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:57:59.589521 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.589421    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac33a9c-11a1-4a91-a458-67697f1642ba-workload-socket\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:57:59.589521 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.589434    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac33a9c-11a1-4a91-a458-67697f1642ba-istiod-ca-cert\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:57:59.589521 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.589461    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac33a9c-11a1-4a91-a458-67697f1642ba-credential-socket\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:57:59.591084 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.591050    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ac33a9c-11a1-4a91-a458-67697f1642ba-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3ac33a9c-11a1-4a91-a458-67697f1642ba" (UID: "3ac33a9c-11a1-4a91-a458-67697f1642ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 20:57:59.591084 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.591051    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac33a9c-11a1-4a91-a458-67697f1642ba-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3ac33a9c-11a1-4a91-a458-67697f1642ba" (UID: "3ac33a9c-11a1-4a91-a458-67697f1642ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:57:59.591414 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.591387    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac33a9c-11a1-4a91-a458-67697f1642ba-kube-api-access-rkpxh" (OuterVolumeSpecName: "kube-api-access-rkpxh") pod "3ac33a9c-11a1-4a91-a458-67697f1642ba" (UID: "3ac33a9c-11a1-4a91-a458-67697f1642ba"). InnerVolumeSpecName "kube-api-access-rkpxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:57:59.591414 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.591401    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac33a9c-11a1-4a91-a458-67697f1642ba-istio-token" (OuterVolumeSpecName: "istio-token") pod "3ac33a9c-11a1-4a91-a458-67697f1642ba" (UID: "3ac33a9c-11a1-4a91-a458-67697f1642ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:57:59.690817 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.690767    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ac33a9c-11a1-4a91-a458-67697f1642ba-istio-token\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:57:59.690817 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.690811    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac33a9c-11a1-4a91-a458-67697f1642ba-istio-podinfo\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:57:59.690817 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.690821    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac33a9c-11a1-4a91-a458-67697f1642ba-istio-envoy\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:57:59.690817 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.690830    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkpxh\" (UniqueName: \"kubernetes.io/projected/3ac33a9c-11a1-4a91-a458-67697f1642ba-kube-api-access-rkpxh\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:57:59.824466 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.824428    2564 generic.go:356] "Generic (PLEG): container finished" podID="3ac33a9c-11a1-4a91-a458-67697f1642ba" containerID="90bea3965e202b4ae504fa4e099448d82ce00262b8df5728c1277b96b2e78ef3" exitCode=0
Sep 10 20:57:59.824634 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.824492    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"
Sep 10 20:57:59.824634 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.824512    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww" event={"ID":"3ac33a9c-11a1-4a91-a458-67697f1642ba","Type":"ContainerDied","Data":"90bea3965e202b4ae504fa4e099448d82ce00262b8df5728c1277b96b2e78ef3"}
Sep 10 20:57:59.824634 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.824569    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww" event={"ID":"3ac33a9c-11a1-4a91-a458-67697f1642ba","Type":"ContainerDied","Data":"7ff7aac2627b7553eb5d416fab32a8171bc4c03a6882bb28d178a744ea6a0392"}
Sep 10 20:57:59.824634 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.824592    2564 scope.go:117] "RemoveContainer" containerID="90bea3965e202b4ae504fa4e099448d82ce00262b8df5728c1277b96b2e78ef3"
Sep 10 20:57:59.834388 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.834368    2564 scope.go:117] "RemoveContainer" containerID="7428e9dde8f934a68076e08d5a74e369edf0a9d54f4f38ba9f6d196fdac8d613"
Sep 10 20:57:59.843083 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.843066    2564 scope.go:117] "RemoveContainer" containerID="90bea3965e202b4ae504fa4e099448d82ce00262b8df5728c1277b96b2e78ef3"
Sep 10 20:57:59.843388 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:57:59.843369    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90bea3965e202b4ae504fa4e099448d82ce00262b8df5728c1277b96b2e78ef3\": container with ID starting with 90bea3965e202b4ae504fa4e099448d82ce00262b8df5728c1277b96b2e78ef3 not found: ID does not exist" containerID="90bea3965e202b4ae504fa4e099448d82ce00262b8df5728c1277b96b2e78ef3"
Sep 10 20:57:59.843432 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.843397    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90bea3965e202b4ae504fa4e099448d82ce00262b8df5728c1277b96b2e78ef3"} err="failed to get container status \"90bea3965e202b4ae504fa4e099448d82ce00262b8df5728c1277b96b2e78ef3\": rpc error: code = NotFound desc = could not find container \"90bea3965e202b4ae504fa4e099448d82ce00262b8df5728c1277b96b2e78ef3\": container with ID starting with 90bea3965e202b4ae504fa4e099448d82ce00262b8df5728c1277b96b2e78ef3 not found: ID does not exist"
Sep 10 20:57:59.843432 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.843417    2564 scope.go:117] "RemoveContainer" containerID="7428e9dde8f934a68076e08d5a74e369edf0a9d54f4f38ba9f6d196fdac8d613"
Sep 10 20:57:59.843691 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:57:59.843674    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7428e9dde8f934a68076e08d5a74e369edf0a9d54f4f38ba9f6d196fdac8d613\": container with ID starting with 7428e9dde8f934a68076e08d5a74e369edf0a9d54f4f38ba9f6d196fdac8d613 not found: ID does not exist" containerID="7428e9dde8f934a68076e08d5a74e369edf0a9d54f4f38ba9f6d196fdac8d613"
Sep 10 20:57:59.843731 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.843698    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7428e9dde8f934a68076e08d5a74e369edf0a9d54f4f38ba9f6d196fdac8d613"} err="failed to get container status \"7428e9dde8f934a68076e08d5a74e369edf0a9d54f4f38ba9f6d196fdac8d613\": rpc error: code = NotFound desc = could not find container \"7428e9dde8f934a68076e08d5a74e369edf0a9d54f4f38ba9f6d196fdac8d613\": container with ID starting with 7428e9dde8f934a68076e08d5a74e369edf0a9d54f4f38ba9f6d196fdac8d613 not found: ID does not exist"
Sep 10 20:57:59.857696 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.857665    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"]
Sep 10 20:57:59.864730 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:57:59.864704    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-80aa25-openshift-default-6f497c79b-bd4ww"]
Sep 10 20:58:00.540289 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:00.540257    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ac33a9c-11a1-4a91-a458-67697f1642ba" path="/var/lib/kubelet/pods/3ac33a9c-11a1-4a91-a458-67697f1642ba/volumes"
Sep 10 20:58:01.361337 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.361288    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cd89ac-5497d785db-gksj6"]
Sep 10 20:58:01.361986 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.361963    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ca34531-dc05-421e-adb6-f6d62a8ecabe" containerName="revoke-keys"
Sep 10 20:58:01.362103 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.361989    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ca34531-dc05-421e-adb6-f6d62a8ecabe" containerName="revoke-keys"
Sep 10 20:58:01.362103 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.362003    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ac33a9c-11a1-4a91-a458-67697f1642ba" containerName="istio-proxy"
Sep 10 20:58:01.362103 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.362012    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ac33a9c-11a1-4a91-a458-67697f1642ba" containerName="istio-proxy"
Sep 10 20:58:01.362103 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.362034    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b3755d8-15be-417c-8661-1381d26afd0d" containerName="main"
Sep 10 20:58:01.362103 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.362043    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b3755d8-15be-417c-8661-1381d26afd0d" containerName="main"
Sep 10 20:58:01.362103 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.362061    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="020714b8-832a-441b-84e0-ce38db14f968" containerName="main"
Sep 10 20:58:01.362103 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.362086    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="020714b8-832a-441b-84e0-ce38db14f968" containerName="main"
Sep 10 20:58:01.362103 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.362100    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0e77222-8445-4c91-8999-8686eb8dabb4" containerName="payload-pre-processing"
Sep 10 20:58:01.362495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.362109    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0e77222-8445-4c91-8999-8686eb8dabb4" containerName="payload-pre-processing"
Sep 10 20:58:01.362495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.362132    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ac33a9c-11a1-4a91-a458-67697f1642ba" containerName="istio-proxy"
Sep 10 20:58:01.362495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.362140    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ac33a9c-11a1-4a91-a458-67697f1642ba" containerName="istio-proxy"
Sep 10 20:58:01.362495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.362181    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78291f5f-f180-4c2f-b36e-24181627b9e8" containerName="payload-processing"
Sep 10 20:58:01.362495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.362190    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="78291f5f-f180-4c2f-b36e-24181627b9e8" containerName="payload-processing"
Sep 10 20:58:01.362495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.362325    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ca34531-dc05-421e-adb6-f6d62a8ecabe" containerName="revoke-keys"
Sep 10 20:58:01.362495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.362345    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="020714b8-832a-441b-84e0-ce38db14f968" containerName="main"
Sep 10 20:58:01.362495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.362362    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ac33a9c-11a1-4a91-a458-67697f1642ba" containerName="istio-proxy"
Sep 10 20:58:01.362495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.362371    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="78291f5f-f180-4c2f-b36e-24181627b9e8" containerName="payload-processing"
Sep 10 20:58:01.362495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.362386    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0e77222-8445-4c91-8999-8686eb8dabb4" containerName="payload-pre-processing"
Sep 10 20:58:01.362495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.362396    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b3755d8-15be-417c-8661-1381d26afd0d" containerName="main"
Sep 10 20:58:01.362495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.362408    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ac33a9c-11a1-4a91-a458-67697f1642ba" containerName="istio-proxy"
Sep 10 20:58:01.367846 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.367824    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cd89ac-5497d785db-gksj6"
Sep 10 20:58:01.370684 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.370662    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-cd89ac-dockercfg-7h4g9\""
Sep 10 20:58:01.370801 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.370686    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-cd89ac\""
Sep 10 20:58:01.377595 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.377573    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cd89ac-5497d785db-gksj6"]
Sep 10 20:58:01.407334 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.407303    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b832e901-0ff9-457e-ae81-a35d8ab799b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-cd89ac-5497d785db-gksj6\" (UID: \"b832e901-0ff9-457e-ae81-a35d8ab799b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cd89ac-5497d785db-gksj6"
Sep 10 20:58:01.407439 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.407344    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfcxn\" (UniqueName: \"kubernetes.io/projected/b832e901-0ff9-457e-ae81-a35d8ab799b6-kube-api-access-jfcxn\") pod \"payload-pre-processing-e2e-shared-a-w2-cd89ac-5497d785db-gksj6\" (UID: \"b832e901-0ff9-457e-ae81-a35d8ab799b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cd89ac-5497d785db-gksj6"
Sep 10 20:58:01.450366 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.450339    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cd89ac-cf8d5c4d9-lhgft"]
Sep 10 20:58:01.453554 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.453539    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cd89ac-cf8d5c4d9-lhgft"
Sep 10 20:58:01.469154 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.469130    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cd89ac-cf8d5c4d9-lhgft"]
Sep 10 20:58:01.508484 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.508455    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfcxn\" (UniqueName: \"kubernetes.io/projected/b832e901-0ff9-457e-ae81-a35d8ab799b6-kube-api-access-jfcxn\") pod \"payload-pre-processing-e2e-shared-a-w2-cd89ac-5497d785db-gksj6\" (UID: \"b832e901-0ff9-457e-ae81-a35d8ab799b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cd89ac-5497d785db-gksj6"
Sep 10 20:58:01.508604 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.508498    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6zrs\" (UniqueName: \"kubernetes.io/projected/df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e-kube-api-access-q6zrs\") pod \"payload-processing-e2e-shared-a-w2-cd89ac-cf8d5c4d9-lhgft\" (UID: \"df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cd89ac-cf8d5c4d9-lhgft"
Sep 10 20:58:01.508604 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.508537    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-cd89ac-cf8d5c4d9-lhgft\" (UID: \"df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cd89ac-cf8d5c4d9-lhgft"
Sep 10 20:58:01.508604 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.508568    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b832e901-0ff9-457e-ae81-a35d8ab799b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-cd89ac-5497d785db-gksj6\" (UID: \"b832e901-0ff9-457e-ae81-a35d8ab799b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cd89ac-5497d785db-gksj6"
Sep 10 20:58:01.509258 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.509237    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b832e901-0ff9-457e-ae81-a35d8ab799b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-cd89ac-5497d785db-gksj6\" (UID: \"b832e901-0ff9-457e-ae81-a35d8ab799b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cd89ac-5497d785db-gksj6"
Sep 10 20:58:01.524085 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.524065    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfcxn\" (UniqueName: \"kubernetes.io/projected/b832e901-0ff9-457e-ae81-a35d8ab799b6-kube-api-access-jfcxn\") pod \"payload-pre-processing-e2e-shared-a-w2-cd89ac-5497d785db-gksj6\" (UID: \"b832e901-0ff9-457e-ae81-a35d8ab799b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cd89ac-5497d785db-gksj6"
Sep 10 20:58:01.609372 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.609345    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6zrs\" (UniqueName: \"kubernetes.io/projected/df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e-kube-api-access-q6zrs\") pod \"payload-processing-e2e-shared-a-w2-cd89ac-cf8d5c4d9-lhgft\" (UID: \"df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cd89ac-cf8d5c4d9-lhgft"
Sep 10 20:58:01.609669 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.609394    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-cd89ac-cf8d5c4d9-lhgft\" (UID: \"df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cd89ac-cf8d5c4d9-lhgft"
Sep 10 20:58:01.610044 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.610027    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-cd89ac-cf8d5c4d9-lhgft\" (UID: \"df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cd89ac-cf8d5c4d9-lhgft"
Sep 10 20:58:01.618549 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.618495    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6zrs\" (UniqueName: \"kubernetes.io/projected/df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e-kube-api-access-q6zrs\") pod \"payload-processing-e2e-shared-a-w2-cd89ac-cf8d5c4d9-lhgft\" (UID: \"df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cd89ac-cf8d5c4d9-lhgft"
Sep 10 20:58:01.678486 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.678459    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cd89ac-5497d785db-gksj6"
Sep 10 20:58:01.763500 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.763466    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cd89ac-cf8d5c4d9-lhgft"
Sep 10 20:58:01.972873 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:01.972846    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cd89ac-5497d785db-gksj6"]
Sep 10 20:58:01.973270 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:58:01.973246    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb832e901_0ff9_457e_ae81_a35d8ab799b6.slice/crio-1f467621fb7403075a04bf9add47440dc41901c6661a651bd11051832e227a44 WatchSource:0}: Error finding container 1f467621fb7403075a04bf9add47440dc41901c6661a651bd11051832e227a44: Status 404 returned error can't find the container with id 1f467621fb7403075a04bf9add47440dc41901c6661a651bd11051832e227a44
Sep 10 20:58:02.050279 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:02.050187    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cd89ac-cf8d5c4d9-lhgft"]
Sep 10 20:58:02.053735 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:58:02.053711    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf51c2f2_6b1b_4eb9_85ff_7b972ac06f8e.slice/crio-b642f3e4618f6f1d566987743fd9bc49dd8c3d62300de05e11aff9e9f54a2454 WatchSource:0}: Error finding container b642f3e4618f6f1d566987743fd9bc49dd8c3d62300de05e11aff9e9f54a2454: Status 404 returned error can't find the container with id b642f3e4618f6f1d566987743fd9bc49dd8c3d62300de05e11aff9e9f54a2454
Sep 10 20:58:02.840859 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:02.840823    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cd89ac-5497d785db-gksj6" event={"ID":"b832e901-0ff9-457e-ae81-a35d8ab799b6","Type":"ContainerStarted","Data":"12a396ae77df424330deac0827940f32e85c71a479f937cfda23e0d9e8f33527"}
Sep 10 20:58:02.841260 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:02.840882    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cd89ac-5497d785db-gksj6" event={"ID":"b832e901-0ff9-457e-ae81-a35d8ab799b6","Type":"ContainerStarted","Data":"1f467621fb7403075a04bf9add47440dc41901c6661a651bd11051832e227a44"}
Sep 10 20:58:02.841260 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:02.840911    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cd89ac-5497d785db-gksj6"
Sep 10 20:58:02.842339 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:02.842313    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cd89ac-cf8d5c4d9-lhgft" event={"ID":"df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e","Type":"ContainerStarted","Data":"56867c50c50b90d50b1752e4bd8169d2f57a6f67229bbd3f7239ba27ef335e24"}
Sep 10 20:58:02.842429 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:02.842360    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cd89ac-cf8d5c4d9-lhgft" event={"ID":"df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e","Type":"ContainerStarted","Data":"b642f3e4618f6f1d566987743fd9bc49dd8c3d62300de05e11aff9e9f54a2454"}
Sep 10 20:58:02.842429 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:02.842380    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cd89ac-cf8d5c4d9-lhgft"
Sep 10 20:58:02.861826 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:02.861785    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cd89ac-5497d785db-gksj6" podStartSLOduration=1.86176673 podStartE2EDuration="1.86176673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:02.860306358 +0000 UTC m=+1322.825374535" watchObservedRunningTime="2026-09-10 20:58:02.86176673 +0000 UTC m=+1322.826834927"
Sep 10 20:58:02.883239 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:02.883196    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cd89ac-cf8d5c4d9-lhgft" podStartSLOduration=1.883179516 podStartE2EDuration="1.883179516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:02.880519924 +0000 UTC m=+1322.845588115" watchObservedRunningTime="2026-09-10 20:58:02.883179516 +0000 UTC m=+1322.848247704"
Sep 10 20:58:05.932551 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:05.932510    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-467fc3-7ddc48679c-dg28p"]
Sep 10 20:58:05.933111 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:05.932786    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-467fc3-7ddc48679c-dg28p" podUID="6af4ce55-b9fc-4633-a9e3-5b85dd8ad789" containerName="payload-processing" containerID="cri-o://34da3b5b8491cb7f890fb17f3d70c8c18dc7325cd331e79d8816dfcbbb2a1c71" gracePeriod=30
Sep 10 20:58:06.187601 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:06.187525    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-467fc3-7ddc48679c-dg28p"
Sep 10 20:58:06.244491 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:06.244462    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnqcr\" (UniqueName: \"kubernetes.io/projected/6af4ce55-b9fc-4633-a9e3-5b85dd8ad789-kube-api-access-cnqcr\") pod \"6af4ce55-b9fc-4633-a9e3-5b85dd8ad789\" (UID: \"6af4ce55-b9fc-4633-a9e3-5b85dd8ad789\") "
Sep 10 20:58:06.244632 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:06.244595    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6af4ce55-b9fc-4633-a9e3-5b85dd8ad789-plugins-config-volume\") pod \"6af4ce55-b9fc-4633-a9e3-5b85dd8ad789\" (UID: \"6af4ce55-b9fc-4633-a9e3-5b85dd8ad789\") "
Sep 10 20:58:06.244957 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:06.244931    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6af4ce55-b9fc-4633-a9e3-5b85dd8ad789-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6af4ce55-b9fc-4633-a9e3-5b85dd8ad789" (UID: "6af4ce55-b9fc-4633-a9e3-5b85dd8ad789"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:58:06.246509 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:06.246490    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6af4ce55-b9fc-4633-a9e3-5b85dd8ad789-kube-api-access-cnqcr" (OuterVolumeSpecName: "kube-api-access-cnqcr") pod "6af4ce55-b9fc-4633-a9e3-5b85dd8ad789" (UID: "6af4ce55-b9fc-4633-a9e3-5b85dd8ad789"). InnerVolumeSpecName "kube-api-access-cnqcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:58:06.345701 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:06.345667    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6af4ce55-b9fc-4633-a9e3-5b85dd8ad789-plugins-config-volume\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:58:06.345701 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:06.345696    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnqcr\" (UniqueName: \"kubernetes.io/projected/6af4ce55-b9fc-4633-a9e3-5b85dd8ad789-kube-api-access-cnqcr\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 20:58:06.867453 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:06.867416    2564 generic.go:356] "Generic (PLEG): container finished" podID="6af4ce55-b9fc-4633-a9e3-5b85dd8ad789" containerID="34da3b5b8491cb7f890fb17f3d70c8c18dc7325cd331e79d8816dfcbbb2a1c71" exitCode=0
Sep 10 20:58:06.867590 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:06.867460    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-467fc3-7ddc48679c-dg28p"
Sep 10 20:58:06.867590 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:06.867459    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-467fc3-7ddc48679c-dg28p" event={"ID":"6af4ce55-b9fc-4633-a9e3-5b85dd8ad789","Type":"ContainerDied","Data":"34da3b5b8491cb7f890fb17f3d70c8c18dc7325cd331e79d8816dfcbbb2a1c71"}
Sep 10 20:58:06.867679 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:06.867597    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-467fc3-7ddc48679c-dg28p" event={"ID":"6af4ce55-b9fc-4633-a9e3-5b85dd8ad789","Type":"ContainerDied","Data":"c08a0791dfb12561f6fab7033029231004d530e8fdd66e1fc8cd53f9cdcd4e02"}
Sep 10 20:58:06.867679 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:06.867631    2564 scope.go:117] "RemoveContainer" containerID="34da3b5b8491cb7f890fb17f3d70c8c18dc7325cd331e79d8816dfcbbb2a1c71"
Sep 10 20:58:06.876482 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:06.876461    2564 scope.go:117] "RemoveContainer" containerID="34da3b5b8491cb7f890fb17f3d70c8c18dc7325cd331e79d8816dfcbbb2a1c71"
Sep 10 20:58:06.876768 ip-10-0-141-185 kubenswrapper[2564]: E0910 20:58:06.876746    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34da3b5b8491cb7f890fb17f3d70c8c18dc7325cd331e79d8816dfcbbb2a1c71\": container with ID starting with 34da3b5b8491cb7f890fb17f3d70c8c18dc7325cd331e79d8816dfcbbb2a1c71 not found: ID does not exist" containerID="34da3b5b8491cb7f890fb17f3d70c8c18dc7325cd331e79d8816dfcbbb2a1c71"
Sep 10 20:58:06.876820 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:06.876777    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34da3b5b8491cb7f890fb17f3d70c8c18dc7325cd331e79d8816dfcbbb2a1c71"} err="failed to get container status \"34da3b5b8491cb7f890fb17f3d70c8c18dc7325cd331e79d8816dfcbbb2a1c71\": rpc error: code = NotFound desc = could not find container \"34da3b5b8491cb7f890fb17f3d70c8c18dc7325cd331e79d8816dfcbbb2a1c71\": container with ID starting with 34da3b5b8491cb7f890fb17f3d70c8c18dc7325cd331e79d8816dfcbbb2a1c71 not found: ID does not exist"
Sep 10 20:58:06.887160 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:06.887133    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-467fc3-7ddc48679c-dg28p"]
Sep 10 20:58:06.890061 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:06.890039    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-467fc3-7ddc48679c-dg28p"]
Sep 10 20:58:08.541707 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:08.541668    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6af4ce55-b9fc-4633-a9e3-5b85dd8ad789" path="/var/lib/kubelet/pods/6af4ce55-b9fc-4633-a9e3-5b85dd8ad789/volumes"
Sep 10 20:58:13.850333 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:13.850264    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cd89ac-cf8d5c4d9-lhgft"
Sep 10 20:58:13.850750 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:13.850346    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cd89ac-5497d785db-gksj6"
Sep 10 20:58:24.869927 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:24.869844    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"]
Sep 10 20:58:24.870377 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:24.870347    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6af4ce55-b9fc-4633-a9e3-5b85dd8ad789" containerName="payload-processing"
Sep 10 20:58:24.870377 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:24.870367    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6af4ce55-b9fc-4633-a9e3-5b85dd8ad789" containerName="payload-processing"
Sep 10 20:58:24.870493 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:24.870477    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6af4ce55-b9fc-4633-a9e3-5b85dd8ad789" containerName="payload-processing"
Sep 10 20:58:24.874764 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:24.874739    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:24.877277 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:24.877255    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-3bbb40-openshift-default-dockercfg-555mx\""
Sep 10 20:58:24.905892 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:24.905870    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"]
Sep 10 20:58:24.995849 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:24.995812    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c328c839-e286-49b2-ad8b-3a472c32d624-workload-certs\") pod \"e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") " pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:24.995849 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:24.995852    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c328c839-e286-49b2-ad8b-3a472c32d624-credential-socket\") pod \"e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") " pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:24.996019 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:24.995910    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzpc6\" (UniqueName: \"kubernetes.io/projected/c328c839-e286-49b2-ad8b-3a472c32d624-kube-api-access-bzpc6\") pod \"e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") " pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:24.996019 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:24.995942    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c328c839-e286-49b2-ad8b-3a472c32d624-workload-socket\") pod \"e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") " pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:24.996019 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:24.995997    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c328c839-e286-49b2-ad8b-3a472c32d624-istio-podinfo\") pod \"e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") " pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:24.996118 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:24.996040    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c328c839-e286-49b2-ad8b-3a472c32d624-istio-token\") pod \"e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") " pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:24.996118 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:24.996066    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c328c839-e286-49b2-ad8b-3a472c32d624-istio-envoy\") pod \"e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") " pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:24.996118 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:24.996096    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c328c839-e286-49b2-ad8b-3a472c32d624-istiod-ca-cert\") pod \"e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") " pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:24.996246 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:24.996119    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c328c839-e286-49b2-ad8b-3a472c32d624-istio-data\") pod \"e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") " pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:25.096845 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:25.096817    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c328c839-e286-49b2-ad8b-3a472c32d624-istio-podinfo\") pod \"e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") " pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:25.096975 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:25.096869    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c328c839-e286-49b2-ad8b-3a472c32d624-istio-token\") pod \"e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") " pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:25.096975 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:25.096896    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c328c839-e286-49b2-ad8b-3a472c32d624-istio-envoy\") pod \"e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") " pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:25.096975 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:25.096915    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c328c839-e286-49b2-ad8b-3a472c32d624-istiod-ca-cert\") pod \"e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") " pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:25.096975 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:25.096934    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c328c839-e286-49b2-ad8b-3a472c32d624-istio-data\") pod \"e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") " pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:25.096975 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:25.096968    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c328c839-e286-49b2-ad8b-3a472c32d624-workload-certs\") pod \"e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") " pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:25.097258 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:25.097048    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c328c839-e286-49b2-ad8b-3a472c32d624-credential-socket\") pod \"e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") " pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:25.097258 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:25.097114    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzpc6\" (UniqueName: \"kubernetes.io/projected/c328c839-e286-49b2-ad8b-3a472c32d624-kube-api-access-bzpc6\") pod \"e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") " pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:25.097258 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:25.097145    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c328c839-e286-49b2-ad8b-3a472c32d624-workload-socket\") pod \"e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") " pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:25.097493 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:25.097470    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c328c839-e286-49b2-ad8b-3a472c32d624-workload-certs\") pod \"e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") " pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:25.097578 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:25.097542    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c328c839-e286-49b2-ad8b-3a472c32d624-istio-data\") pod \"e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") " pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:25.097578 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:25.097570    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c328c839-e286-49b2-ad8b-3a472c32d624-workload-socket\") pod \"e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") " pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:25.097803 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:25.097776    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c328c839-e286-49b2-ad8b-3a472c32d624-istiod-ca-cert\") pod \"e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") " pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:25.097844 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:25.097810    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c328c839-e286-49b2-ad8b-3a472c32d624-credential-socket\") pod \"e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") " pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:25.099284 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:25.099263    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c328c839-e286-49b2-ad8b-3a472c32d624-istio-envoy\") pod \"e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") " pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:25.099442 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:25.099425    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c328c839-e286-49b2-ad8b-3a472c32d624-istio-podinfo\") pod \"e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") " pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:25.107932 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:25.107905    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c328c839-e286-49b2-ad8b-3a472c32d624-istio-token\") pod \"e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") " pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:25.108084 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:25.108061    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzpc6\" (UniqueName: \"kubernetes.io/projected/c328c839-e286-49b2-ad8b-3a472c32d624-kube-api-access-bzpc6\") pod \"e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") " pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:25.185694 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:25.185630    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:25.472770 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:25.472746    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"]
Sep 10 20:58:25.474573 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:58:25.474546    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc328c839_e286_49b2_ad8b_3a472c32d624.slice/crio-df294fd3f451eff07008f612825cc38a074d7c99ce25e495bc68dfac0c0921e2 WatchSource:0}: Error finding container df294fd3f451eff07008f612825cc38a074d7c99ce25e495bc68dfac0c0921e2: Status 404 returned error can't find the container with id df294fd3f451eff07008f612825cc38a074d7c99ce25e495bc68dfac0c0921e2
Sep 10 20:58:25.476690 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:25.476657    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 20:58:25.476765 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:25.476724    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 20:58:25.476765 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:25.476751    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 20:58:25.972782 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:25.972745    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn" event={"ID":"c328c839-e286-49b2-ad8b-3a472c32d624","Type":"ContainerStarted","Data":"b6cb6654ae6bceed64e2024f25280c7d0798a208561ad3a62b89726823d00d1e"}
Sep 10 20:58:25.973194 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:25.972804    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn" event={"ID":"c328c839-e286-49b2-ad8b-3a472c32d624","Type":"ContainerStarted","Data":"df294fd3f451eff07008f612825cc38a074d7c99ce25e495bc68dfac0c0921e2"}
Sep 10 20:58:25.997093 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:25.997043    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn" podStartSLOduration=1.997028583 podStartE2EDuration="1.997028583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:25.994483697 +0000 UTC m=+1345.959551890" watchObservedRunningTime="2026-09-10 20:58:25.997028583 +0000 UTC m=+1345.962096849"
Sep 10 20:58:26.186691 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:26.186661    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:26.191877 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:26.191853    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:26.977870 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:26.977834    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:26.978676 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:26.978653    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:30.709662 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:30.709620    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bbb40-848bbfbcb6-k4cgb"]
Sep 10 20:58:30.712953 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:30.712932    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bbb40-848bbfbcb6-k4cgb"
Sep 10 20:58:30.717086 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:30.717010    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-3bbb40\""
Sep 10 20:58:30.717695 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:30.717675    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-3bbb40-dockercfg-dztbh\""
Sep 10 20:58:30.733074 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:30.733048    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bbb40-848bbfbcb6-k4cgb"]
Sep 10 20:58:30.844646 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:30.844613    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvw9h\" (UniqueName: \"kubernetes.io/projected/7dae9e7e-4e67-415f-a209-47f9fd57281c-kube-api-access-hvw9h\") pod \"payload-pre-processing-e2e-shared-b-w2-3bbb40-848bbfbcb6-k4cgb\" (UID: \"7dae9e7e-4e67-415f-a209-47f9fd57281c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bbb40-848bbfbcb6-k4cgb"
Sep 10 20:58:30.844784 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:30.844659    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dae9e7e-4e67-415f-a209-47f9fd57281c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-3bbb40-848bbfbcb6-k4cgb\" (UID: \"7dae9e7e-4e67-415f-a209-47f9fd57281c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bbb40-848bbfbcb6-k4cgb"
Sep 10 20:58:30.945106 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:30.945073    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvw9h\" (UniqueName: \"kubernetes.io/projected/7dae9e7e-4e67-415f-a209-47f9fd57281c-kube-api-access-hvw9h\") pod \"payload-pre-processing-e2e-shared-b-w2-3bbb40-848bbfbcb6-k4cgb\" (UID: \"7dae9e7e-4e67-415f-a209-47f9fd57281c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bbb40-848bbfbcb6-k4cgb"
Sep 10 20:58:30.945266 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:30.945121    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dae9e7e-4e67-415f-a209-47f9fd57281c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-3bbb40-848bbfbcb6-k4cgb\" (UID: \"7dae9e7e-4e67-415f-a209-47f9fd57281c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bbb40-848bbfbcb6-k4cgb"
Sep 10 20:58:30.945760 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:30.945741    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dae9e7e-4e67-415f-a209-47f9fd57281c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-3bbb40-848bbfbcb6-k4cgb\" (UID: \"7dae9e7e-4e67-415f-a209-47f9fd57281c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bbb40-848bbfbcb6-k4cgb"
Sep 10 20:58:30.957883 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:30.957853    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvw9h\" (UniqueName: \"kubernetes.io/projected/7dae9e7e-4e67-415f-a209-47f9fd57281c-kube-api-access-hvw9h\") pod \"payload-pre-processing-e2e-shared-b-w2-3bbb40-848bbfbcb6-k4cgb\" (UID: \"7dae9e7e-4e67-415f-a209-47f9fd57281c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bbb40-848bbfbcb6-k4cgb"
Sep 10 20:58:31.024968 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:31.024911    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bbb40-848bbfbcb6-k4cgb"
Sep 10 20:58:31.324615 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:31.324591    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bbb40-848bbfbcb6-k4cgb"]
Sep 10 20:58:31.327061 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:58:31.327034    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dae9e7e_4e67_415f_a209_47f9fd57281c.slice/crio-8617bc01f0ccb8bf9b1ed57146aa3270d2b8d93581c038d273d22af34f3a49c7 WatchSource:0}: Error finding container 8617bc01f0ccb8bf9b1ed57146aa3270d2b8d93581c038d273d22af34f3a49c7: Status 404 returned error can't find the container with id 8617bc01f0ccb8bf9b1ed57146aa3270d2b8d93581c038d273d22af34f3a49c7
Sep 10 20:58:32.004905 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:32.004869    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bbb40-848bbfbcb6-k4cgb" event={"ID":"7dae9e7e-4e67-415f-a209-47f9fd57281c","Type":"ContainerStarted","Data":"9dba583fd0ab3965adb109d9d4926bcabd77d032de476957f433200c29e2b33b"}
Sep 10 20:58:32.005279 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:32.004922    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bbb40-848bbfbcb6-k4cgb" event={"ID":"7dae9e7e-4e67-415f-a209-47f9fd57281c","Type":"ContainerStarted","Data":"8617bc01f0ccb8bf9b1ed57146aa3270d2b8d93581c038d273d22af34f3a49c7"}
Sep 10 20:58:32.005279 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:32.004977    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bbb40-848bbfbcb6-k4cgb"
Sep 10 20:58:32.027071 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:32.027029    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bbb40-848bbfbcb6-k4cgb" podStartSLOduration=2.027011054 podStartE2EDuration="2.027011054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:32.026088924 +0000 UTC m=+1351.991157132" watchObservedRunningTime="2026-09-10 20:58:32.027011054 +0000 UTC m=+1351.992079251"
Sep 10 20:58:42.979399 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:42.979356    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 20:58:42.979897 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:42.979445    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn" podUID="c328c839-e286-49b2-ad8b-3a472c32d624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 20:58:43.012608 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:43.012572    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bbb40-848bbfbcb6-k4cgb"
Sep 10 20:58:43.966320 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:43.966288    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"]
Sep 10 20:58:43.969749 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:43.969728    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:58:43.972561 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:43.972540    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-f4fe4f-openshift-default-dockercfg-p2bn9\""
Sep 10 20:58:43.983377 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:43.983356    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"]
Sep 10 20:58:44.053042 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:44.053010    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf04e82d-33fb-488d-950a-00e92d693cbc-istio-podinfo\") pod \"e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") " pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:58:44.053207 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:44.053077    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf04e82d-33fb-488d-950a-00e92d693cbc-workload-socket\") pod \"e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") " pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:58:44.053207 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:44.053199    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf04e82d-33fb-488d-950a-00e92d693cbc-istiod-ca-cert\") pod \"e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") " pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:58:44.053332 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:44.053241    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf04e82d-33fb-488d-950a-00e92d693cbc-istio-data\") pod \"e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") " pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:58:44.053332 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:44.053268    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf04e82d-33fb-488d-950a-00e92d693cbc-istio-token\") pod \"e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") " pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:58:44.053446 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:44.053329    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf04e82d-33fb-488d-950a-00e92d693cbc-credential-socket\") pod \"e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") " pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:58:44.053446 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:44.053376    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf04e82d-33fb-488d-950a-00e92d693cbc-istio-envoy\") pod \"e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") " pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:58:44.053446 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:44.053405    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5cpn\" (UniqueName: \"kubernetes.io/projected/bf04e82d-33fb-488d-950a-00e92d693cbc-kube-api-access-j5cpn\") pod \"e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") " pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:58:44.053603 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:44.053446    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf04e82d-33fb-488d-950a-00e92d693cbc-workload-certs\") pod \"e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") " pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:58:44.153913 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:44.153877    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf04e82d-33fb-488d-950a-00e92d693cbc-istio-podinfo\") pod \"e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") " pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:58:44.154072 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:44.153920    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf04e82d-33fb-488d-950a-00e92d693cbc-workload-socket\") pod \"e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") " pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:58:44.154149 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:44.154127    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf04e82d-33fb-488d-950a-00e92d693cbc-istiod-ca-cert\") pod \"e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") " pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:58:44.154232 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:44.154193    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf04e82d-33fb-488d-950a-00e92d693cbc-istio-data\") pod \"e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") " pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:58:44.154232 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:44.154222    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf04e82d-33fb-488d-950a-00e92d693cbc-istio-token\") pod \"e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") " pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:58:44.154348 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:44.154251    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf04e82d-33fb-488d-950a-00e92d693cbc-credential-socket\") pod \"e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") " pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:58:44.154348 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:44.154328    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf04e82d-33fb-488d-950a-00e92d693cbc-workload-socket\") pod \"e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") " pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:58:44.154545 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:44.154524    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf04e82d-33fb-488d-950a-00e92d693cbc-istio-data\") pod \"e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") " pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:58:44.154728 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:44.154707    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf04e82d-33fb-488d-950a-00e92d693cbc-istio-envoy\") pod \"e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") " pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:58:44.154792 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:44.154746    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5cpn\" (UniqueName: \"kubernetes.io/projected/bf04e82d-33fb-488d-950a-00e92d693cbc-kube-api-access-j5cpn\") pod \"e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") " pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:58:44.154849 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:44.154796    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf04e82d-33fb-488d-950a-00e92d693cbc-workload-certs\") pod \"e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") " pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:58:44.154904 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:44.154875    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf04e82d-33fb-488d-950a-00e92d693cbc-credential-socket\") pod \"e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") " pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:58:44.154995 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:44.154969    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf04e82d-33fb-488d-950a-00e92d693cbc-istiod-ca-cert\") pod \"e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") " pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:58:44.155130 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:44.155111    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf04e82d-33fb-488d-950a-00e92d693cbc-workload-certs\") pod \"e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") " pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:58:44.156745 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:44.156725    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf04e82d-33fb-488d-950a-00e92d693cbc-istio-envoy\") pod \"e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") " pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:58:44.156873 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:44.156849    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf04e82d-33fb-488d-950a-00e92d693cbc-istio-podinfo\") pod \"e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") " pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:58:44.164451 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:44.164369    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf04e82d-33fb-488d-950a-00e92d693cbc-istio-token\") pod \"e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") " pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:58:44.164584 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:44.164561    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5cpn\" (UniqueName: \"kubernetes.io/projected/bf04e82d-33fb-488d-950a-00e92d693cbc-kube-api-access-j5cpn\") pod \"e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") " pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:58:44.279799 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:44.279732    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:58:44.566508 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:58:44.566477    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf04e82d_33fb_488d_950a_00e92d693cbc.slice/crio-373553354f58ef9b39cfb0fcbfe883eefc8d10ee18a1e8519ab43e9c59e29d26 WatchSource:0}: Error finding container 373553354f58ef9b39cfb0fcbfe883eefc8d10ee18a1e8519ab43e9c59e29d26: Status 404 returned error can't find the container with id 373553354f58ef9b39cfb0fcbfe883eefc8d10ee18a1e8519ab43e9c59e29d26
Sep 10 20:58:44.566631 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:44.566546    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"]
Sep 10 20:58:44.569061 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:44.569021    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 20:58:44.569217 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:44.569107    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 20:58:44.569217 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:44.569155    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 20:58:45.076492 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:45.076464    2564 generic.go:356] "Generic (PLEG): container finished" podID="c328c839-e286-49b2-ad8b-3a472c32d624" containerID="b6cb6654ae6bceed64e2024f25280c7d0798a208561ad3a62b89726823d00d1e" exitCode=0
Sep 10 20:58:45.076977 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:45.076535    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn" event={"ID":"c328c839-e286-49b2-ad8b-3a472c32d624","Type":"ContainerDied","Data":"b6cb6654ae6bceed64e2024f25280c7d0798a208561ad3a62b89726823d00d1e"}
Sep 10 20:58:45.076977 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:45.076945    2564 scope.go:117] "RemoveContainer" containerID="b6cb6654ae6bceed64e2024f25280c7d0798a208561ad3a62b89726823d00d1e"
Sep 10 20:58:45.077989 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:45.077957    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 20:58:45.078103 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:45.078045    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 20:58:45.078103 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:45.078092    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 20:58:45.078432 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:45.078411    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k" event={"ID":"bf04e82d-33fb-488d-950a-00e92d693cbc","Type":"ContainerStarted","Data":"439aaeaadfd6c24d0689894ccf27460b2c64c87f67b92b7910b22c7554a427bb"}
Sep 10 20:58:45.078493 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:45.078452    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k" event={"ID":"bf04e82d-33fb-488d-950a-00e92d693cbc","Type":"ContainerStarted","Data":"373553354f58ef9b39cfb0fcbfe883eefc8d10ee18a1e8519ab43e9c59e29d26"}
Sep 10 20:58:45.126818 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:45.126764    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k" podStartSLOduration=2.12674251 podStartE2EDuration="2.12674251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:45.123505937 +0000 UTC m=+1365.088574133" watchObservedRunningTime="2026-09-10 20:58:45.12674251 +0000 UTC m=+1365.091810707"
Sep 10 20:58:45.280234 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:45.280201    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:58:45.285137 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:45.285112    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:58:46.086077 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:46.086038    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn" event={"ID":"c328c839-e286-49b2-ad8b-3a472c32d624","Type":"ContainerStarted","Data":"dc7e19fb7a49a523009afc4b620afee54d79f08e3f90c8b838e1fb3998ce1812"}
Sep 10 20:58:46.086599 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:46.086461    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:58:46.087379 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:46.087357    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:58:46.186766 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:46.186733    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:46.187094 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:46.187067    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 10 20:58:46.187222 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:46.187196    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn" podUID="c328c839-e286-49b2-ad8b-3a472c32d624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 10 20:58:47.186129 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:47.186097    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 10 20:58:47.186495 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:47.186155    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn" podUID="c328c839-e286-49b2-ad8b-3a472c32d624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 10 20:58:48.186275 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:48.186233    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 10 20:58:48.186740 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:48.186311    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn" podUID="c328c839-e286-49b2-ad8b-3a472c32d624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 10 20:58:48.881331 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:48.881296    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f4fe4f-698489dd7f-jq8jc"]
Sep 10 20:58:48.884757 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:48.884729    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4fe4f-698489dd7f-jq8jc"
Sep 10 20:58:48.887120 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:48.887097    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-f4fe4f-dockercfg-sqpnr\""
Sep 10 20:58:48.887120 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:48.887112    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-f4fe4f\""
Sep 10 20:58:48.893771 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:48.893748    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f4fe4f-698489dd7f-jq8jc"]
Sep 10 20:58:48.896917 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:48.896897    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/415e549a-63ca-459a-ad45-452a01d6b98a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-f4fe4f-698489dd7f-jq8jc\" (UID: \"415e549a-63ca-459a-ad45-452a01d6b98a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4fe4f-698489dd7f-jq8jc"
Sep 10 20:58:48.897027 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:48.896954    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrjwg\" (UniqueName: \"kubernetes.io/projected/415e549a-63ca-459a-ad45-452a01d6b98a-kube-api-access-wrjwg\") pod \"payload-processing-e2e-shared-b-w3-f4fe4f-698489dd7f-jq8jc\" (UID: \"415e549a-63ca-459a-ad45-452a01d6b98a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4fe4f-698489dd7f-jq8jc"
Sep 10 20:58:48.997835 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:48.997803    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrjwg\" (UniqueName: \"kubernetes.io/projected/415e549a-63ca-459a-ad45-452a01d6b98a-kube-api-access-wrjwg\") pod \"payload-processing-e2e-shared-b-w3-f4fe4f-698489dd7f-jq8jc\" (UID: \"415e549a-63ca-459a-ad45-452a01d6b98a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4fe4f-698489dd7f-jq8jc"
Sep 10 20:58:48.998047 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:48.997865    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/415e549a-63ca-459a-ad45-452a01d6b98a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-f4fe4f-698489dd7f-jq8jc\" (UID: \"415e549a-63ca-459a-ad45-452a01d6b98a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4fe4f-698489dd7f-jq8jc"
Sep 10 20:58:48.998593 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:48.998566    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/415e549a-63ca-459a-ad45-452a01d6b98a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-f4fe4f-698489dd7f-jq8jc\" (UID: \"415e549a-63ca-459a-ad45-452a01d6b98a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4fe4f-698489dd7f-jq8jc"
Sep 10 20:58:49.009390 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:49.009360    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrjwg\" (UniqueName: \"kubernetes.io/projected/415e549a-63ca-459a-ad45-452a01d6b98a-kube-api-access-wrjwg\") pod \"payload-processing-e2e-shared-b-w3-f4fe4f-698489dd7f-jq8jc\" (UID: \"415e549a-63ca-459a-ad45-452a01d6b98a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4fe4f-698489dd7f-jq8jc"
Sep 10 20:58:49.187025 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:49.186942    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 10 20:58:49.187025 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:49.187002    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn" podUID="c328c839-e286-49b2-ad8b-3a472c32d624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 10 20:58:49.195974 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:49.195952    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4fe4f-698489dd7f-jq8jc"
Sep 10 20:58:49.479768 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:49.479743    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f4fe4f-698489dd7f-jq8jc"]
Sep 10 20:58:49.482403 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:58:49.482364    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod415e549a_63ca_459a_ad45_452a01d6b98a.slice/crio-2fb9ed9085f6488abb988e56cf2239da1862cb43d12bc6cbfe3e9ad1730ad513 WatchSource:0}: Error finding container 2fb9ed9085f6488abb988e56cf2239da1862cb43d12bc6cbfe3e9ad1730ad513: Status 404 returned error can't find the container with id 2fb9ed9085f6488abb988e56cf2239da1862cb43d12bc6cbfe3e9ad1730ad513
Sep 10 20:58:50.113558 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:50.113519    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4fe4f-698489dd7f-jq8jc" event={"ID":"415e549a-63ca-459a-ad45-452a01d6b98a","Type":"ContainerStarted","Data":"eb2d560db59642071db059e4c2c55a50bbb70acfe9c0535481893c6b3699aff8"}
Sep 10 20:58:50.113765 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:50.113594    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4fe4f-698489dd7f-jq8jc" event={"ID":"415e549a-63ca-459a-ad45-452a01d6b98a","Type":"ContainerStarted","Data":"2fb9ed9085f6488abb988e56cf2239da1862cb43d12bc6cbfe3e9ad1730ad513"}
Sep 10 20:58:50.113765 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:50.113623    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4fe4f-698489dd7f-jq8jc"
Sep 10 20:58:50.134460 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:50.134413    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4fe4f-698489dd7f-jq8jc" podStartSLOduration=2.134398527 podStartE2EDuration="2.134398527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:50.132526887 +0000 UTC m=+1370.097595085" watchObservedRunningTime="2026-09-10 20:58:50.134398527 +0000 UTC m=+1370.099466785"
Sep 10 20:58:50.186972 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:50.186938    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 10 20:58:50.187099 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:50.187005    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn" podUID="c328c839-e286-49b2-ad8b-3a472c32d624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 10 20:58:51.190015 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:51.189986    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:51.190426 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:51.190351    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:58:51.191134 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:58:51.191113    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 20:59:01.122222 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:01.122190    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4fe4f-698489dd7f-jq8jc"
Sep 10 20:59:05.211241 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:05.211202    2564 generic.go:356] "Generic (PLEG): container finished" podID="bf04e82d-33fb-488d-950a-00e92d693cbc" containerID="439aaeaadfd6c24d0689894ccf27460b2c64c87f67b92b7910b22c7554a427bb" exitCode=0
Sep 10 20:59:05.211660 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:05.211254    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k" event={"ID":"bf04e82d-33fb-488d-950a-00e92d693cbc","Type":"ContainerDied","Data":"439aaeaadfd6c24d0689894ccf27460b2c64c87f67b92b7910b22c7554a427bb"}
Sep 10 20:59:05.211660 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:05.211620    2564 scope.go:117] "RemoveContainer" containerID="439aaeaadfd6c24d0689894ccf27460b2c64c87f67b92b7910b22c7554a427bb"
Sep 10 20:59:05.212762 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:05.212722    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 20:59:05.212863 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:05.212807    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 20:59:05.212863 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:05.212841    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 20:59:06.218801 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:06.218763    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k" event={"ID":"bf04e82d-33fb-488d-950a-00e92d693cbc","Type":"ContainerStarted","Data":"4520907ee4e7c68acdf3998032323f2b10e86f668ba3d802e67e96da072a088d"}
Sep 10 20:59:06.280522 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:06.280478    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:59:06.280805 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:06.280777    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 10 20:59:06.280866 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:06.280841    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k" podUID="bf04e82d-33fb-488d-950a-00e92d693cbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 10 20:59:07.281090 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:07.281053    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 10 20:59:07.281538 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:07.281124    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k" podUID="bf04e82d-33fb-488d-950a-00e92d693cbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 10 20:59:08.280432 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:08.280391    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 10 20:59:08.280618 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:08.280471    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k" podUID="bf04e82d-33fb-488d-950a-00e92d693cbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 10 20:59:08.308290 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:08.308260    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3d4d0c56-578467cfd6-mrh8f"]
Sep 10 20:59:08.311824 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:08.311802    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d4d0c56-578467cfd6-mrh8f"
Sep 10 20:59:08.321343 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:08.321324    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3d4d0c56\""
Sep 10 20:59:08.354926 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:08.354901    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3d4d0c56-578467cfd6-mrh8f"]
Sep 10 20:59:08.375596 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:08.375561    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28122557-8469-4a84-ba98-de2bf3389ae9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3d4d0c56-578467cfd6-mrh8f\" (UID: \"28122557-8469-4a84-ba98-de2bf3389ae9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3d4d0c56-578467cfd6-mrh8f"
Sep 10 20:59:08.375726 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:08.375659    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5psh\" (UniqueName: \"kubernetes.io/projected/28122557-8469-4a84-ba98-de2bf3389ae9-kube-api-access-t5psh\") pod \"payload-pre-processing-e2e-mt-3d4d0c56-578467cfd6-mrh8f\" (UID: \"28122557-8469-4a84-ba98-de2bf3389ae9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3d4d0c56-578467cfd6-mrh8f"
Sep 10 20:59:08.477194 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:08.476592    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28122557-8469-4a84-ba98-de2bf3389ae9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3d4d0c56-578467cfd6-mrh8f\" (UID: \"28122557-8469-4a84-ba98-de2bf3389ae9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3d4d0c56-578467cfd6-mrh8f"
Sep 10 20:59:08.477194 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:08.476694    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5psh\" (UniqueName: \"kubernetes.io/projected/28122557-8469-4a84-ba98-de2bf3389ae9-kube-api-access-t5psh\") pod \"payload-pre-processing-e2e-mt-3d4d0c56-578467cfd6-mrh8f\" (UID: \"28122557-8469-4a84-ba98-de2bf3389ae9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3d4d0c56-578467cfd6-mrh8f"
Sep 10 20:59:08.477790 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:08.477764    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28122557-8469-4a84-ba98-de2bf3389ae9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3d4d0c56-578467cfd6-mrh8f\" (UID: \"28122557-8469-4a84-ba98-de2bf3389ae9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3d4d0c56-578467cfd6-mrh8f"
Sep 10 20:59:08.492897 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:08.492865    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5psh\" (UniqueName: \"kubernetes.io/projected/28122557-8469-4a84-ba98-de2bf3389ae9-kube-api-access-t5psh\") pod \"payload-pre-processing-e2e-mt-3d4d0c56-578467cfd6-mrh8f\" (UID: \"28122557-8469-4a84-ba98-de2bf3389ae9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3d4d0c56-578467cfd6-mrh8f"
Sep 10 20:59:08.621567 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:08.621541    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d4d0c56-578467cfd6-mrh8f"
Sep 10 20:59:08.924489 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:59:08.924462    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28122557_8469_4a84_ba98_de2bf3389ae9.slice/crio-029bab400a4ba6109c7cd887411e91b40e4019696dff326f9d4b49bd764b1769 WatchSource:0}: Error finding container 029bab400a4ba6109c7cd887411e91b40e4019696dff326f9d4b49bd764b1769: Status 404 returned error can't find the container with id 029bab400a4ba6109c7cd887411e91b40e4019696dff326f9d4b49bd764b1769
Sep 10 20:59:08.942883 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:08.942857    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3d4d0c56-578467cfd6-mrh8f"]
Sep 10 20:59:09.239131 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:09.239050    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d4d0c56-578467cfd6-mrh8f" event={"ID":"28122557-8469-4a84-ba98-de2bf3389ae9","Type":"ContainerStarted","Data":"b1dcd233570f7f84d3512d0f6fd9b04ced1aa16d3740a7f47bdd999a929e5a48"}
Sep 10 20:59:09.239131 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:09.239102    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d4d0c56-578467cfd6-mrh8f" event={"ID":"28122557-8469-4a84-ba98-de2bf3389ae9","Type":"ContainerStarted","Data":"029bab400a4ba6109c7cd887411e91b40e4019696dff326f9d4b49bd764b1769"}
Sep 10 20:59:09.239341 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:09.239135    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d4d0c56-578467cfd6-mrh8f"
Sep 10 20:59:09.280509 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:09.280473    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 10 20:59:09.280653 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:09.280545    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k" podUID="bf04e82d-33fb-488d-950a-00e92d693cbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 10 20:59:09.311536 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:09.311491    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d4d0c56-578467cfd6-mrh8f" podStartSLOduration=1.311477909 podStartE2EDuration="1.311477909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:09.308507301 +0000 UTC m=+1389.273575495" watchObservedRunningTime="2026-09-10 20:59:09.311477909 +0000 UTC m=+1389.276546129"
Sep 10 20:59:10.281098 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:10.281064    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 10 20:59:10.281289 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:10.281115    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k" podUID="bf04e82d-33fb-488d-950a-00e92d693cbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 10 20:59:11.280956 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:11.280915    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 10 20:59:11.281438 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:11.280971    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k" podUID="bf04e82d-33fb-488d-950a-00e92d693cbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 10 20:59:12.284737 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:12.284708    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:59:12.285184 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:12.284982    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:59:12.285683 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:12.285665    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 20:59:20.247216 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:20.247161    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d4d0c56-578467cfd6-mrh8f"
Sep 10 20:59:21.110779 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:21.110747    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls"]
Sep 10 20:59:21.114439 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:21.114418    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls"
Sep 10 20:59:21.117803 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:21.117781    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3fc2b8\"/\"auth-test-model-3fc2b8-kserve-self-signed-certs\""
Sep 10 20:59:21.117904 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:21.117804    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3fc2b8\"/\"kube-root-ca.crt\""
Sep 10 20:59:21.117904 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:21.117854    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3fc2b8\"/\"openshift-service-ca.crt\""
Sep 10 20:59:21.117985 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:21.117930    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3fc2b8\"/\"default-dockercfg-4hgln\""
Sep 10 20:59:21.127515 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:21.127490    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls"]
Sep 10 20:59:21.282901 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:21.282871    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgj87\" (UniqueName: \"kubernetes.io/projected/2342ad2a-819f-4c84-a90a-d26346a881f7-kube-api-access-bgj87\") pod \"auth-test-model-3fc2b8-kserve-f96f5f95-t4bls\" (UID: \"2342ad2a-819f-4c84-a90a-d26346a881f7\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls"
Sep 10 20:59:21.283229 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:21.282906    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2342ad2a-819f-4c84-a90a-d26346a881f7-tmp-dir\") pod \"auth-test-model-3fc2b8-kserve-f96f5f95-t4bls\" (UID: \"2342ad2a-819f-4c84-a90a-d26346a881f7\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls"
Sep 10 20:59:21.283229 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:21.282985    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2342ad2a-819f-4c84-a90a-d26346a881f7-model-cache\") pod \"auth-test-model-3fc2b8-kserve-f96f5f95-t4bls\" (UID: \"2342ad2a-819f-4c84-a90a-d26346a881f7\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls"
Sep 10 20:59:21.283229 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:21.283019    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2342ad2a-819f-4c84-a90a-d26346a881f7-home\") pod \"auth-test-model-3fc2b8-kserve-f96f5f95-t4bls\" (UID: \"2342ad2a-819f-4c84-a90a-d26346a881f7\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls"
Sep 10 20:59:21.283229 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:21.283050    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2342ad2a-819f-4c84-a90a-d26346a881f7-dshm\") pod \"auth-test-model-3fc2b8-kserve-f96f5f95-t4bls\" (UID: \"2342ad2a-819f-4c84-a90a-d26346a881f7\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls"
Sep 10 20:59:21.283229 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:21.283064    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2342ad2a-819f-4c84-a90a-d26346a881f7-tls-certs\") pod \"auth-test-model-3fc2b8-kserve-f96f5f95-t4bls\" (UID: \"2342ad2a-819f-4c84-a90a-d26346a881f7\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls"
Sep 10 20:59:21.384110 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:21.384030    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgj87\" (UniqueName: \"kubernetes.io/projected/2342ad2a-819f-4c84-a90a-d26346a881f7-kube-api-access-bgj87\") pod \"auth-test-model-3fc2b8-kserve-f96f5f95-t4bls\" (UID: \"2342ad2a-819f-4c84-a90a-d26346a881f7\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls"
Sep 10 20:59:21.384110 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:21.384067    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2342ad2a-819f-4c84-a90a-d26346a881f7-tmp-dir\") pod \"auth-test-model-3fc2b8-kserve-f96f5f95-t4bls\" (UID: \"2342ad2a-819f-4c84-a90a-d26346a881f7\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls"
Sep 10 20:59:21.384354 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:21.384113    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2342ad2a-819f-4c84-a90a-d26346a881f7-model-cache\") pod \"auth-test-model-3fc2b8-kserve-f96f5f95-t4bls\" (UID: \"2342ad2a-819f-4c84-a90a-d26346a881f7\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls"
Sep 10 20:59:21.384354 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:21.384136    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2342ad2a-819f-4c84-a90a-d26346a881f7-home\") pod \"auth-test-model-3fc2b8-kserve-f96f5f95-t4bls\" (UID: \"2342ad2a-819f-4c84-a90a-d26346a881f7\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls"
Sep 10 20:59:21.384354 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:21.384184    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2342ad2a-819f-4c84-a90a-d26346a881f7-dshm\") pod \"auth-test-model-3fc2b8-kserve-f96f5f95-t4bls\" (UID: \"2342ad2a-819f-4c84-a90a-d26346a881f7\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls"
Sep 10 20:59:21.384354 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:21.384199    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2342ad2a-819f-4c84-a90a-d26346a881f7-tls-certs\") pod \"auth-test-model-3fc2b8-kserve-f96f5f95-t4bls\" (UID: \"2342ad2a-819f-4c84-a90a-d26346a881f7\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls"
Sep 10 20:59:21.384595 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:21.384568    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2342ad2a-819f-4c84-a90a-d26346a881f7-model-cache\") pod \"auth-test-model-3fc2b8-kserve-f96f5f95-t4bls\" (UID: \"2342ad2a-819f-4c84-a90a-d26346a881f7\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls"
Sep 10 20:59:21.384676 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:21.384651    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2342ad2a-819f-4c84-a90a-d26346a881f7-tmp-dir\") pod \"auth-test-model-3fc2b8-kserve-f96f5f95-t4bls\" (UID: \"2342ad2a-819f-4c84-a90a-d26346a881f7\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls"
Sep 10 20:59:21.384736 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:21.384657    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2342ad2a-819f-4c84-a90a-d26346a881f7-home\") pod \"auth-test-model-3fc2b8-kserve-f96f5f95-t4bls\" (UID: \"2342ad2a-819f-4c84-a90a-d26346a881f7\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls"
Sep 10 20:59:21.386543 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:21.386511    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2342ad2a-819f-4c84-a90a-d26346a881f7-dshm\") pod \"auth-test-model-3fc2b8-kserve-f96f5f95-t4bls\" (UID: \"2342ad2a-819f-4c84-a90a-d26346a881f7\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls"
Sep 10 20:59:21.386803 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:21.386783    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2342ad2a-819f-4c84-a90a-d26346a881f7-tls-certs\") pod \"auth-test-model-3fc2b8-kserve-f96f5f95-t4bls\" (UID: \"2342ad2a-819f-4c84-a90a-d26346a881f7\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls"
Sep 10 20:59:21.394298 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:21.394272    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgj87\" (UniqueName: \"kubernetes.io/projected/2342ad2a-819f-4c84-a90a-d26346a881f7-kube-api-access-bgj87\") pod \"auth-test-model-3fc2b8-kserve-f96f5f95-t4bls\" (UID: \"2342ad2a-819f-4c84-a90a-d26346a881f7\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls"
Sep 10 20:59:21.423868 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:21.423843    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls"
Sep 10 20:59:21.695570 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:21.695397    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls"]
Sep 10 20:59:21.698465 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:59:21.698439    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2342ad2a_819f_4c84_a90a_d26346a881f7.slice/crio-07293d6ab7b28cfa191008e91e658424de7dbf13a720455bb296f68f74739eba WatchSource:0}: Error finding container 07293d6ab7b28cfa191008e91e658424de7dbf13a720455bb296f68f74739eba: Status 404 returned error can't find the container with id 07293d6ab7b28cfa191008e91e658424de7dbf13a720455bb296f68f74739eba
Sep 10 20:59:22.327353 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:22.327312    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls" event={"ID":"2342ad2a-819f-4c84-a90a-d26346a881f7","Type":"ContainerStarted","Data":"ad87c4c51cc4091e40f5e3aad74944d29eecdf1e749830d09551e62fb193d35c"}
Sep 10 20:59:22.327762 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:22.327374    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls" event={"ID":"2342ad2a-819f-4c84-a90a-d26346a881f7","Type":"ContainerStarted","Data":"07293d6ab7b28cfa191008e91e658424de7dbf13a720455bb296f68f74739eba"}
Sep 10 20:59:22.350888 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:22.350834    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls" podStartSLOduration=1.350819788 podStartE2EDuration="1.350819788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:22.348432928 +0000 UTC m=+1402.313501144" watchObservedRunningTime="2026-09-10 20:59:22.350819788 +0000 UTC m=+1402.315887982"
Sep 10 20:59:31.424497 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:31.424462    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls"
Sep 10 20:59:31.424915 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:31.424509    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls"
Sep 10 20:59:31.436778 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:31.436755    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls"
Sep 10 20:59:32.396278 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:32.396250    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls"
Sep 10 20:59:56.021010 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:56.020930    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf"]
Sep 10 20:59:56.024803 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:56.024779    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf"
Sep 10 20:59:56.028432 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:56.028407    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3d4d0c56\"/\"e2e-lifecycle-model-3d4d0c56-kserve-self-signed-certs\""
Sep 10 20:59:56.028560 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:56.028435    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3d4d0c56\"/\"openshift-service-ca.crt\""
Sep 10 20:59:56.028560 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:56.028464    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3d4d0c56\"/\"kube-root-ca.crt\""
Sep 10 20:59:56.028560 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:56.028468    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3d4d0c56\"/\"default-dockercfg-jckf6\""
Sep 10 20:59:56.036409 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:56.036386    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf"]
Sep 10 20:59:56.053092 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:56.053066    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9233dba-d866-4f6b-a68b-7761bda89097-tmp-dir\") pod \"e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf\" (UID: \"d9233dba-d866-4f6b-a68b-7761bda89097\") " pod="ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf"
Sep 10 20:59:56.053218 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:56.053109    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmj9z\" (UniqueName: \"kubernetes.io/projected/d9233dba-d866-4f6b-a68b-7761bda89097-kube-api-access-wmj9z\") pod \"e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf\" (UID: \"d9233dba-d866-4f6b-a68b-7761bda89097\") " pod="ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf"
Sep 10 20:59:56.053218 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:56.053194    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9233dba-d866-4f6b-a68b-7761bda89097-model-cache\") pod \"e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf\" (UID: \"d9233dba-d866-4f6b-a68b-7761bda89097\") " pod="ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf"
Sep 10 20:59:56.053321 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:56.053225    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9233dba-d866-4f6b-a68b-7761bda89097-tls-certs\") pod \"e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf\" (UID: \"d9233dba-d866-4f6b-a68b-7761bda89097\") " pod="ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf"
Sep 10 20:59:56.053321 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:56.053257    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9233dba-d866-4f6b-a68b-7761bda89097-home\") pod \"e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf\" (UID: \"d9233dba-d866-4f6b-a68b-7761bda89097\") " pod="ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf"
Sep 10 20:59:56.053321 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:56.053305    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9233dba-d866-4f6b-a68b-7761bda89097-dshm\") pod \"e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf\" (UID: \"d9233dba-d866-4f6b-a68b-7761bda89097\") " pod="ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf"
Sep 10 20:59:56.154750 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:56.154719    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9233dba-d866-4f6b-a68b-7761bda89097-model-cache\") pod \"e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf\" (UID: \"d9233dba-d866-4f6b-a68b-7761bda89097\") " pod="ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf"
Sep 10 20:59:56.154927 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:56.154778    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9233dba-d866-4f6b-a68b-7761bda89097-tls-certs\") pod \"e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf\" (UID: \"d9233dba-d866-4f6b-a68b-7761bda89097\") " pod="ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf"
Sep 10 20:59:56.154927 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:56.154828    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9233dba-d866-4f6b-a68b-7761bda89097-home\") pod \"e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf\" (UID: \"d9233dba-d866-4f6b-a68b-7761bda89097\") " pod="ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf"
Sep 10 20:59:56.154927 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:56.154852    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9233dba-d866-4f6b-a68b-7761bda89097-dshm\") pod \"e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf\" (UID: \"d9233dba-d866-4f6b-a68b-7761bda89097\") " pod="ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf"
Sep 10 20:59:56.154927 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:56.154887    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9233dba-d866-4f6b-a68b-7761bda89097-tmp-dir\") pod \"e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf\" (UID: \"d9233dba-d866-4f6b-a68b-7761bda89097\") " pod="ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf"
Sep 10 20:59:56.155155 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:56.154932    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmj9z\" (UniqueName: \"kubernetes.io/projected/d9233dba-d866-4f6b-a68b-7761bda89097-kube-api-access-wmj9z\") pod \"e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf\" (UID: \"d9233dba-d866-4f6b-a68b-7761bda89097\") " pod="ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf"
Sep 10 20:59:56.155516 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:56.155489    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9233dba-d866-4f6b-a68b-7761bda89097-home\") pod \"e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf\" (UID: \"d9233dba-d866-4f6b-a68b-7761bda89097\") " pod="ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf"
Sep 10 20:59:56.155628 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:56.155537    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9233dba-d866-4f6b-a68b-7761bda89097-model-cache\") pod \"e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf\" (UID: \"d9233dba-d866-4f6b-a68b-7761bda89097\") " pod="ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf"
Sep 10 20:59:56.155693 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:56.155646    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9233dba-d866-4f6b-a68b-7761bda89097-tmp-dir\") pod \"e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf\" (UID: \"d9233dba-d866-4f6b-a68b-7761bda89097\") " pod="ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf"
Sep 10 20:59:56.157218 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:56.157196    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9233dba-d866-4f6b-a68b-7761bda89097-dshm\") pod \"e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf\" (UID: \"d9233dba-d866-4f6b-a68b-7761bda89097\") " pod="ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf"
Sep 10 20:59:56.157392 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:56.157373    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9233dba-d866-4f6b-a68b-7761bda89097-tls-certs\") pod \"e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf\" (UID: \"d9233dba-d866-4f6b-a68b-7761bda89097\") " pod="ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf"
Sep 10 20:59:56.162777 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:56.162756    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmj9z\" (UniqueName: \"kubernetes.io/projected/d9233dba-d866-4f6b-a68b-7761bda89097-kube-api-access-wmj9z\") pod \"e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf\" (UID: \"d9233dba-d866-4f6b-a68b-7761bda89097\") " pod="ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf"
Sep 10 20:59:56.336553 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:56.336530    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf"
Sep 10 20:59:56.615522 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:56.615499    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf"]
Sep 10 20:59:56.617718 ip-10-0-141-185 kubenswrapper[2564]: W0910 20:59:56.617689    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9233dba_d866_4f6b_a68b_7761bda89097.slice/crio-1217693551f3093f42568a79aa37f3a29a0c5b0fdc8c6fa6af2166a765986ffd WatchSource:0}: Error finding container 1217693551f3093f42568a79aa37f3a29a0c5b0fdc8c6fa6af2166a765986ffd: Status 404 returned error can't find the container with id 1217693551f3093f42568a79aa37f3a29a0c5b0fdc8c6fa6af2166a765986ffd
Sep 10 20:59:57.530475 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:57.530439    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf" event={"ID":"d9233dba-d866-4f6b-a68b-7761bda89097","Type":"ContainerStarted","Data":"cf447daa51aaf1054343d12b86d82b701f368474957b9b0759671735a51ea08d"}
Sep 10 20:59:57.530824 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:57.530480    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf" event={"ID":"d9233dba-d866-4f6b-a68b-7761bda89097","Type":"ContainerStarted","Data":"1217693551f3093f42568a79aa37f3a29a0c5b0fdc8c6fa6af2166a765986ffd"}
Sep 10 20:59:57.552831 ip-10-0-141-185 kubenswrapper[2564]: I0910 20:59:57.552785    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf" podStartSLOduration=2.552770194 podStartE2EDuration="2.552770194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:57.551482312 +0000 UTC m=+1437.516550507" watchObservedRunningTime="2026-09-10 20:59:57.552770194 +0000 UTC m=+1437.517838388"
Sep 10 21:00:00.164254 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:00.164212    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bbb40-29817900-22lht"]
Sep 10 21:00:00.168089 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:00.168064    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bbb40-29817900-22lht"
Sep 10 21:00:00.170756 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:00.170733    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plv2h\""
Sep 10 21:00:00.178746 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:00.178722    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bbb40-29817900-22lht"
Sep 10 21:00:00.186571 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:00.186548    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cd89ac-29817900-qftgk"]
Sep 10 21:00:00.200529 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:00.200496    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bbb40-29817900-22lht"]
Sep 10 21:00:00.200665 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:00.200559    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817900-cg7gk"]
Sep 10 21:00:00.200665 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:00.200608    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cd89ac-29817900-qftgk"
Sep 10 21:00:00.208208 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:00.208134    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cd89ac-29817900-qftgk"]
Sep 10 21:00:00.208847 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:00.208826    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817900-cg7gk"
Sep 10 21:00:00.211868 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:00.211414    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817900-cg7gk"]
Sep 10 21:00:00.215704 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:00.215675    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3fc2b8-29817900-dh9fz"]
Sep 10 21:00:00.219995 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:00.219967    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3fc2b8-29817900-dh9fz"
Sep 10 21:00:00.228906 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:00.228881    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3fc2b8-29817900-dh9fz"]
Sep 10 21:00:00.232077 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:00.232050    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cd89ac-29817900-qftgk"
Sep 10 21:00:00.243564 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:00.243547    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817900-cg7gk"
Sep 10 21:00:00.291012 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:00.290982    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3fc2b8-29817900-dh9fz"
Sep 10 21:00:00.475661 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:00.475636    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bbb40-29817900-22lht"]
Sep 10 21:00:00.478375 ip-10-0-141-185 kubenswrapper[2564]: W0910 21:00:00.478349    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa5c29a_1c7a_46a1_809c_269ca28a4057.slice/crio-340731f6c026f60c9d4671754e330102d63dc92880b4dd126f5643a85b37c9af WatchSource:0}: Error finding container 340731f6c026f60c9d4671754e330102d63dc92880b4dd126f5643a85b37c9af: Status 404 returned error can't find the container with id 340731f6c026f60c9d4671754e330102d63dc92880b4dd126f5643a85b37c9af
Sep 10 21:00:00.542805 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:00.542782    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cd89ac-29817900-qftgk"]
Sep 10 21:00:00.544386 ip-10-0-141-185 kubenswrapper[2564]: W0910 21:00:00.544359    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8bc1141_a53e_4e3c_9abd_85fb096a5c95.slice/crio-c1f93fdd9fd4e420a7f4a0fac34f59c47013be09bbe92d9d7cab13ac9ccbb9cd WatchSource:0}: Error finding container c1f93fdd9fd4e420a7f4a0fac34f59c47013be09bbe92d9d7cab13ac9ccbb9cd: Status 404 returned error can't find the container with id c1f93fdd9fd4e420a7f4a0fac34f59c47013be09bbe92d9d7cab13ac9ccbb9cd
Sep 10 21:00:00.547640 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:00.547579    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817900-cg7gk"]
Sep 10 21:00:00.549085 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:00.549065    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bbb40-29817900-22lht" event={"ID":"2fa5c29a-1c7a-46a1-809c-269ca28a4057","Type":"ContainerStarted","Data":"340731f6c026f60c9d4671754e330102d63dc92880b4dd126f5643a85b37c9af"}
Sep 10 21:00:00.549556 ip-10-0-141-185 kubenswrapper[2564]: W0910 21:00:00.549539    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a8d782_4828_4beb_8614_93cb5b754b4d.slice/crio-3e01e456f07c3323a57cc7550fd3254997786e1008e2a3a0f979b33631f3ddae WatchSource:0}: Error finding container 3e01e456f07c3323a57cc7550fd3254997786e1008e2a3a0f979b33631f3ddae: Status 404 returned error can't find the container with id 3e01e456f07c3323a57cc7550fd3254997786e1008e2a3a0f979b33631f3ddae
Sep 10 21:00:00.573503 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:00.573484    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3fc2b8-29817900-dh9fz"]
Sep 10 21:00:00.575198 ip-10-0-141-185 kubenswrapper[2564]: W0910 21:00:00.575149    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ae561c_a2b9_4a2c_a95d_9c1125fc2d8e.slice/crio-2e5bf9e70fce523d0c4464090fe4a774d0c63055ac03e093c6cd3dfb9bb5189b WatchSource:0}: Error finding container 2e5bf9e70fce523d0c4464090fe4a774d0c63055ac03e093c6cd3dfb9bb5189b: Status 404 returned error can't find the container with id 2e5bf9e70fce523d0c4464090fe4a774d0c63055ac03e093c6cd3dfb9bb5189b
Sep 10 21:00:01.557635 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:01.557536    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3fc2b8-29817900-dh9fz" event={"ID":"09ae561c-a2b9-4a2c-a95d-9c1125fc2d8e","Type":"ContainerStarted","Data":"f219ca5bb8eb84fca80857e6f2b5df20f99c5d4e199e0a741afa2d552917dc61"}
Sep 10 21:00:01.557635 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:01.557599    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3fc2b8-29817900-dh9fz" event={"ID":"09ae561c-a2b9-4a2c-a95d-9c1125fc2d8e","Type":"ContainerStarted","Data":"2e5bf9e70fce523d0c4464090fe4a774d0c63055ac03e093c6cd3dfb9bb5189b"}
Sep 10 21:00:01.559733 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:01.559704    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bbb40-29817900-22lht" event={"ID":"2fa5c29a-1c7a-46a1-809c-269ca28a4057","Type":"ContainerStarted","Data":"6ef1160eba92ce11cdeca69cb0585fc5b3bab23ce1a649216a1e19cf29357869"}
Sep 10 21:00:01.561405 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:01.561315    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817900-cg7gk" event={"ID":"49a8d782-4828-4beb-8614-93cb5b754b4d","Type":"ContainerStarted","Data":"66b223a454204b62fcc17cf804ef61fc9441e8bdbd7522669abd39d858e6fc69"}
Sep 10 21:00:01.561405 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:01.561363    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817900-cg7gk" event={"ID":"49a8d782-4828-4beb-8614-93cb5b754b4d","Type":"ContainerStarted","Data":"3e01e456f07c3323a57cc7550fd3254997786e1008e2a3a0f979b33631f3ddae"}
Sep 10 21:00:01.564813 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:01.564508    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cd89ac-29817900-qftgk" event={"ID":"f8bc1141-a53e-4e3c-9abd-85fb096a5c95","Type":"ContainerStarted","Data":"78f206d30c974fc4803652e0d8c8280115cd191f0b22488c8d26a1b23d9c610b"}
Sep 10 21:00:01.564813 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:01.564613    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cd89ac-29817900-qftgk" event={"ID":"f8bc1141-a53e-4e3c-9abd-85fb096a5c95","Type":"ContainerStarted","Data":"c1f93fdd9fd4e420a7f4a0fac34f59c47013be09bbe92d9d7cab13ac9ccbb9cd"}
Sep 10 21:00:01.585131 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:01.585087    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3fc2b8-29817900-dh9fz" podStartSLOduration=1.585074528 podStartE2EDuration="1.585074528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:01.583188523 +0000 UTC m=+1441.548256719" watchObservedRunningTime="2026-09-10 21:00:01.585074528 +0000 UTC m=+1441.550142722"
Sep 10 21:00:01.617244 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:01.617204    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817900-cg7gk" podStartSLOduration=1.6171913770000002 podStartE2EDuration="1.617191377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:01.599991149 +0000 UTC m=+1441.565059345" watchObservedRunningTime="2026-09-10 21:00:01.617191377 +0000 UTC m=+1441.582259563"
Sep 10 21:00:01.617960 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:01.617935    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bbb40-29817900-22lht" podStartSLOduration=1.6179256629999998 podStartE2EDuration="1.617925663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:01.616377056 +0000 UTC m=+1441.581445251" watchObservedRunningTime="2026-09-10 21:00:01.617925663 +0000 UTC m=+1441.582993858"
Sep 10 21:00:01.633245 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:01.633209    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cd89ac-29817900-qftgk" podStartSLOduration=1.633196539 podStartE2EDuration="1.633196539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:01.632813448 +0000 UTC m=+1441.597881642" watchObservedRunningTime="2026-09-10 21:00:01.633196539 +0000 UTC m=+1441.598264734"
Sep 10 21:00:02.571062 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:02.571028    2564 generic.go:356] "Generic (PLEG): container finished" podID="09ae561c-a2b9-4a2c-a95d-9c1125fc2d8e" containerID="f219ca5bb8eb84fca80857e6f2b5df20f99c5d4e199e0a741afa2d552917dc61" exitCode=0
Sep 10 21:00:02.571477 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:02.571081    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3fc2b8-29817900-dh9fz" event={"ID":"09ae561c-a2b9-4a2c-a95d-9c1125fc2d8e","Type":"ContainerDied","Data":"f219ca5bb8eb84fca80857e6f2b5df20f99c5d4e199e0a741afa2d552917dc61"}
Sep 10 21:00:02.572764 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:02.572742    2564 generic.go:356] "Generic (PLEG): container finished" podID="2fa5c29a-1c7a-46a1-809c-269ca28a4057" containerID="6ef1160eba92ce11cdeca69cb0585fc5b3bab23ce1a649216a1e19cf29357869" exitCode=0
Sep 10 21:00:02.572894 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:02.572819    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bbb40-29817900-22lht" event={"ID":"2fa5c29a-1c7a-46a1-809c-269ca28a4057","Type":"ContainerDied","Data":"6ef1160eba92ce11cdeca69cb0585fc5b3bab23ce1a649216a1e19cf29357869"}
Sep 10 21:00:02.574339 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:02.574317    2564 generic.go:356] "Generic (PLEG): container finished" podID="49a8d782-4828-4beb-8614-93cb5b754b4d" containerID="66b223a454204b62fcc17cf804ef61fc9441e8bdbd7522669abd39d858e6fc69" exitCode=0
Sep 10 21:00:02.574442 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:02.574378    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817900-cg7gk" event={"ID":"49a8d782-4828-4beb-8614-93cb5b754b4d","Type":"ContainerDied","Data":"66b223a454204b62fcc17cf804ef61fc9441e8bdbd7522669abd39d858e6fc69"}
Sep 10 21:00:02.575816 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:02.575795    2564 generic.go:356] "Generic (PLEG): container finished" podID="f8bc1141-a53e-4e3c-9abd-85fb096a5c95" containerID="78f206d30c974fc4803652e0d8c8280115cd191f0b22488c8d26a1b23d9c610b" exitCode=0
Sep 10 21:00:02.575923 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:02.575873    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cd89ac-29817900-qftgk" event={"ID":"f8bc1141-a53e-4e3c-9abd-85fb096a5c95","Type":"ContainerDied","Data":"78f206d30c974fc4803652e0d8c8280115cd191f0b22488c8d26a1b23d9c610b"}
Sep 10 21:00:03.723144 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:03.723124    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bbb40-29817900-22lht"
Sep 10 21:00:03.780953 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:03.780930    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3fc2b8-29817900-dh9fz"
Sep 10 21:00:03.796935 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:03.796866    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817900-cg7gk"
Sep 10 21:00:03.807984 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:03.807960    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cd89ac-29817900-qftgk"
Sep 10 21:00:04.589509 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:04.589478    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3fc2b8-29817900-dh9fz"
Sep 10 21:00:04.589641 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:04.589508    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3fc2b8-29817900-dh9fz" event={"ID":"09ae561c-a2b9-4a2c-a95d-9c1125fc2d8e","Type":"ContainerDied","Data":"2e5bf9e70fce523d0c4464090fe4a774d0c63055ac03e093c6cd3dfb9bb5189b"}
Sep 10 21:00:04.589641 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:04.589591    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e5bf9e70fce523d0c4464090fe4a774d0c63055ac03e093c6cd3dfb9bb5189b"
Sep 10 21:00:04.591016 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:04.590988    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bbb40-29817900-22lht" event={"ID":"2fa5c29a-1c7a-46a1-809c-269ca28a4057","Type":"ContainerDied","Data":"340731f6c026f60c9d4671754e330102d63dc92880b4dd126f5643a85b37c9af"}
Sep 10 21:00:04.591016 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:04.591008    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bbb40-29817900-22lht"
Sep 10 21:00:04.591228 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:04.591034    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="340731f6c026f60c9d4671754e330102d63dc92880b4dd126f5643a85b37c9af"
Sep 10 21:00:04.592631 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:04.592602    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817900-cg7gk" event={"ID":"49a8d782-4828-4beb-8614-93cb5b754b4d","Type":"ContainerDied","Data":"3e01e456f07c3323a57cc7550fd3254997786e1008e2a3a0f979b33631f3ddae"}
Sep 10 21:00:04.592804 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:04.592654    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e01e456f07c3323a57cc7550fd3254997786e1008e2a3a0f979b33631f3ddae"
Sep 10 21:00:04.592804 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:04.592732    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817900-cg7gk"
Sep 10 21:00:04.594329 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:04.594303    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cd89ac-29817900-qftgk" event={"ID":"f8bc1141-a53e-4e3c-9abd-85fb096a5c95","Type":"ContainerDied","Data":"c1f93fdd9fd4e420a7f4a0fac34f59c47013be09bbe92d9d7cab13ac9ccbb9cd"}
Sep 10 21:00:04.594451 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:04.594432    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1f93fdd9fd4e420a7f4a0fac34f59c47013be09bbe92d9d7cab13ac9ccbb9cd"
Sep 10 21:00:04.594451 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:04.594380    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cd89ac-29817900-qftgk"
Sep 10 21:00:06.337003 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:06.336973    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf"
Sep 10 21:00:06.337397 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:06.337013    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf"
Sep 10 21:00:06.349005 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:06.348983    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf"
Sep 10 21:00:06.616904 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:06.616834    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf"
Sep 10 21:00:13.606878 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.606838    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp"]
Sep 10 21:00:13.607487 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.607464    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8bc1141-a53e-4e3c-9abd-85fb096a5c95" containerName="cleanup"
Sep 10 21:00:13.607543 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.607493    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8bc1141-a53e-4e3c-9abd-85fb096a5c95" containerName="cleanup"
Sep 10 21:00:13.607543 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.607517    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fa5c29a-1c7a-46a1-809c-269ca28a4057" containerName="cleanup"
Sep 10 21:00:13.607543 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.607525    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fa5c29a-1c7a-46a1-809c-269ca28a4057" containerName="cleanup"
Sep 10 21:00:13.607543 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.607540    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09ae561c-a2b9-4a2c-a95d-9c1125fc2d8e" containerName="cleanup"
Sep 10 21:00:13.607673 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.607549    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="09ae561c-a2b9-4a2c-a95d-9c1125fc2d8e" containerName="cleanup"
Sep 10 21:00:13.607673 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.607573    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49a8d782-4828-4beb-8614-93cb5b754b4d" containerName="cleanup"
Sep 10 21:00:13.607673 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.607582    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="49a8d782-4828-4beb-8614-93cb5b754b4d" containerName="cleanup"
Sep 10 21:00:13.607770 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.607692    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8bc1141-a53e-4e3c-9abd-85fb096a5c95" containerName="cleanup"
Sep 10 21:00:13.607770 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.607706    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="09ae561c-a2b9-4a2c-a95d-9c1125fc2d8e" containerName="cleanup"
Sep 10 21:00:13.607770 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.607715    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fa5c29a-1c7a-46a1-809c-269ca28a4057" containerName="cleanup"
Sep 10 21:00:13.607770 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.607727    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="49a8d782-4828-4beb-8614-93cb5b754b4d" containerName="cleanup"
Sep 10 21:00:13.611353 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.611333    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp"
Sep 10 21:00:13.619935 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.619909    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4fe4f\"/\"default-dockercfg-qf9qn\""
Sep 10 21:00:13.620067 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.619985    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4fe4f\"/\"kube-root-ca.crt\""
Sep 10 21:00:13.621034 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.621008    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4fe4f\"/\"auth-test-model-f4fe4f-kserve-self-signed-certs\""
Sep 10 21:00:13.623006 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.622984    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4fe4f\"/\"openshift-service-ca.crt\""
Sep 10 21:00:13.631702 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.630286    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp"]
Sep 10 21:00:13.686828 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.686801    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-home\") pod \"auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp\" (UID: \"faaacbf2-cd3d-4e06-a5d8-ee5cc257651a\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp"
Sep 10 21:00:13.686938 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.686840    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-model-cache\") pod \"auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp\" (UID: \"faaacbf2-cd3d-4e06-a5d8-ee5cc257651a\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp"
Sep 10 21:00:13.686938 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.686869    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-dshm\") pod \"auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp\" (UID: \"faaacbf2-cd3d-4e06-a5d8-ee5cc257651a\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp"
Sep 10 21:00:13.687051 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.686936    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssk67\" (UniqueName: \"kubernetes.io/projected/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-kube-api-access-ssk67\") pod \"auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp\" (UID: \"faaacbf2-cd3d-4e06-a5d8-ee5cc257651a\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp"
Sep 10 21:00:13.687118 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.687056    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-tls-certs\") pod \"auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp\" (UID: \"faaacbf2-cd3d-4e06-a5d8-ee5cc257651a\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp"
Sep 10 21:00:13.687118 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.687107    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-tmp-dir\") pod \"auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp\" (UID: \"faaacbf2-cd3d-4e06-a5d8-ee5cc257651a\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp"
Sep 10 21:00:13.788060 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.788035    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-model-cache\") pod \"auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp\" (UID: \"faaacbf2-cd3d-4e06-a5d8-ee5cc257651a\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp"
Sep 10 21:00:13.788154 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.788072    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-dshm\") pod \"auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp\" (UID: \"faaacbf2-cd3d-4e06-a5d8-ee5cc257651a\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp"
Sep 10 21:00:13.788154 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.788094    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssk67\" (UniqueName: \"kubernetes.io/projected/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-kube-api-access-ssk67\") pod \"auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp\" (UID: \"faaacbf2-cd3d-4e06-a5d8-ee5cc257651a\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp"
Sep 10 21:00:13.788154 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.788138    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-tls-certs\") pod \"auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp\" (UID: \"faaacbf2-cd3d-4e06-a5d8-ee5cc257651a\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp"
Sep 10 21:00:13.788363 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.788160    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-tmp-dir\") pod \"auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp\" (UID: \"faaacbf2-cd3d-4e06-a5d8-ee5cc257651a\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp"
Sep 10 21:00:13.788363 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.788226    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-home\") pod \"auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp\" (UID: \"faaacbf2-cd3d-4e06-a5d8-ee5cc257651a\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp"
Sep 10 21:00:13.788612 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.788591    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-model-cache\") pod \"auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp\" (UID: \"faaacbf2-cd3d-4e06-a5d8-ee5cc257651a\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp"
Sep 10 21:00:13.788685 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.788637    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-tmp-dir\") pod \"auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp\" (UID: \"faaacbf2-cd3d-4e06-a5d8-ee5cc257651a\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp"
Sep 10 21:00:13.788723 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.788685    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-home\") pod \"auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp\" (UID: \"faaacbf2-cd3d-4e06-a5d8-ee5cc257651a\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp"
Sep 10 21:00:13.790515 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.790497    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-dshm\") pod \"auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp\" (UID: \"faaacbf2-cd3d-4e06-a5d8-ee5cc257651a\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp"
Sep 10 21:00:13.790702 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.790684    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-tls-certs\") pod \"auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp\" (UID: \"faaacbf2-cd3d-4e06-a5d8-ee5cc257651a\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp"
Sep 10 21:00:13.797065 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.797042    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssk67\" (UniqueName: \"kubernetes.io/projected/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-kube-api-access-ssk67\") pod \"auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp\" (UID: \"faaacbf2-cd3d-4e06-a5d8-ee5cc257651a\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp"
Sep 10 21:00:13.921794 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:13.921721    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp"
Sep 10 21:00:14.197748 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:14.197722    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp"]
Sep 10 21:00:14.198836 ip-10-0-141-185 kubenswrapper[2564]: W0910 21:00:14.198809    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfaaacbf2_cd3d_4e06_a5d8_ee5cc257651a.slice/crio-07e206ea78c237799d29570c60f8c245da87ee5fe70a42ee339cbf4bc857b6dc WatchSource:0}: Error finding container 07e206ea78c237799d29570c60f8c245da87ee5fe70a42ee339cbf4bc857b6dc: Status 404 returned error can't find the container with id 07e206ea78c237799d29570c60f8c245da87ee5fe70a42ee339cbf4bc857b6dc
Sep 10 21:00:14.656608 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:14.656574    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp" event={"ID":"faaacbf2-cd3d-4e06-a5d8-ee5cc257651a","Type":"ContainerStarted","Data":"c40be20beead2b1bce40797b2bc29769a69fb605b0f1b880f9ae34ad4fa009f8"}
Sep 10 21:00:14.656970 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:14.656619    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp" event={"ID":"faaacbf2-cd3d-4e06-a5d8-ee5cc257651a","Type":"ContainerStarted","Data":"07e206ea78c237799d29570c60f8c245da87ee5fe70a42ee339cbf4bc857b6dc"}
Sep 10 21:00:14.676294 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:14.676249    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp" podStartSLOduration=1.676233825 podStartE2EDuration="1.676233825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:00:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:14.674145697 +0000 UTC m=+1454.639213894" watchObservedRunningTime="2026-09-10 21:00:14.676233825 +0000 UTC m=+1454.641302067"
Sep 10 21:00:23.922578 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:23.922545    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp"
Sep 10 21:00:23.922976 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:23.922593    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp"
Sep 10 21:00:23.935033 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:23.935009    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp"
Sep 10 21:00:24.721847 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:24.721819    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp"
Sep 10 21:00:46.144141 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:46.144099    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d4d0c56-g4d4z"]
Sep 10 21:00:46.147966 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:46.147942    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d4d0c56-g4d4z"
Sep 10 21:00:46.151875 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:46.151852    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plv2h\""
Sep 10 21:00:46.152078 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:46.152063    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:00:46.159119 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:46.159090    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d4d0c56-g4d4z"]
Sep 10 21:00:46.259015 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:46.258984    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6cebe15-be9e-400e-a108-70d9673eccbf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3d4d0c56-g4d4z\" (UID: \"f6cebe15-be9e-400e-a108-70d9673eccbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d4d0c56-g4d4z"
Sep 10 21:00:46.359503 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:46.359470    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6cebe15-be9e-400e-a108-70d9673eccbf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3d4d0c56-g4d4z\" (UID: \"f6cebe15-be9e-400e-a108-70d9673eccbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d4d0c56-g4d4z"
Sep 10 21:00:46.360082 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:46.360056    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6cebe15-be9e-400e-a108-70d9673eccbf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3d4d0c56-g4d4z\" (UID: \"f6cebe15-be9e-400e-a108-70d9673eccbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d4d0c56-g4d4z"
Sep 10 21:00:46.460307 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:46.460241    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d4d0c56-g4d4z"
Sep 10 21:00:46.739742 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:46.739713    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d4d0c56-g4d4z"]
Sep 10 21:00:46.742105 ip-10-0-141-185 kubenswrapper[2564]: W0910 21:00:46.742079    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6cebe15_be9e_400e_a108_70d9673eccbf.slice/crio-5c0979aef62c173d51c9bf8aeb173a90061a1155fa3c0e23cf51b21aba38dd81 WatchSource:0}: Error finding container 5c0979aef62c173d51c9bf8aeb173a90061a1155fa3c0e23cf51b21aba38dd81: Status 404 returned error can't find the container with id 5c0979aef62c173d51c9bf8aeb173a90061a1155fa3c0e23cf51b21aba38dd81
Sep 10 21:00:46.836648 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:46.836619    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d4d0c56-g4d4z" event={"ID":"f6cebe15-be9e-400e-a108-70d9673eccbf","Type":"ContainerStarted","Data":"5c0979aef62c173d51c9bf8aeb173a90061a1155fa3c0e23cf51b21aba38dd81"}
Sep 10 21:00:47.844520 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:47.844485    2564 generic.go:356] "Generic (PLEG): container finished" podID="f6cebe15-be9e-400e-a108-70d9673eccbf" containerID="1bed497035860851e0c7bebf855dd44c77ad2ff5018728ef9588e37fad20d52a" exitCode=0
Sep 10 21:00:47.844937 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:47.844571    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d4d0c56-g4d4z" event={"ID":"f6cebe15-be9e-400e-a108-70d9673eccbf","Type":"ContainerDied","Data":"1bed497035860851e0c7bebf855dd44c77ad2ff5018728ef9588e37fad20d52a"}
Sep 10 21:00:48.992510 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:48.992487    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d4d0c56-g4d4z"
Sep 10 21:00:49.185283 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:49.185220    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6cebe15-be9e-400e-a108-70d9673eccbf-maas-api-service-ca\") pod \"f6cebe15-be9e-400e-a108-70d9673eccbf\" (UID: \"f6cebe15-be9e-400e-a108-70d9673eccbf\") "
Sep 10 21:00:49.185549 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:49.185527    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6cebe15-be9e-400e-a108-70d9673eccbf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f6cebe15-be9e-400e-a108-70d9673eccbf" (UID: "f6cebe15-be9e-400e-a108-70d9673eccbf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:00:49.286560 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:49.286530    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6cebe15-be9e-400e-a108-70d9673eccbf-maas-api-service-ca\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:00:49.858723 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:49.858695    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d4d0c56-g4d4z"
Sep 10 21:00:49.858723 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:49.858706    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d4d0c56-g4d4z" event={"ID":"f6cebe15-be9e-400e-a108-70d9673eccbf","Type":"ContainerDied","Data":"5c0979aef62c173d51c9bf8aeb173a90061a1155fa3c0e23cf51b21aba38dd81"}
Sep 10 21:00:49.858922 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:00:49.858764    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c0979aef62c173d51c9bf8aeb173a90061a1155fa3c0e23cf51b21aba38dd81"
Sep 10 21:01:02.064013 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:02.063977    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3d4d0c56-578467cfd6-mrh8f"]
Sep 10 21:01:02.064496 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:02.064209    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d4d0c56-578467cfd6-mrh8f" podUID="28122557-8469-4a84-ba98-de2bf3389ae9" containerName="payload-pre-processing" containerID="cri-o://b1dcd233570f7f84d3512d0f6fd9b04ced1aa16d3740a7f47bdd999a929e5a48" gracePeriod=30
Sep 10 21:01:02.313916 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:02.313891    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d4d0c56-578467cfd6-mrh8f"
Sep 10 21:01:02.328757 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:02.328730    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d4d0c56-g4d4z"]
Sep 10 21:01:02.332759 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:02.332737    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d4d0c56-g4d4z"]
Sep 10 21:01:02.385906 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:02.385878    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5psh\" (UniqueName: \"kubernetes.io/projected/28122557-8469-4a84-ba98-de2bf3389ae9-kube-api-access-t5psh\") pod \"28122557-8469-4a84-ba98-de2bf3389ae9\" (UID: \"28122557-8469-4a84-ba98-de2bf3389ae9\") "
Sep 10 21:01:02.386035 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:02.385934    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28122557-8469-4a84-ba98-de2bf3389ae9-plugins-config-volume\") pod \"28122557-8469-4a84-ba98-de2bf3389ae9\" (UID: \"28122557-8469-4a84-ba98-de2bf3389ae9\") "
Sep 10 21:01:02.386386 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:02.386366    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28122557-8469-4a84-ba98-de2bf3389ae9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "28122557-8469-4a84-ba98-de2bf3389ae9" (UID: "28122557-8469-4a84-ba98-de2bf3389ae9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:01:02.388009 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:02.387990    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28122557-8469-4a84-ba98-de2bf3389ae9-kube-api-access-t5psh" (OuterVolumeSpecName: "kube-api-access-t5psh") pod "28122557-8469-4a84-ba98-de2bf3389ae9" (UID: "28122557-8469-4a84-ba98-de2bf3389ae9"). InnerVolumeSpecName "kube-api-access-t5psh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:01:02.487001 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:02.486978    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5psh\" (UniqueName: \"kubernetes.io/projected/28122557-8469-4a84-ba98-de2bf3389ae9-kube-api-access-t5psh\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:01:02.487001 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:02.487000    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28122557-8469-4a84-ba98-de2bf3389ae9-plugins-config-volume\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:01:02.541339 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:02.541316    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6cebe15-be9e-400e-a108-70d9673eccbf" path="/var/lib/kubelet/pods/f6cebe15-be9e-400e-a108-70d9673eccbf/volumes"
Sep 10 21:01:02.938488 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:02.938459    2564 generic.go:356] "Generic (PLEG): container finished" podID="28122557-8469-4a84-ba98-de2bf3389ae9" containerID="b1dcd233570f7f84d3512d0f6fd9b04ced1aa16d3740a7f47bdd999a929e5a48" exitCode=0
Sep 10 21:01:02.938657 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:02.938526    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d4d0c56-578467cfd6-mrh8f"
Sep 10 21:01:02.938657 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:02.938529    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d4d0c56-578467cfd6-mrh8f" event={"ID":"28122557-8469-4a84-ba98-de2bf3389ae9","Type":"ContainerDied","Data":"b1dcd233570f7f84d3512d0f6fd9b04ced1aa16d3740a7f47bdd999a929e5a48"}
Sep 10 21:01:02.938763 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:02.938691    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d4d0c56-578467cfd6-mrh8f" event={"ID":"28122557-8469-4a84-ba98-de2bf3389ae9","Type":"ContainerDied","Data":"029bab400a4ba6109c7cd887411e91b40e4019696dff326f9d4b49bd764b1769"}
Sep 10 21:01:02.939248 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:02.938723    2564 scope.go:117] "RemoveContainer" containerID="b1dcd233570f7f84d3512d0f6fd9b04ced1aa16d3740a7f47bdd999a929e5a48"
Sep 10 21:01:02.948846 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:02.948829    2564 scope.go:117] "RemoveContainer" containerID="b1dcd233570f7f84d3512d0f6fd9b04ced1aa16d3740a7f47bdd999a929e5a48"
Sep 10 21:01:02.949154 ip-10-0-141-185 kubenswrapper[2564]: E0910 21:01:02.949128    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1dcd233570f7f84d3512d0f6fd9b04ced1aa16d3740a7f47bdd999a929e5a48\": container with ID starting with b1dcd233570f7f84d3512d0f6fd9b04ced1aa16d3740a7f47bdd999a929e5a48 not found: ID does not exist" containerID="b1dcd233570f7f84d3512d0f6fd9b04ced1aa16d3740a7f47bdd999a929e5a48"
Sep 10 21:01:02.949290 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:02.949157    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1dcd233570f7f84d3512d0f6fd9b04ced1aa16d3740a7f47bdd999a929e5a48"} err="failed to get container status \"b1dcd233570f7f84d3512d0f6fd9b04ced1aa16d3740a7f47bdd999a929e5a48\": rpc error: code = NotFound desc = could not find container \"b1dcd233570f7f84d3512d0f6fd9b04ced1aa16d3740a7f47bdd999a929e5a48\": container with ID starting with b1dcd233570f7f84d3512d0f6fd9b04ced1aa16d3740a7f47bdd999a929e5a48 not found: ID does not exist"
Sep 10 21:01:02.962045 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:02.962022    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3d4d0c56-578467cfd6-mrh8f"]
Sep 10 21:01:02.964561 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:02.964537    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3d4d0c56-578467cfd6-mrh8f"]
Sep 10 21:01:04.542135 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:04.542100    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28122557-8469-4a84-ba98-de2bf3389ae9" path="/var/lib/kubelet/pods/28122557-8469-4a84-ba98-de2bf3389ae9/volumes"
Sep 10 21:01:10.511439 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:10.511394    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf"]
Sep 10 21:01:25.531917 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:25.531799    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf" podUID="d9233dba-d866-4f6b-a68b-7761bda89097" containerName="main" containerID="cri-o://cf447daa51aaf1054343d12b86d82b701f368474957b9b0759671735a51ea08d" gracePeriod=45
Sep 10 21:01:25.784282 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:25.784227    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf"
Sep 10 21:01:25.871629 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:25.870627    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmj9z\" (UniqueName: \"kubernetes.io/projected/d9233dba-d866-4f6b-a68b-7761bda89097-kube-api-access-wmj9z\") pod \"d9233dba-d866-4f6b-a68b-7761bda89097\" (UID: \"d9233dba-d866-4f6b-a68b-7761bda89097\") "
Sep 10 21:01:25.871629 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:25.870701    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9233dba-d866-4f6b-a68b-7761bda89097-model-cache\") pod \"d9233dba-d866-4f6b-a68b-7761bda89097\" (UID: \"d9233dba-d866-4f6b-a68b-7761bda89097\") "
Sep 10 21:01:25.871629 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:25.870740    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9233dba-d866-4f6b-a68b-7761bda89097-dshm\") pod \"d9233dba-d866-4f6b-a68b-7761bda89097\" (UID: \"d9233dba-d866-4f6b-a68b-7761bda89097\") "
Sep 10 21:01:25.871629 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:25.870780    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9233dba-d866-4f6b-a68b-7761bda89097-tls-certs\") pod \"d9233dba-d866-4f6b-a68b-7761bda89097\" (UID: \"d9233dba-d866-4f6b-a68b-7761bda89097\") "
Sep 10 21:01:25.871629 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:25.870844    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9233dba-d866-4f6b-a68b-7761bda89097-tmp-dir\") pod \"d9233dba-d866-4f6b-a68b-7761bda89097\" (UID: \"d9233dba-d866-4f6b-a68b-7761bda89097\") "
Sep 10 21:01:25.871629 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:25.870878    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9233dba-d866-4f6b-a68b-7761bda89097-home\") pod \"d9233dba-d866-4f6b-a68b-7761bda89097\" (UID: \"d9233dba-d866-4f6b-a68b-7761bda89097\") "
Sep 10 21:01:25.872126 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:25.872094    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9233dba-d866-4f6b-a68b-7761bda89097-home" (OuterVolumeSpecName: "home") pod "d9233dba-d866-4f6b-a68b-7761bda89097" (UID: "d9233dba-d866-4f6b-a68b-7761bda89097"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:01:25.874332 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:25.874301    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9233dba-d866-4f6b-a68b-7761bda89097-model-cache" (OuterVolumeSpecName: "model-cache") pod "d9233dba-d866-4f6b-a68b-7761bda89097" (UID: "d9233dba-d866-4f6b-a68b-7761bda89097"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:01:25.874434 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:25.874348    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9233dba-d866-4f6b-a68b-7761bda89097-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d9233dba-d866-4f6b-a68b-7761bda89097" (UID: "d9233dba-d866-4f6b-a68b-7761bda89097"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:01:25.874775 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:25.874750    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9233dba-d866-4f6b-a68b-7761bda89097-kube-api-access-wmj9z" (OuterVolumeSpecName: "kube-api-access-wmj9z") pod "d9233dba-d866-4f6b-a68b-7761bda89097" (UID: "d9233dba-d866-4f6b-a68b-7761bda89097"). InnerVolumeSpecName "kube-api-access-wmj9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:01:25.875541 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:25.875519    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9233dba-d866-4f6b-a68b-7761bda89097-dshm" (OuterVolumeSpecName: "dshm") pod "d9233dba-d866-4f6b-a68b-7761bda89097" (UID: "d9233dba-d866-4f6b-a68b-7761bda89097"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:01:25.876215 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:25.876192    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9233dba-d866-4f6b-a68b-7761bda89097-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d9233dba-d866-4f6b-a68b-7761bda89097" (UID: "d9233dba-d866-4f6b-a68b-7761bda89097"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:01:25.972014 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:25.971992    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmj9z\" (UniqueName: \"kubernetes.io/projected/d9233dba-d866-4f6b-a68b-7761bda89097-kube-api-access-wmj9z\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:01:25.972014 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:25.972012    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9233dba-d866-4f6b-a68b-7761bda89097-model-cache\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:01:25.972145 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:25.972021    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9233dba-d866-4f6b-a68b-7761bda89097-dshm\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:01:25.972145 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:25.972029    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9233dba-d866-4f6b-a68b-7761bda89097-tls-certs\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:01:25.972145 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:25.972038    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9233dba-d866-4f6b-a68b-7761bda89097-tmp-dir\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:01:25.972145 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:25.972045    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9233dba-d866-4f6b-a68b-7761bda89097-home\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:01:26.077605 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:26.077578    2564 generic.go:356] "Generic (PLEG): container finished" podID="d9233dba-d866-4f6b-a68b-7761bda89097" containerID="cf447daa51aaf1054343d12b86d82b701f368474957b9b0759671735a51ea08d" exitCode=0
Sep 10 21:01:26.077712 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:26.077621    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf" event={"ID":"d9233dba-d866-4f6b-a68b-7761bda89097","Type":"ContainerDied","Data":"cf447daa51aaf1054343d12b86d82b701f368474957b9b0759671735a51ea08d"}
Sep 10 21:01:26.077712 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:26.077634    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf"
Sep 10 21:01:26.077712 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:26.077660    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf" event={"ID":"d9233dba-d866-4f6b-a68b-7761bda89097","Type":"ContainerDied","Data":"1217693551f3093f42568a79aa37f3a29a0c5b0fdc8c6fa6af2166a765986ffd"}
Sep 10 21:01:26.077712 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:26.077682    2564 scope.go:117] "RemoveContainer" containerID="cf447daa51aaf1054343d12b86d82b701f368474957b9b0759671735a51ea08d"
Sep 10 21:01:26.087086 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:26.087069    2564 scope.go:117] "RemoveContainer" containerID="cf447daa51aaf1054343d12b86d82b701f368474957b9b0759671735a51ea08d"
Sep 10 21:01:26.087369 ip-10-0-141-185 kubenswrapper[2564]: E0910 21:01:26.087350    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf447daa51aaf1054343d12b86d82b701f368474957b9b0759671735a51ea08d\": container with ID starting with cf447daa51aaf1054343d12b86d82b701f368474957b9b0759671735a51ea08d not found: ID does not exist" containerID="cf447daa51aaf1054343d12b86d82b701f368474957b9b0759671735a51ea08d"
Sep 10 21:01:26.087425 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:26.087376    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf447daa51aaf1054343d12b86d82b701f368474957b9b0759671735a51ea08d"} err="failed to get container status \"cf447daa51aaf1054343d12b86d82b701f368474957b9b0759671735a51ea08d\": rpc error: code = NotFound desc = could not find container \"cf447daa51aaf1054343d12b86d82b701f368474957b9b0759671735a51ea08d\": container with ID starting with cf447daa51aaf1054343d12b86d82b701f368474957b9b0759671735a51ea08d not found: ID does not exist"
Sep 10 21:01:26.099855 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:26.099835    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf"]
Sep 10 21:01:26.104232 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:26.104212    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-3d4d0c56/e2e-lifecycle-model-3d4d0c56-kserve-84f85f455f-wlxpf"]
Sep 10 21:01:26.542127 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:01:26.542046    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9233dba-d866-4f6b-a68b-7761bda89097" path="/var/lib/kubelet/pods/d9233dba-d866-4f6b-a68b-7761bda89097/volumes"
Sep 10 21:02:43.481723 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:43.481686    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls"]
Sep 10 21:02:44.514676 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:44.514643    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp"]
Sep 10 21:02:57.185687 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:57.185607    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d66961-589f585555-zdn8k"]
Sep 10 21:02:57.186154 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:57.186134    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9233dba-d866-4f6b-a68b-7761bda89097" containerName="main"
Sep 10 21:02:57.186258 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:57.186157    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9233dba-d866-4f6b-a68b-7761bda89097" containerName="main"
Sep 10 21:02:57.186258 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:57.186209    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28122557-8469-4a84-ba98-de2bf3389ae9" containerName="payload-pre-processing"
Sep 10 21:02:57.186258 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:57.186219    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="28122557-8469-4a84-ba98-de2bf3389ae9" containerName="payload-pre-processing"
Sep 10 21:02:57.186258 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:57.186239    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6cebe15-be9e-400e-a108-70d9673eccbf" containerName="revoke-keys"
Sep 10 21:02:57.186258 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:57.186249    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6cebe15-be9e-400e-a108-70d9673eccbf" containerName="revoke-keys"
Sep 10 21:02:57.186508 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:57.186352    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9233dba-d866-4f6b-a68b-7761bda89097" containerName="main"
Sep 10 21:02:57.186508 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:57.186364    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6cebe15-be9e-400e-a108-70d9673eccbf" containerName="revoke-keys"
Sep 10 21:02:57.186508 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:57.186378    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="28122557-8469-4a84-ba98-de2bf3389ae9" containerName="payload-pre-processing"
Sep 10 21:02:57.189532 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:57.189507    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d66961-589f585555-zdn8k"
Sep 10 21:02:57.192151 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:57.192129    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d66961-dockercfg-jw6m2\""
Sep 10 21:02:57.192299 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:57.192137    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d66961\""
Sep 10 21:02:57.201485 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:57.201462    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d66961-589f585555-zdn8k"]
Sep 10 21:02:57.322425 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:57.322394    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d56c672e-5ce0-484f-9bb5-ff101fc00463-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d66961-589f585555-zdn8k\" (UID: \"d56c672e-5ce0-484f-9bb5-ff101fc00463\") " pod="openshift-ingress/payload-processing-e2e-api-a-d66961-589f585555-zdn8k"
Sep 10 21:02:57.322586 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:57.322442    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvzfr\" (UniqueName: \"kubernetes.io/projected/d56c672e-5ce0-484f-9bb5-ff101fc00463-kube-api-access-vvzfr\") pod \"payload-processing-e2e-api-a-d66961-589f585555-zdn8k\" (UID: \"d56c672e-5ce0-484f-9bb5-ff101fc00463\") " pod="openshift-ingress/payload-processing-e2e-api-a-d66961-589f585555-zdn8k"
Sep 10 21:02:57.422961 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:57.422928    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvzfr\" (UniqueName: \"kubernetes.io/projected/d56c672e-5ce0-484f-9bb5-ff101fc00463-kube-api-access-vvzfr\") pod \"payload-processing-e2e-api-a-d66961-589f585555-zdn8k\" (UID: \"d56c672e-5ce0-484f-9bb5-ff101fc00463\") " pod="openshift-ingress/payload-processing-e2e-api-a-d66961-589f585555-zdn8k"
Sep 10 21:02:57.423224 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:57.423044    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d56c672e-5ce0-484f-9bb5-ff101fc00463-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d66961-589f585555-zdn8k\" (UID: \"d56c672e-5ce0-484f-9bb5-ff101fc00463\") " pod="openshift-ingress/payload-processing-e2e-api-a-d66961-589f585555-zdn8k"
Sep 10 21:02:57.423878 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:57.423848    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d56c672e-5ce0-484f-9bb5-ff101fc00463-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d66961-589f585555-zdn8k\" (UID: \"d56c672e-5ce0-484f-9bb5-ff101fc00463\") " pod="openshift-ingress/payload-processing-e2e-api-a-d66961-589f585555-zdn8k"
Sep 10 21:02:57.438363 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:57.438306    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvzfr\" (UniqueName: \"kubernetes.io/projected/d56c672e-5ce0-484f-9bb5-ff101fc00463-kube-api-access-vvzfr\") pod \"payload-processing-e2e-api-a-d66961-589f585555-zdn8k\" (UID: \"d56c672e-5ce0-484f-9bb5-ff101fc00463\") " pod="openshift-ingress/payload-processing-e2e-api-a-d66961-589f585555-zdn8k"
Sep 10 21:02:57.499966 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:57.499942    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d66961-589f585555-zdn8k"
Sep 10 21:02:57.777322 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:57.777293    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d66961-589f585555-zdn8k"]
Sep 10 21:02:57.779879 ip-10-0-141-185 kubenswrapper[2564]: W0910 21:02:57.779850    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56c672e_5ce0_484f_9bb5_ff101fc00463.slice/crio-6bc1ca01ef7109155ea7348123fa0eddc7b1fb79c19f7488221821e246272a99 WatchSource:0}: Error finding container 6bc1ca01ef7109155ea7348123fa0eddc7b1fb79c19f7488221821e246272a99: Status 404 returned error can't find the container with id 6bc1ca01ef7109155ea7348123fa0eddc7b1fb79c19f7488221821e246272a99
Sep 10 21:02:58.501037 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:58.500995    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls" podUID="2342ad2a-819f-4c84-a90a-d26346a881f7" containerName="main" containerID="cri-o://ad87c4c51cc4091e40f5e3aad74944d29eecdf1e749830d09551e62fb193d35c" gracePeriod=45
Sep 10 21:02:58.626224 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:58.626163    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d66961-589f585555-zdn8k" event={"ID":"d56c672e-5ce0-484f-9bb5-ff101fc00463","Type":"ContainerStarted","Data":"8515e6f23b8a6e11c1d4d262054a346ca85cb364415a586f12ca6f005250c501"}
Sep 10 21:02:58.626356 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:58.626320    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-d66961-589f585555-zdn8k"
Sep 10 21:02:58.626356 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:58.626352    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d66961-589f585555-zdn8k" event={"ID":"d56c672e-5ce0-484f-9bb5-ff101fc00463","Type":"ContainerStarted","Data":"6bc1ca01ef7109155ea7348123fa0eddc7b1fb79c19f7488221821e246272a99"}
Sep 10 21:02:58.628104 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:58.628071    2564 generic.go:356] "Generic (PLEG): container finished" podID="2342ad2a-819f-4c84-a90a-d26346a881f7" containerID="ad87c4c51cc4091e40f5e3aad74944d29eecdf1e749830d09551e62fb193d35c" exitCode=0
Sep 10 21:02:58.628226 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:58.628132    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls" event={"ID":"2342ad2a-819f-4c84-a90a-d26346a881f7","Type":"ContainerDied","Data":"ad87c4c51cc4091e40f5e3aad74944d29eecdf1e749830d09551e62fb193d35c"}
Sep 10 21:02:58.646027 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:58.645980    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-d66961-589f585555-zdn8k" podStartSLOduration=1.645964178 podStartE2EDuration="1.645964178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:02:58.64326894 +0000 UTC m=+1618.608337137" watchObservedRunningTime="2026-09-10 21:02:58.645964178 +0000 UTC m=+1618.611032373"
Sep 10 21:02:58.760890 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:58.760835    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls"
Sep 10 21:02:58.937207 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:58.937159    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2342ad2a-819f-4c84-a90a-d26346a881f7-tls-certs\") pod \"2342ad2a-819f-4c84-a90a-d26346a881f7\" (UID: \"2342ad2a-819f-4c84-a90a-d26346a881f7\") "
Sep 10 21:02:58.937393 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:58.937248    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2342ad2a-819f-4c84-a90a-d26346a881f7-dshm\") pod \"2342ad2a-819f-4c84-a90a-d26346a881f7\" (UID: \"2342ad2a-819f-4c84-a90a-d26346a881f7\") "
Sep 10 21:02:58.937393 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:58.937315    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2342ad2a-819f-4c84-a90a-d26346a881f7-home\") pod \"2342ad2a-819f-4c84-a90a-d26346a881f7\" (UID: \"2342ad2a-819f-4c84-a90a-d26346a881f7\") "
Sep 10 21:02:58.937393 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:58.937352    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgj87\" (UniqueName: \"kubernetes.io/projected/2342ad2a-819f-4c84-a90a-d26346a881f7-kube-api-access-bgj87\") pod \"2342ad2a-819f-4c84-a90a-d26346a881f7\" (UID: \"2342ad2a-819f-4c84-a90a-d26346a881f7\") "
Sep 10 21:02:58.937570 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:58.937480    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2342ad2a-819f-4c84-a90a-d26346a881f7-model-cache\") pod \"2342ad2a-819f-4c84-a90a-d26346a881f7\" (UID: \"2342ad2a-819f-4c84-a90a-d26346a881f7\") "
Sep 10 21:02:58.937570 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:58.937525    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2342ad2a-819f-4c84-a90a-d26346a881f7-tmp-dir\") pod \"2342ad2a-819f-4c84-a90a-d26346a881f7\" (UID: \"2342ad2a-819f-4c84-a90a-d26346a881f7\") "
Sep 10 21:02:58.937764 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:58.937736    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2342ad2a-819f-4c84-a90a-d26346a881f7-home" (OuterVolumeSpecName: "home") pod "2342ad2a-819f-4c84-a90a-d26346a881f7" (UID: "2342ad2a-819f-4c84-a90a-d26346a881f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:02:58.937905 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:58.937848    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2342ad2a-819f-4c84-a90a-d26346a881f7-home\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:02:58.938136 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:58.938105    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2342ad2a-819f-4c84-a90a-d26346a881f7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2342ad2a-819f-4c84-a90a-d26346a881f7" (UID: "2342ad2a-819f-4c84-a90a-d26346a881f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:02:58.938659 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:58.938623    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2342ad2a-819f-4c84-a90a-d26346a881f7-model-cache" (OuterVolumeSpecName: "model-cache") pod "2342ad2a-819f-4c84-a90a-d26346a881f7" (UID: "2342ad2a-819f-4c84-a90a-d26346a881f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:02:58.940648 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:58.940626    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2342ad2a-819f-4c84-a90a-d26346a881f7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2342ad2a-819f-4c84-a90a-d26346a881f7" (UID: "2342ad2a-819f-4c84-a90a-d26346a881f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:02:58.940737 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:58.940719    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2342ad2a-819f-4c84-a90a-d26346a881f7-kube-api-access-bgj87" (OuterVolumeSpecName: "kube-api-access-bgj87") pod "2342ad2a-819f-4c84-a90a-d26346a881f7" (UID: "2342ad2a-819f-4c84-a90a-d26346a881f7"). InnerVolumeSpecName "kube-api-access-bgj87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:02:58.941070 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:58.941047    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2342ad2a-819f-4c84-a90a-d26346a881f7-dshm" (OuterVolumeSpecName: "dshm") pod "2342ad2a-819f-4c84-a90a-d26346a881f7" (UID: "2342ad2a-819f-4c84-a90a-d26346a881f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:02:59.039233 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:59.039157    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2342ad2a-819f-4c84-a90a-d26346a881f7-tmp-dir\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:02:59.039233 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:59.039203    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2342ad2a-819f-4c84-a90a-d26346a881f7-tls-certs\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:02:59.039233 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:59.039212    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2342ad2a-819f-4c84-a90a-d26346a881f7-dshm\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:02:59.039233 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:59.039220    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgj87\" (UniqueName: \"kubernetes.io/projected/2342ad2a-819f-4c84-a90a-d26346a881f7-kube-api-access-bgj87\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:02:59.039233 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:59.039229    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2342ad2a-819f-4c84-a90a-d26346a881f7-model-cache\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:02:59.534202 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:59.534146    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp" podUID="faaacbf2-cd3d-4e06-a5d8-ee5cc257651a" containerName="main" containerID="cri-o://c40be20beead2b1bce40797b2bc29769a69fb605b0f1b880f9ae34ad4fa009f8" gracePeriod=45
Sep 10 21:02:59.636284 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:59.636262    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls"
Sep 10 21:02:59.636384 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:59.636304    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls" event={"ID":"2342ad2a-819f-4c84-a90a-d26346a881f7","Type":"ContainerDied","Data":"07293d6ab7b28cfa191008e91e658424de7dbf13a720455bb296f68f74739eba"}
Sep 10 21:02:59.636384 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:59.636359    2564 scope.go:117] "RemoveContainer" containerID="ad87c4c51cc4091e40f5e3aad74944d29eecdf1e749830d09551e62fb193d35c"
Sep 10 21:02:59.658865 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:59.658841    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls"]
Sep 10 21:02:59.662929 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:59.662908    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3fc2b8/auth-test-model-3fc2b8-kserve-f96f5f95-t4bls"]
Sep 10 21:02:59.786213 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:59.786153    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp"
Sep 10 21:02:59.947146 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:59.947120    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssk67\" (UniqueName: \"kubernetes.io/projected/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-kube-api-access-ssk67\") pod \"faaacbf2-cd3d-4e06-a5d8-ee5cc257651a\" (UID: \"faaacbf2-cd3d-4e06-a5d8-ee5cc257651a\") "
Sep 10 21:02:59.947289 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:59.947186    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-home\") pod \"faaacbf2-cd3d-4e06-a5d8-ee5cc257651a\" (UID: \"faaacbf2-cd3d-4e06-a5d8-ee5cc257651a\") "
Sep 10 21:02:59.947289 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:59.947206    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-tmp-dir\") pod \"faaacbf2-cd3d-4e06-a5d8-ee5cc257651a\" (UID: \"faaacbf2-cd3d-4e06-a5d8-ee5cc257651a\") "
Sep 10 21:02:59.947289 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:59.947246    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-model-cache\") pod \"faaacbf2-cd3d-4e06-a5d8-ee5cc257651a\" (UID: \"faaacbf2-cd3d-4e06-a5d8-ee5cc257651a\") "
Sep 10 21:02:59.947289 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:59.947266    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-tls-certs\") pod \"faaacbf2-cd3d-4e06-a5d8-ee5cc257651a\" (UID: \"faaacbf2-cd3d-4e06-a5d8-ee5cc257651a\") "
Sep 10 21:02:59.947289 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:59.947283    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-dshm\") pod \"faaacbf2-cd3d-4e06-a5d8-ee5cc257651a\" (UID: \"faaacbf2-cd3d-4e06-a5d8-ee5cc257651a\") "
Sep 10 21:02:59.947536 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:59.947502    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "faaacbf2-cd3d-4e06-a5d8-ee5cc257651a" (UID: "faaacbf2-cd3d-4e06-a5d8-ee5cc257651a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:02:59.947536 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:59.947516    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-model-cache" (OuterVolumeSpecName: "model-cache") pod "faaacbf2-cd3d-4e06-a5d8-ee5cc257651a" (UID: "faaacbf2-cd3d-4e06-a5d8-ee5cc257651a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:02:59.947646 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:59.947530    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-home" (OuterVolumeSpecName: "home") pod "faaacbf2-cd3d-4e06-a5d8-ee5cc257651a" (UID: "faaacbf2-cd3d-4e06-a5d8-ee5cc257651a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:02:59.949385 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:59.949356    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-kube-api-access-ssk67" (OuterVolumeSpecName: "kube-api-access-ssk67") pod "faaacbf2-cd3d-4e06-a5d8-ee5cc257651a" (UID: "faaacbf2-cd3d-4e06-a5d8-ee5cc257651a"). InnerVolumeSpecName "kube-api-access-ssk67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:02:59.949512 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:59.949485    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "faaacbf2-cd3d-4e06-a5d8-ee5cc257651a" (UID: "faaacbf2-cd3d-4e06-a5d8-ee5cc257651a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:02:59.949570 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:02:59.949489    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-dshm" (OuterVolumeSpecName: "dshm") pod "faaacbf2-cd3d-4e06-a5d8-ee5cc257651a" (UID: "faaacbf2-cd3d-4e06-a5d8-ee5cc257651a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:03:00.048278 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:00.048228    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-tls-certs\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:03:00.048278 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:00.048248    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-dshm\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:03:00.048278 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:00.048257    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssk67\" (UniqueName: \"kubernetes.io/projected/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-kube-api-access-ssk67\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:03:00.048278 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:00.048267    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-home\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:03:00.048278 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:00.048276    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-tmp-dir\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:03:00.048496 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:00.048284    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a-model-cache\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:03:00.542201 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:00.542152    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2342ad2a-819f-4c84-a90a-d26346a881f7" path="/var/lib/kubelet/pods/2342ad2a-819f-4c84-a90a-d26346a881f7/volumes"
Sep 10 21:03:00.642951 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:00.642924    2564 generic.go:356] "Generic (PLEG): container finished" podID="faaacbf2-cd3d-4e06-a5d8-ee5cc257651a" containerID="c40be20beead2b1bce40797b2bc29769a69fb605b0f1b880f9ae34ad4fa009f8" exitCode=0
Sep 10 21:03:00.643061 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:00.642961    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp"
Sep 10 21:03:00.643061 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:00.643007    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp" event={"ID":"faaacbf2-cd3d-4e06-a5d8-ee5cc257651a","Type":"ContainerDied","Data":"c40be20beead2b1bce40797b2bc29769a69fb605b0f1b880f9ae34ad4fa009f8"}
Sep 10 21:03:00.643061 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:00.643051    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp" event={"ID":"faaacbf2-cd3d-4e06-a5d8-ee5cc257651a","Type":"ContainerDied","Data":"07e206ea78c237799d29570c60f8c245da87ee5fe70a42ee339cbf4bc857b6dc"}
Sep 10 21:03:00.643256 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:00.643070    2564 scope.go:117] "RemoveContainer" containerID="c40be20beead2b1bce40797b2bc29769a69fb605b0f1b880f9ae34ad4fa009f8"
Sep 10 21:03:00.651817 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:00.651802    2564 scope.go:117] "RemoveContainer" containerID="c40be20beead2b1bce40797b2bc29769a69fb605b0f1b880f9ae34ad4fa009f8"
Sep 10 21:03:00.652069 ip-10-0-141-185 kubenswrapper[2564]: E0910 21:03:00.652051    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c40be20beead2b1bce40797b2bc29769a69fb605b0f1b880f9ae34ad4fa009f8\": container with ID starting with c40be20beead2b1bce40797b2bc29769a69fb605b0f1b880f9ae34ad4fa009f8 not found: ID does not exist" containerID="c40be20beead2b1bce40797b2bc29769a69fb605b0f1b880f9ae34ad4fa009f8"
Sep 10 21:03:00.652115 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:00.652077    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c40be20beead2b1bce40797b2bc29769a69fb605b0f1b880f9ae34ad4fa009f8"} err="failed to get container status \"c40be20beead2b1bce40797b2bc29769a69fb605b0f1b880f9ae34ad4fa009f8\": rpc error: code = NotFound desc = could not find container \"c40be20beead2b1bce40797b2bc29769a69fb605b0f1b880f9ae34ad4fa009f8\": container with ID starting with c40be20beead2b1bce40797b2bc29769a69fb605b0f1b880f9ae34ad4fa009f8 not found: ID does not exist"
Sep 10 21:03:00.661074 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:00.661054    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp"]
Sep 10 21:03:00.666838 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:00.666820    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4fe4f/auth-test-model-f4fe4f-kserve-c5d6d5b75-sxsdp"]
Sep 10 21:03:00.679095 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:00.679076    2564 scope.go:117] "RemoveContainer" containerID="db80012f62c206e66ddc909d84157bc948cd3e62a4d49e74198f00b6b201a158"
Sep 10 21:03:02.542527 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:02.542491    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="faaacbf2-cd3d-4e06-a5d8-ee5cc257651a" path="/var/lib/kubelet/pods/faaacbf2-cd3d-4e06-a5d8-ee5cc257651a/volumes"
Sep 10 21:03:09.638544 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:09.638510    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-d66961-589f585555-zdn8k"
Sep 10 21:03:25.944900 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:25.944864    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-849fe6-64bf9fdb9c-qm86f"]
Sep 10 21:03:25.945351 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:25.945249    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faaacbf2-cd3d-4e06-a5d8-ee5cc257651a" containerName="main"
Sep 10 21:03:25.945351 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:25.945261    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="faaacbf2-cd3d-4e06-a5d8-ee5cc257651a" containerName="main"
Sep 10 21:03:25.945351 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:25.945284    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2342ad2a-819f-4c84-a90a-d26346a881f7" containerName="main"
Sep 10 21:03:25.945351 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:25.945289    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2342ad2a-819f-4c84-a90a-d26346a881f7" containerName="main"
Sep 10 21:03:25.945351 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:25.945350    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2342ad2a-819f-4c84-a90a-d26346a881f7" containerName="main"
Sep 10 21:03:25.945518 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:25.945358    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="faaacbf2-cd3d-4e06-a5d8-ee5cc257651a" containerName="main"
Sep 10 21:03:25.948628 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:25.948612    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-849fe6-64bf9fdb9c-qm86f"
Sep 10 21:03:25.951011 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:25.950984    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-849fe6\""
Sep 10 21:03:25.951143 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:25.950984    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-849fe6-dockercfg-x8jxg\""
Sep 10 21:03:25.959628 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:25.959605    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-849fe6-64bf9fdb9c-qm86f"]
Sep 10 21:03:26.054054 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:26.054011    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m2f5\" (UniqueName: \"kubernetes.io/projected/c99a7daa-cd4b-476c-940f-1637f7f80088-kube-api-access-4m2f5\") pod \"payload-processing-e2e-api-b-849fe6-64bf9fdb9c-qm86f\" (UID: \"c99a7daa-cd4b-476c-940f-1637f7f80088\") " pod="openshift-ingress/payload-processing-e2e-api-b-849fe6-64bf9fdb9c-qm86f"
Sep 10 21:03:26.054245 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:26.054214    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c99a7daa-cd4b-476c-940f-1637f7f80088-plugins-config-volume\") pod \"payload-processing-e2e-api-b-849fe6-64bf9fdb9c-qm86f\" (UID: \"c99a7daa-cd4b-476c-940f-1637f7f80088\") " pod="openshift-ingress/payload-processing-e2e-api-b-849fe6-64bf9fdb9c-qm86f"
Sep 10 21:03:26.155403 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:26.155370    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c99a7daa-cd4b-476c-940f-1637f7f80088-plugins-config-volume\") pod \"payload-processing-e2e-api-b-849fe6-64bf9fdb9c-qm86f\" (UID: \"c99a7daa-cd4b-476c-940f-1637f7f80088\") " pod="openshift-ingress/payload-processing-e2e-api-b-849fe6-64bf9fdb9c-qm86f"
Sep 10 21:03:26.155561 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:26.155415    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m2f5\" (UniqueName: \"kubernetes.io/projected/c99a7daa-cd4b-476c-940f-1637f7f80088-kube-api-access-4m2f5\") pod \"payload-processing-e2e-api-b-849fe6-64bf9fdb9c-qm86f\" (UID: \"c99a7daa-cd4b-476c-940f-1637f7f80088\") " pod="openshift-ingress/payload-processing-e2e-api-b-849fe6-64bf9fdb9c-qm86f"
Sep 10 21:03:26.156062 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:26.156042    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c99a7daa-cd4b-476c-940f-1637f7f80088-plugins-config-volume\") pod \"payload-processing-e2e-api-b-849fe6-64bf9fdb9c-qm86f\" (UID: \"c99a7daa-cd4b-476c-940f-1637f7f80088\") " pod="openshift-ingress/payload-processing-e2e-api-b-849fe6-64bf9fdb9c-qm86f"
Sep 10 21:03:26.165138 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:26.165112    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m2f5\" (UniqueName: \"kubernetes.io/projected/c99a7daa-cd4b-476c-940f-1637f7f80088-kube-api-access-4m2f5\") pod \"payload-processing-e2e-api-b-849fe6-64bf9fdb9c-qm86f\" (UID: \"c99a7daa-cd4b-476c-940f-1637f7f80088\") " pod="openshift-ingress/payload-processing-e2e-api-b-849fe6-64bf9fdb9c-qm86f"
Sep 10 21:03:26.260231 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:26.260145    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-849fe6-64bf9fdb9c-qm86f"
Sep 10 21:03:26.543311 ip-10-0-141-185 kubenswrapper[2564]: W0910 21:03:26.542920    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc99a7daa_cd4b_476c_940f_1637f7f80088.slice/crio-e37f96754454687f893d1b135821e16941a515fc89e40041a538e61d00835728 WatchSource:0}: Error finding container e37f96754454687f893d1b135821e16941a515fc89e40041a538e61d00835728: Status 404 returned error can't find the container with id e37f96754454687f893d1b135821e16941a515fc89e40041a538e61d00835728
Sep 10 21:03:26.547499 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:26.547476    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-849fe6-64bf9fdb9c-qm86f"]
Sep 10 21:03:26.795910 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:26.795824    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-849fe6-64bf9fdb9c-qm86f" event={"ID":"c99a7daa-cd4b-476c-940f-1637f7f80088","Type":"ContainerStarted","Data":"cf81b9c077f296bdab7c42004d1b7f496d384bca886f992cfe814773a186e43f"}
Sep 10 21:03:26.795910 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:26.795888    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-849fe6-64bf9fdb9c-qm86f" event={"ID":"c99a7daa-cd4b-476c-940f-1637f7f80088","Type":"ContainerStarted","Data":"e37f96754454687f893d1b135821e16941a515fc89e40041a538e61d00835728"}
Sep 10 21:03:26.796083 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:26.795920    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-849fe6-64bf9fdb9c-qm86f"
Sep 10 21:03:26.814480 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:26.814435    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-849fe6-64bf9fdb9c-qm86f" podStartSLOduration=1.814422413 podStartE2EDuration="1.814422413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:03:26.812954804 +0000 UTC m=+1646.778023000" watchObservedRunningTime="2026-09-10 21:03:26.814422413 +0000 UTC m=+1646.779490660"
Sep 10 21:03:35.610433 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:35.610399    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw"]
Sep 10 21:03:35.614094 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:35.614069    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw"
Sep 10 21:03:35.616953 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:35.616925    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4fe4f\"/\"default-dockercfg-qf9qn\""
Sep 10 21:03:35.616953 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:35.616947    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4fe4f\"/\"kube-root-ca.crt\""
Sep 10 21:03:35.617146 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:35.617036    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4fe4f\"/\"openshift-service-ca.crt\""
Sep 10 21:03:35.617146 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:35.617036    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4fe4f\"/\"sub-test-model-f4fe4f-kserve-self-signed-certs\""
Sep 10 21:03:35.622910 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:35.622894    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw"]
Sep 10 21:03:35.731111 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:35.731082    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e7c676d-b4b1-4aab-bce5-4966103f16a1-dshm\") pod \"sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw\" (UID: \"4e7c676d-b4b1-4aab-bce5-4966103f16a1\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw"
Sep 10 21:03:35.731279 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:35.731126    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e7c676d-b4b1-4aab-bce5-4966103f16a1-tmp-dir\") pod \"sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw\" (UID: \"4e7c676d-b4b1-4aab-bce5-4966103f16a1\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw"
Sep 10 21:03:35.731279 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:35.731150    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e7c676d-b4b1-4aab-bce5-4966103f16a1-model-cache\") pod \"sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw\" (UID: \"4e7c676d-b4b1-4aab-bce5-4966103f16a1\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw"
Sep 10 21:03:35.731279 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:35.731217    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e7c676d-b4b1-4aab-bce5-4966103f16a1-home\") pod \"sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw\" (UID: \"4e7c676d-b4b1-4aab-bce5-4966103f16a1\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw"
Sep 10 21:03:35.731279 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:35.731241    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e7c676d-b4b1-4aab-bce5-4966103f16a1-tls-certs\") pod \"sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw\" (UID: \"4e7c676d-b4b1-4aab-bce5-4966103f16a1\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw"
Sep 10 21:03:35.731431 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:35.731289    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd6rx\" (UniqueName: \"kubernetes.io/projected/4e7c676d-b4b1-4aab-bce5-4966103f16a1-kube-api-access-fd6rx\") pod \"sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw\" (UID: \"4e7c676d-b4b1-4aab-bce5-4966103f16a1\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw"
Sep 10 21:03:35.832353 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:35.832316    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e7c676d-b4b1-4aab-bce5-4966103f16a1-home\") pod \"sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw\" (UID: \"4e7c676d-b4b1-4aab-bce5-4966103f16a1\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw"
Sep 10 21:03:35.832502 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:35.832375    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e7c676d-b4b1-4aab-bce5-4966103f16a1-tls-certs\") pod \"sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw\" (UID: \"4e7c676d-b4b1-4aab-bce5-4966103f16a1\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw"
Sep 10 21:03:35.832502 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:35.832406    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd6rx\" (UniqueName: \"kubernetes.io/projected/4e7c676d-b4b1-4aab-bce5-4966103f16a1-kube-api-access-fd6rx\") pod \"sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw\" (UID: \"4e7c676d-b4b1-4aab-bce5-4966103f16a1\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw"
Sep 10 21:03:35.832502 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:35.832474    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e7c676d-b4b1-4aab-bce5-4966103f16a1-dshm\") pod \"sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw\" (UID: \"4e7c676d-b4b1-4aab-bce5-4966103f16a1\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw"
Sep 10 21:03:35.832689 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:35.832519    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e7c676d-b4b1-4aab-bce5-4966103f16a1-tmp-dir\") pod \"sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw\" (UID: \"4e7c676d-b4b1-4aab-bce5-4966103f16a1\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw"
Sep 10 21:03:35.832689 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:35.832540    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e7c676d-b4b1-4aab-bce5-4966103f16a1-model-cache\") pod \"sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw\" (UID: \"4e7c676d-b4b1-4aab-bce5-4966103f16a1\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw"
Sep 10 21:03:35.832807 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:35.832750    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e7c676d-b4b1-4aab-bce5-4966103f16a1-home\") pod \"sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw\" (UID: \"4e7c676d-b4b1-4aab-bce5-4966103f16a1\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw"
Sep 10 21:03:35.832869 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:35.832828    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e7c676d-b4b1-4aab-bce5-4966103f16a1-model-cache\") pod \"sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw\" (UID: \"4e7c676d-b4b1-4aab-bce5-4966103f16a1\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw"
Sep 10 21:03:35.833049 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:35.833028    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e7c676d-b4b1-4aab-bce5-4966103f16a1-tmp-dir\") pod \"sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw\" (UID: \"4e7c676d-b4b1-4aab-bce5-4966103f16a1\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw"
Sep 10 21:03:35.834811 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:35.834791    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e7c676d-b4b1-4aab-bce5-4966103f16a1-dshm\") pod \"sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw\" (UID: \"4e7c676d-b4b1-4aab-bce5-4966103f16a1\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw"
Sep 10 21:03:35.834911 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:35.834898    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e7c676d-b4b1-4aab-bce5-4966103f16a1-tls-certs\") pod \"sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw\" (UID: \"4e7c676d-b4b1-4aab-bce5-4966103f16a1\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw"
Sep 10 21:03:35.846207 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:35.846157    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd6rx\" (UniqueName: \"kubernetes.io/projected/4e7c676d-b4b1-4aab-bce5-4966103f16a1-kube-api-access-fd6rx\") pod \"sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw\" (UID: \"4e7c676d-b4b1-4aab-bce5-4966103f16a1\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw"
Sep 10 21:03:35.926237 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:35.926135    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw"
Sep 10 21:03:36.202636 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:36.202609    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw"]
Sep 10 21:03:36.203772 ip-10-0-141-185 kubenswrapper[2564]: W0910 21:03:36.203749    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e7c676d_b4b1_4aab_bce5_4966103f16a1.slice/crio-706b5279987a4c702a1a940970821a5bab1008753937eb99cfd0a0155524e509 WatchSource:0}: Error finding container 706b5279987a4c702a1a940970821a5bab1008753937eb99cfd0a0155524e509: Status 404 returned error can't find the container with id 706b5279987a4c702a1a940970821a5bab1008753937eb99cfd0a0155524e509
Sep 10 21:03:36.854919 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:36.854887    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw" event={"ID":"4e7c676d-b4b1-4aab-bce5-4966103f16a1","Type":"ContainerStarted","Data":"8860e20a3e464c2b7207c2e1243bdf5c95941a24de13ec95382b7c57682b4482"}
Sep 10 21:03:36.854919 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:36.854931    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw" event={"ID":"4e7c676d-b4b1-4aab-bce5-4966103f16a1","Type":"ContainerStarted","Data":"706b5279987a4c702a1a940970821a5bab1008753937eb99cfd0a0155524e509"}
Sep 10 21:03:36.874289 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:36.874243    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw" podStartSLOduration=1.8742283130000001 podStartE2EDuration="1.874228313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:03:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:03:36.871019117 +0000 UTC m=+1656.836087306" watchObservedRunningTime="2026-09-10 21:03:36.874228313 +0000 UTC m=+1656.839296507"
Sep 10 21:03:37.803831 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:37.803803    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-849fe6-64bf9fdb9c-qm86f"
Sep 10 21:03:45.926350 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:45.926315    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw"
Sep 10 21:03:45.926861 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:45.926575    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw"
Sep 10 21:03:45.939086 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:45.939060    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw"
Sep 10 21:03:46.925948 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:46.925917    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw"
Sep 10 21:03:51.769000 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:51.768956    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d66961-qkbs4"]
Sep 10 21:03:51.772617 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:51.772591    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d66961-qkbs4"
Sep 10 21:03:51.774776 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:51.774755    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plv2h\""
Sep 10 21:03:51.774861 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:51.774782    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:03:51.779587 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:51.779558    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d66961-qkbs4"]
Sep 10 21:03:51.872518 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:51.872489    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb200f3c-8bd3-49c1-8ded-4269c3af7d91-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d66961-qkbs4\" (UID: \"fb200f3c-8bd3-49c1-8ded-4269c3af7d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d66961-qkbs4"
Sep 10 21:03:51.973429 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:51.973394    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb200f3c-8bd3-49c1-8ded-4269c3af7d91-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d66961-qkbs4\" (UID: \"fb200f3c-8bd3-49c1-8ded-4269c3af7d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d66961-qkbs4"
Sep 10 21:03:51.974003 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:51.973982    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb200f3c-8bd3-49c1-8ded-4269c3af7d91-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d66961-qkbs4\" (UID: \"fb200f3c-8bd3-49c1-8ded-4269c3af7d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d66961-qkbs4"
Sep 10 21:03:52.086206 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:52.086159    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d66961-qkbs4"
Sep 10 21:03:52.360131 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:52.360108    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d66961-qkbs4"]
Sep 10 21:03:52.361630 ip-10-0-141-185 kubenswrapper[2564]: W0910 21:03:52.361602    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb200f3c_8bd3_49c1_8ded_4269c3af7d91.slice/crio-07884f0c14c245ee11b8df4333e84384712b4eceec53391227284adcaf185226 WatchSource:0}: Error finding container 07884f0c14c245ee11b8df4333e84384712b4eceec53391227284adcaf185226: Status 404 returned error can't find the container with id 07884f0c14c245ee11b8df4333e84384712b4eceec53391227284adcaf185226
Sep 10 21:03:52.949399 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:52.949318    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d66961-qkbs4" event={"ID":"fb200f3c-8bd3-49c1-8ded-4269c3af7d91","Type":"ContainerStarted","Data":"d355a172c4e00bc8be735cdcb265b1858440eb5e9df267e43b35bfa02e28d7a6"}
Sep 10 21:03:52.949399 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:52.949370    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d66961-qkbs4" event={"ID":"fb200f3c-8bd3-49c1-8ded-4269c3af7d91","Type":"ContainerStarted","Data":"07884f0c14c245ee11b8df4333e84384712b4eceec53391227284adcaf185226"}
Sep 10 21:03:52.966011 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:52.965965    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d66961-qkbs4" podStartSLOduration=1.965950688 podStartE2EDuration="1.965950688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:03:52.964902978 +0000 UTC m=+1672.929971163" watchObservedRunningTime="2026-09-10 21:03:52.965950688 +0000 UTC m=+1672.931018883"
Sep 10 21:03:53.963458 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:53.963418    2564 generic.go:356] "Generic (PLEG): container finished" podID="fb200f3c-8bd3-49c1-8ded-4269c3af7d91" containerID="d355a172c4e00bc8be735cdcb265b1858440eb5e9df267e43b35bfa02e28d7a6" exitCode=0
Sep 10 21:03:53.963841 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:53.963511    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d66961-qkbs4" event={"ID":"fb200f3c-8bd3-49c1-8ded-4269c3af7d91","Type":"ContainerDied","Data":"d355a172c4e00bc8be735cdcb265b1858440eb5e9df267e43b35bfa02e28d7a6"}
Sep 10 21:03:55.108960 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:55.108932    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d66961-qkbs4"
Sep 10 21:03:55.202048 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:55.202025    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb200f3c-8bd3-49c1-8ded-4269c3af7d91-maas-api-service-ca\") pod \"fb200f3c-8bd3-49c1-8ded-4269c3af7d91\" (UID: \"fb200f3c-8bd3-49c1-8ded-4269c3af7d91\") "
Sep 10 21:03:55.202422 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:55.202403    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb200f3c-8bd3-49c1-8ded-4269c3af7d91-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fb200f3c-8bd3-49c1-8ded-4269c3af7d91" (UID: "fb200f3c-8bd3-49c1-8ded-4269c3af7d91"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:03:55.303613 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:55.303555    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb200f3c-8bd3-49c1-8ded-4269c3af7d91-maas-api-service-ca\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:03:55.977045 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:55.977019    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d66961-qkbs4"
Sep 10 21:03:55.977225 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:55.977022    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d66961-qkbs4" event={"ID":"fb200f3c-8bd3-49c1-8ded-4269c3af7d91","Type":"ContainerDied","Data":"07884f0c14c245ee11b8df4333e84384712b4eceec53391227284adcaf185226"}
Sep 10 21:03:55.977225 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:03:55.977151    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07884f0c14c245ee11b8df4333e84384712b4eceec53391227284adcaf185226"
Sep 10 21:04:00.705721 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:00.705679    2564 scope.go:117] "RemoveContainer" containerID="a978d9d9a1185f9a114a54b521442c3c6f7797c9c27abe74191d3b09fa49de1d"
Sep 10 21:04:07.583559 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:07.583526    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d66961-589f585555-zdn8k"]
Sep 10 21:04:07.586083 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:07.583822    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-d66961-589f585555-zdn8k" podUID="d56c672e-5ce0-484f-9bb5-ff101fc00463" containerName="payload-processing" containerID="cri-o://8515e6f23b8a6e11c1d4d262054a346ca85cb364415a586f12ca6f005250c501" gracePeriod=30
Sep 10 21:04:07.784668 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:07.784633    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d66961-qkbs4"]
Sep 10 21:04:07.790309 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:07.790277    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d66961-qkbs4"]
Sep 10 21:04:07.840230 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:07.840162    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d66961-589f585555-zdn8k"
Sep 10 21:04:08.003730 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:08.003698    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvzfr\" (UniqueName: \"kubernetes.io/projected/d56c672e-5ce0-484f-9bb5-ff101fc00463-kube-api-access-vvzfr\") pod \"d56c672e-5ce0-484f-9bb5-ff101fc00463\" (UID: \"d56c672e-5ce0-484f-9bb5-ff101fc00463\") "
Sep 10 21:04:08.003897 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:08.003761    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d56c672e-5ce0-484f-9bb5-ff101fc00463-plugins-config-volume\") pod \"d56c672e-5ce0-484f-9bb5-ff101fc00463\" (UID: \"d56c672e-5ce0-484f-9bb5-ff101fc00463\") "
Sep 10 21:04:08.004154 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:08.004127    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d56c672e-5ce0-484f-9bb5-ff101fc00463-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d56c672e-5ce0-484f-9bb5-ff101fc00463" (UID: "d56c672e-5ce0-484f-9bb5-ff101fc00463"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:04:08.005837 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:08.005804    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d56c672e-5ce0-484f-9bb5-ff101fc00463-kube-api-access-vvzfr" (OuterVolumeSpecName: "kube-api-access-vvzfr") pod "d56c672e-5ce0-484f-9bb5-ff101fc00463" (UID: "d56c672e-5ce0-484f-9bb5-ff101fc00463"). InnerVolumeSpecName "kube-api-access-vvzfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:04:08.051439 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:08.051413    2564 generic.go:356] "Generic (PLEG): container finished" podID="d56c672e-5ce0-484f-9bb5-ff101fc00463" containerID="8515e6f23b8a6e11c1d4d262054a346ca85cb364415a586f12ca6f005250c501" exitCode=0
Sep 10 21:04:08.051535 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:08.051458    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d66961-589f585555-zdn8k"
Sep 10 21:04:08.051535 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:08.051501    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d66961-589f585555-zdn8k" event={"ID":"d56c672e-5ce0-484f-9bb5-ff101fc00463","Type":"ContainerDied","Data":"8515e6f23b8a6e11c1d4d262054a346ca85cb364415a586f12ca6f005250c501"}
Sep 10 21:04:08.051611 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:08.051559    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d66961-589f585555-zdn8k" event={"ID":"d56c672e-5ce0-484f-9bb5-ff101fc00463","Type":"ContainerDied","Data":"6bc1ca01ef7109155ea7348123fa0eddc7b1fb79c19f7488221821e246272a99"}
Sep 10 21:04:08.051611 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:08.051588    2564 scope.go:117] "RemoveContainer" containerID="8515e6f23b8a6e11c1d4d262054a346ca85cb364415a586f12ca6f005250c501"
Sep 10 21:04:08.060672 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:08.060656    2564 scope.go:117] "RemoveContainer" containerID="8515e6f23b8a6e11c1d4d262054a346ca85cb364415a586f12ca6f005250c501"
Sep 10 21:04:08.060937 ip-10-0-141-185 kubenswrapper[2564]: E0910 21:04:08.060916    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8515e6f23b8a6e11c1d4d262054a346ca85cb364415a586f12ca6f005250c501\": container with ID starting with 8515e6f23b8a6e11c1d4d262054a346ca85cb364415a586f12ca6f005250c501 not found: ID does not exist" containerID="8515e6f23b8a6e11c1d4d262054a346ca85cb364415a586f12ca6f005250c501"
Sep 10 21:04:08.060983 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:08.060945    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8515e6f23b8a6e11c1d4d262054a346ca85cb364415a586f12ca6f005250c501"} err="failed to get container status \"8515e6f23b8a6e11c1d4d262054a346ca85cb364415a586f12ca6f005250c501\": rpc error: code = NotFound desc = could not find container \"8515e6f23b8a6e11c1d4d262054a346ca85cb364415a586f12ca6f005250c501\": container with ID starting with 8515e6f23b8a6e11c1d4d262054a346ca85cb364415a586f12ca6f005250c501 not found: ID does not exist"
Sep 10 21:04:08.075588 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:08.075568    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d66961-589f585555-zdn8k"]
Sep 10 21:04:08.079513 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:08.079493    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d66961-589f585555-zdn8k"]
Sep 10 21:04:08.105203 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:08.105108    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvzfr\" (UniqueName: \"kubernetes.io/projected/d56c672e-5ce0-484f-9bb5-ff101fc00463-kube-api-access-vvzfr\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:04:08.105203 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:08.105135    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d56c672e-5ce0-484f-9bb5-ff101fc00463-plugins-config-volume\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:04:08.541931 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:08.541848    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d56c672e-5ce0-484f-9bb5-ff101fc00463" path="/var/lib/kubelet/pods/d56c672e-5ce0-484f-9bb5-ff101fc00463/volumes"
Sep 10 21:04:08.542224 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:08.542210    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb200f3c-8bd3-49c1-8ded-4269c3af7d91" path="/var/lib/kubelet/pods/fb200f3c-8bd3-49c1-8ded-4269c3af7d91/volumes"
Sep 10 21:04:15.077782 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:15.077752    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-849fe6-5bk8b"]
Sep 10 21:04:15.078156 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:15.078116    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb200f3c-8bd3-49c1-8ded-4269c3af7d91" containerName="revoke-keys"
Sep 10 21:04:15.078156 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:15.078126    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb200f3c-8bd3-49c1-8ded-4269c3af7d91" containerName="revoke-keys"
Sep 10 21:04:15.078156 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:15.078142    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d56c672e-5ce0-484f-9bb5-ff101fc00463" containerName="payload-processing"
Sep 10 21:04:15.078156 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:15.078147    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d56c672e-5ce0-484f-9bb5-ff101fc00463" containerName="payload-processing"
Sep 10 21:04:15.078309 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:15.078226    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d56c672e-5ce0-484f-9bb5-ff101fc00463" containerName="payload-processing"
Sep 10 21:04:15.078309 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:15.078235    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb200f3c-8bd3-49c1-8ded-4269c3af7d91" containerName="revoke-keys"
Sep 10 21:04:15.082555 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:15.082529    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-849fe6-5bk8b"
Sep 10 21:04:15.084948 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:15.084925    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:04:15.085056 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:15.084982    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plv2h\""
Sep 10 21:04:15.090698 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:15.090671    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-849fe6-5bk8b"]
Sep 10 21:04:15.159787 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:15.159760    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9284aa5d-66b5-4b8c-a177-5b33ed8e6f6c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-849fe6-5bk8b\" (UID: \"9284aa5d-66b5-4b8c-a177-5b33ed8e6f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-849fe6-5bk8b"
Sep 10 21:04:15.261187 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:15.261135    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9284aa5d-66b5-4b8c-a177-5b33ed8e6f6c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-849fe6-5bk8b\" (UID: \"9284aa5d-66b5-4b8c-a177-5b33ed8e6f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-849fe6-5bk8b"
Sep 10 21:04:15.261778 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:15.261755    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9284aa5d-66b5-4b8c-a177-5b33ed8e6f6c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-849fe6-5bk8b\" (UID: \"9284aa5d-66b5-4b8c-a177-5b33ed8e6f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-849fe6-5bk8b"
Sep 10 21:04:15.395324 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:15.395251    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-849fe6-5bk8b"
Sep 10 21:04:15.669643 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:15.669613    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-849fe6-5bk8b"]
Sep 10 21:04:15.670939 ip-10-0-141-185 kubenswrapper[2564]: W0910 21:04:15.670913    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9284aa5d_66b5_4b8c_a177_5b33ed8e6f6c.slice/crio-5746d794970bab25db50ee9823f30b2a906f0746dcaa54e59d9925b7ded71d34 WatchSource:0}: Error finding container 5746d794970bab25db50ee9823f30b2a906f0746dcaa54e59d9925b7ded71d34: Status 404 returned error can't find the container with id 5746d794970bab25db50ee9823f30b2a906f0746dcaa54e59d9925b7ded71d34
Sep 10 21:04:16.103078 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:16.103036    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-849fe6-5bk8b" event={"ID":"9284aa5d-66b5-4b8c-a177-5b33ed8e6f6c","Type":"ContainerStarted","Data":"b1c9d8501ed66fd7ccac6a2d6d0281117777c10f8ac65a44dfe1d5d39461b69f"}
Sep 10 21:04:16.103432 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:16.103135    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-849fe6-5bk8b" event={"ID":"9284aa5d-66b5-4b8c-a177-5b33ed8e6f6c","Type":"ContainerStarted","Data":"5746d794970bab25db50ee9823f30b2a906f0746dcaa54e59d9925b7ded71d34"}
Sep 10 21:04:16.123463 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:16.123418    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-849fe6-5bk8b" podStartSLOduration=1.123402609 podStartE2EDuration="1.123402609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:04:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:04:16.120960985 +0000 UTC m=+1696.086029179" watchObservedRunningTime="2026-09-10 21:04:16.123402609 +0000 UTC m=+1696.088470804"
Sep 10 21:04:17.110394 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:17.110358    2564 generic.go:356] "Generic (PLEG): container finished" podID="9284aa5d-66b5-4b8c-a177-5b33ed8e6f6c" containerID="b1c9d8501ed66fd7ccac6a2d6d0281117777c10f8ac65a44dfe1d5d39461b69f" exitCode=0
Sep 10 21:04:17.110842 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:17.110445    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-849fe6-5bk8b" event={"ID":"9284aa5d-66b5-4b8c-a177-5b33ed8e6f6c","Type":"ContainerDied","Data":"b1c9d8501ed66fd7ccac6a2d6d0281117777c10f8ac65a44dfe1d5d39461b69f"}
Sep 10 21:04:18.245605 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:18.245581    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-849fe6-5bk8b"
Sep 10 21:04:18.285573 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:18.285545    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9284aa5d-66b5-4b8c-a177-5b33ed8e6f6c-maas-api-service-ca\") pod \"9284aa5d-66b5-4b8c-a177-5b33ed8e6f6c\" (UID: \"9284aa5d-66b5-4b8c-a177-5b33ed8e6f6c\") "
Sep 10 21:04:18.285882 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:18.285861    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9284aa5d-66b5-4b8c-a177-5b33ed8e6f6c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9284aa5d-66b5-4b8c-a177-5b33ed8e6f6c" (UID: "9284aa5d-66b5-4b8c-a177-5b33ed8e6f6c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:04:18.386846 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:18.386774    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9284aa5d-66b5-4b8c-a177-5b33ed8e6f6c-maas-api-service-ca\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:04:19.123049 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:19.123020    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-849fe6-5bk8b"
Sep 10 21:04:19.123049 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:19.123038    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-849fe6-5bk8b" event={"ID":"9284aa5d-66b5-4b8c-a177-5b33ed8e6f6c","Type":"ContainerDied","Data":"5746d794970bab25db50ee9823f30b2a906f0746dcaa54e59d9925b7ded71d34"}
Sep 10 21:04:19.123279 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:19.123091    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5746d794970bab25db50ee9823f30b2a906f0746dcaa54e59d9925b7ded71d34"
Sep 10 21:04:23.485184 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:23.485083    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw"]
Sep 10 21:04:30.703934 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:30.703900    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd"]
Sep 10 21:04:30.704413 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:30.704396    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9284aa5d-66b5-4b8c-a177-5b33ed8e6f6c" containerName="revoke-keys"
Sep 10 21:04:30.704413 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:30.704415    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9284aa5d-66b5-4b8c-a177-5b33ed8e6f6c" containerName="revoke-keys"
Sep 10 21:04:30.704499 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:30.704484    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9284aa5d-66b5-4b8c-a177-5b33ed8e6f6c" containerName="revoke-keys"
Sep 10 21:04:30.707794 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:30.707774    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd"
Sep 10 21:04:30.710712 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:30.710687    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3fc2b8\"/\"default-dockercfg-4hgln\""
Sep 10 21:04:30.710712 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:30.710702    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3fc2b8\"/\"openshift-service-ca.crt\""
Sep 10 21:04:30.710896 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:30.710731    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3fc2b8\"/\"kube-root-ca.crt\""
Sep 10 21:04:30.710896 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:30.710875    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3fc2b8\"/\"rate-test-model-3fc2b8-kserve-self-signed-certs\""
Sep 10 21:04:30.717212 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:30.717188    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd"]
Sep 10 21:04:30.787060 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:30.787036    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4581ab35-d3ec-41db-a488-c9515df6afca-tls-certs\") pod \"rate-test-model-3fc2b8-kserve-bddc577f5-74znd\" (UID: \"4581ab35-d3ec-41db-a488-c9515df6afca\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd"
Sep 10 21:04:30.787196 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:30.787064    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rpw8\" (UniqueName: \"kubernetes.io/projected/4581ab35-d3ec-41db-a488-c9515df6afca-kube-api-access-7rpw8\") pod \"rate-test-model-3fc2b8-kserve-bddc577f5-74znd\" (UID: \"4581ab35-d3ec-41db-a488-c9515df6afca\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd"
Sep 10 21:04:30.787196 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:30.787094    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4581ab35-d3ec-41db-a488-c9515df6afca-dshm\") pod \"rate-test-model-3fc2b8-kserve-bddc577f5-74znd\" (UID: \"4581ab35-d3ec-41db-a488-c9515df6afca\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd"
Sep 10 21:04:30.787196 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:30.787158    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4581ab35-d3ec-41db-a488-c9515df6afca-home\") pod \"rate-test-model-3fc2b8-kserve-bddc577f5-74znd\" (UID: \"4581ab35-d3ec-41db-a488-c9515df6afca\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd"
Sep 10 21:04:30.787307 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:30.787214    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4581ab35-d3ec-41db-a488-c9515df6afca-model-cache\") pod \"rate-test-model-3fc2b8-kserve-bddc577f5-74znd\" (UID: \"4581ab35-d3ec-41db-a488-c9515df6afca\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd"
Sep 10 21:04:30.787307 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:30.787277    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4581ab35-d3ec-41db-a488-c9515df6afca-tmp-dir\") pod \"rate-test-model-3fc2b8-kserve-bddc577f5-74znd\" (UID: \"4581ab35-d3ec-41db-a488-c9515df6afca\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd"
Sep 10 21:04:30.888326 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:30.888292    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4581ab35-d3ec-41db-a488-c9515df6afca-tmp-dir\") pod \"rate-test-model-3fc2b8-kserve-bddc577f5-74znd\" (UID: \"4581ab35-d3ec-41db-a488-c9515df6afca\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd"
Sep 10 21:04:30.888482 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:30.888353    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4581ab35-d3ec-41db-a488-c9515df6afca-tls-certs\") pod \"rate-test-model-3fc2b8-kserve-bddc577f5-74znd\" (UID: \"4581ab35-d3ec-41db-a488-c9515df6afca\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd"
Sep 10 21:04:30.888482 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:30.888383    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rpw8\" (UniqueName: \"kubernetes.io/projected/4581ab35-d3ec-41db-a488-c9515df6afca-kube-api-access-7rpw8\") pod \"rate-test-model-3fc2b8-kserve-bddc577f5-74znd\" (UID: \"4581ab35-d3ec-41db-a488-c9515df6afca\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd"
Sep 10 21:04:30.888482 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:30.888429    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4581ab35-d3ec-41db-a488-c9515df6afca-dshm\") pod \"rate-test-model-3fc2b8-kserve-bddc577f5-74znd\" (UID: \"4581ab35-d3ec-41db-a488-c9515df6afca\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd"
Sep 10 21:04:30.888482 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:30.888467    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4581ab35-d3ec-41db-a488-c9515df6afca-home\") pod \"rate-test-model-3fc2b8-kserve-bddc577f5-74znd\" (UID: \"4581ab35-d3ec-41db-a488-c9515df6afca\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd"
Sep 10 21:04:30.888699 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:30.888493    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4581ab35-d3ec-41db-a488-c9515df6afca-model-cache\") pod \"rate-test-model-3fc2b8-kserve-bddc577f5-74znd\" (UID: \"4581ab35-d3ec-41db-a488-c9515df6afca\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd"
Sep 10 21:04:30.888759 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:30.888732    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4581ab35-d3ec-41db-a488-c9515df6afca-tmp-dir\") pod \"rate-test-model-3fc2b8-kserve-bddc577f5-74znd\" (UID: \"4581ab35-d3ec-41db-a488-c9515df6afca\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd"
Sep 10 21:04:30.888875 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:30.888850    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4581ab35-d3ec-41db-a488-c9515df6afca-home\") pod \"rate-test-model-3fc2b8-kserve-bddc577f5-74znd\" (UID: \"4581ab35-d3ec-41db-a488-c9515df6afca\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd"
Sep 10 21:04:30.888996 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:30.888857    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4581ab35-d3ec-41db-a488-c9515df6afca-model-cache\") pod \"rate-test-model-3fc2b8-kserve-bddc577f5-74znd\" (UID: \"4581ab35-d3ec-41db-a488-c9515df6afca\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd"
Sep 10 21:04:30.890747 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:30.890729    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4581ab35-d3ec-41db-a488-c9515df6afca-dshm\") pod \"rate-test-model-3fc2b8-kserve-bddc577f5-74znd\" (UID: \"4581ab35-d3ec-41db-a488-c9515df6afca\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd"
Sep 10 21:04:30.890858 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:30.890841    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4581ab35-d3ec-41db-a488-c9515df6afca-tls-certs\") pod \"rate-test-model-3fc2b8-kserve-bddc577f5-74znd\" (UID: \"4581ab35-d3ec-41db-a488-c9515df6afca\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd"
Sep 10 21:04:30.896473 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:30.896448    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rpw8\" (UniqueName: \"kubernetes.io/projected/4581ab35-d3ec-41db-a488-c9515df6afca-kube-api-access-7rpw8\") pod \"rate-test-model-3fc2b8-kserve-bddc577f5-74znd\" (UID: \"4581ab35-d3ec-41db-a488-c9515df6afca\") " pod="ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd"
Sep 10 21:04:31.019411 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:31.019339    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd"
Sep 10 21:04:31.052610 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:31.052578    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-849fe6-64bf9fdb9c-qm86f"]
Sep 10 21:04:31.052954 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:31.052839    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-849fe6-64bf9fdb9c-qm86f" podUID="c99a7daa-cd4b-476c-940f-1637f7f80088" containerName="payload-processing" containerID="cri-o://cf81b9c077f296bdab7c42004d1b7f496d384bca886f992cfe814773a186e43f" gracePeriod=30
Sep 10 21:04:31.192661 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:31.192626    2564 generic.go:356] "Generic (PLEG): container finished" podID="c99a7daa-cd4b-476c-940f-1637f7f80088" containerID="cf81b9c077f296bdab7c42004d1b7f496d384bca886f992cfe814773a186e43f" exitCode=0
Sep 10 21:04:31.192837 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:31.192700    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-849fe6-64bf9fdb9c-qm86f" event={"ID":"c99a7daa-cd4b-476c-940f-1637f7f80088","Type":"ContainerDied","Data":"cf81b9c077f296bdab7c42004d1b7f496d384bca886f992cfe814773a186e43f"}
Sep 10 21:04:31.229773 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:31.229745    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-849fe6-5bk8b"]
Sep 10 21:04:31.232248 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:31.232223    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-849fe6-5bk8b"]
Sep 10 21:04:31.299573 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:31.299550    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-849fe6-64bf9fdb9c-qm86f"
Sep 10 21:04:31.312158 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:31.312127    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd"]
Sep 10 21:04:31.314319 ip-10-0-141-185 kubenswrapper[2564]: W0910 21:04:31.314296    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4581ab35_d3ec_41db_a488_c9515df6afca.slice/crio-bb0c18ce7c3d9622eb0d6c40bfa16974c15ff97594bf243ef62c2136b5c4ee74 WatchSource:0}: Error finding container bb0c18ce7c3d9622eb0d6c40bfa16974c15ff97594bf243ef62c2136b5c4ee74: Status 404 returned error can't find the container with id bb0c18ce7c3d9622eb0d6c40bfa16974c15ff97594bf243ef62c2136b5c4ee74
Sep 10 21:04:31.393569 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:31.393537    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c99a7daa-cd4b-476c-940f-1637f7f80088-plugins-config-volume\") pod \"c99a7daa-cd4b-476c-940f-1637f7f80088\" (UID: \"c99a7daa-cd4b-476c-940f-1637f7f80088\") "
Sep 10 21:04:31.393705 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:31.393596    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4m2f5\" (UniqueName: \"kubernetes.io/projected/c99a7daa-cd4b-476c-940f-1637f7f80088-kube-api-access-4m2f5\") pod \"c99a7daa-cd4b-476c-940f-1637f7f80088\" (UID: \"c99a7daa-cd4b-476c-940f-1637f7f80088\") "
Sep 10 21:04:31.394842 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:31.394748    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c99a7daa-cd4b-476c-940f-1637f7f80088-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c99a7daa-cd4b-476c-940f-1637f7f80088" (UID: "c99a7daa-cd4b-476c-940f-1637f7f80088"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:04:31.397222 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:31.397157    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c99a7daa-cd4b-476c-940f-1637f7f80088-kube-api-access-4m2f5" (OuterVolumeSpecName: "kube-api-access-4m2f5") pod "c99a7daa-cd4b-476c-940f-1637f7f80088" (UID: "c99a7daa-cd4b-476c-940f-1637f7f80088"). InnerVolumeSpecName "kube-api-access-4m2f5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:04:31.494699 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:31.494668    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m2f5\" (UniqueName: \"kubernetes.io/projected/c99a7daa-cd4b-476c-940f-1637f7f80088-kube-api-access-4m2f5\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:04:31.494699 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:31.494693    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c99a7daa-cd4b-476c-940f-1637f7f80088-plugins-config-volume\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:04:32.200233 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:32.200127    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd" event={"ID":"4581ab35-d3ec-41db-a488-c9515df6afca","Type":"ContainerStarted","Data":"ab70c22f1311debfd0a1635898235b54c23317e89030a0ef90ef253b4721557f"}
Sep 10 21:04:32.200233 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:32.200200    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd" event={"ID":"4581ab35-d3ec-41db-a488-c9515df6afca","Type":"ContainerStarted","Data":"bb0c18ce7c3d9622eb0d6c40bfa16974c15ff97594bf243ef62c2136b5c4ee74"}
Sep 10 21:04:32.201642 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:32.201618    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-849fe6-64bf9fdb9c-qm86f"
Sep 10 21:04:32.201736 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:32.201615    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-849fe6-64bf9fdb9c-qm86f" event={"ID":"c99a7daa-cd4b-476c-940f-1637f7f80088","Type":"ContainerDied","Data":"e37f96754454687f893d1b135821e16941a515fc89e40041a538e61d00835728"}
Sep 10 21:04:32.201784 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:32.201752    2564 scope.go:117] "RemoveContainer" containerID="cf81b9c077f296bdab7c42004d1b7f496d384bca886f992cfe814773a186e43f"
Sep 10 21:04:32.220759 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:32.220719    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd" podStartSLOduration=2.220705248 podStartE2EDuration="2.220705248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:04:32.217705146 +0000 UTC m=+1712.182773354" watchObservedRunningTime="2026-09-10 21:04:32.220705248 +0000 UTC m=+1712.185773443"
Sep 10 21:04:32.230882 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:32.230860    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-849fe6-64bf9fdb9c-qm86f"]
Sep 10 21:04:32.234400 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:32.234381    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-849fe6-64bf9fdb9c-qm86f"]
Sep 10 21:04:32.541647 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:32.541552    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9284aa5d-66b5-4b8c-a177-5b33ed8e6f6c" path="/var/lib/kubelet/pods/9284aa5d-66b5-4b8c-a177-5b33ed8e6f6c/volumes"
Sep 10 21:04:32.542006 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:32.541985    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c99a7daa-cd4b-476c-940f-1637f7f80088" path="/var/lib/kubelet/pods/c99a7daa-cd4b-476c-940f-1637f7f80088/volumes"
Sep 10 21:04:38.270528 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:38.270493    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cd89ac-s67zg"]
Sep 10 21:04:38.271007 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:38.270875    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c99a7daa-cd4b-476c-940f-1637f7f80088" containerName="payload-processing"
Sep 10 21:04:38.271007 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:38.270886    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c99a7daa-cd4b-476c-940f-1637f7f80088" containerName="payload-processing"
Sep 10 21:04:38.271007 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:38.270958    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c99a7daa-cd4b-476c-940f-1637f7f80088" containerName="payload-processing"
Sep 10 21:04:38.273860 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:38.273845    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cd89ac-s67zg"
Sep 10 21:04:38.276412 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:38.276386    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plv2h\""
Sep 10 21:04:38.276510 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:38.276386    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:04:38.282312 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:38.282287    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cd89ac-s67zg"]
Sep 10 21:04:38.356071 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:38.356043    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06522a7d-3df3-431c-be76-78f845ea76ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-cd89ac-s67zg\" (UID: \"06522a7d-3df3-431c-be76-78f845ea76ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cd89ac-s67zg"
Sep 10 21:04:38.456576 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:38.456543    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06522a7d-3df3-431c-be76-78f845ea76ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-cd89ac-s67zg\" (UID: \"06522a7d-3df3-431c-be76-78f845ea76ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cd89ac-s67zg"
Sep 10 21:04:38.457241 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:38.457223    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06522a7d-3df3-431c-be76-78f845ea76ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-cd89ac-s67zg\" (UID: \"06522a7d-3df3-431c-be76-78f845ea76ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cd89ac-s67zg"
Sep 10 21:04:38.505070 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:38.505037    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw" podUID="4e7c676d-b4b1-4aab-bce5-4966103f16a1" containerName="main" containerID="cri-o://8860e20a3e464c2b7207c2e1243bdf5c95941a24de13ec95382b7c57682b4482" gracePeriod=45
Sep 10 21:04:38.585833 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:38.585809    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cd89ac-s67zg"
Sep 10 21:04:38.772581 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:38.772558    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw"
Sep 10 21:04:38.859449 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:38.859426    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cd89ac-s67zg"]
Sep 10 21:04:38.859748 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:38.859734    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e7c676d-b4b1-4aab-bce5-4966103f16a1-tls-certs\") pod \"4e7c676d-b4b1-4aab-bce5-4966103f16a1\" (UID: \"4e7c676d-b4b1-4aab-bce5-4966103f16a1\") "
Sep 10 21:04:38.859801 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:38.859761    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e7c676d-b4b1-4aab-bce5-4966103f16a1-home\") pod \"4e7c676d-b4b1-4aab-bce5-4966103f16a1\" (UID: \"4e7c676d-b4b1-4aab-bce5-4966103f16a1\") "
Sep 10 21:04:38.859856 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:38.859803    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e7c676d-b4b1-4aab-bce5-4966103f16a1-tmp-dir\") pod \"4e7c676d-b4b1-4aab-bce5-4966103f16a1\" (UID: \"4e7c676d-b4b1-4aab-bce5-4966103f16a1\") "
Sep 10 21:04:38.859856 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:38.859832    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e7c676d-b4b1-4aab-bce5-4966103f16a1-model-cache\") pod \"4e7c676d-b4b1-4aab-bce5-4966103f16a1\" (UID: \"4e7c676d-b4b1-4aab-bce5-4966103f16a1\") "
Sep 10 21:04:38.859856 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:38.859851    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fd6rx\" (UniqueName: \"kubernetes.io/projected/4e7c676d-b4b1-4aab-bce5-4966103f16a1-kube-api-access-fd6rx\") pod \"4e7c676d-b4b1-4aab-bce5-4966103f16a1\" (UID: \"4e7c676d-b4b1-4aab-bce5-4966103f16a1\") "
Sep 10 21:04:38.860014 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:38.859873    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e7c676d-b4b1-4aab-bce5-4966103f16a1-dshm\") pod \"4e7c676d-b4b1-4aab-bce5-4966103f16a1\" (UID: \"4e7c676d-b4b1-4aab-bce5-4966103f16a1\") "
Sep 10 21:04:38.860070 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:38.860016    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7c676d-b4b1-4aab-bce5-4966103f16a1-home" (OuterVolumeSpecName: "home") pod "4e7c676d-b4b1-4aab-bce5-4966103f16a1" (UID: "4e7c676d-b4b1-4aab-bce5-4966103f16a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:04:38.860149 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:38.860106    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7c676d-b4b1-4aab-bce5-4966103f16a1-model-cache" (OuterVolumeSpecName: "model-cache") pod "4e7c676d-b4b1-4aab-bce5-4966103f16a1" (UID: "4e7c676d-b4b1-4aab-bce5-4966103f16a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:04:38.860251 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:38.860152    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e7c676d-b4b1-4aab-bce5-4966103f16a1-home\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:04:38.860251 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:38.860177    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7c676d-b4b1-4aab-bce5-4966103f16a1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4e7c676d-b4b1-4aab-bce5-4966103f16a1" (UID: "4e7c676d-b4b1-4aab-bce5-4966103f16a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:04:38.861293 ip-10-0-141-185 kubenswrapper[2564]: W0910 21:04:38.861264    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06522a7d_3df3_431c_be76_78f845ea76ba.slice/crio-3a0844038ce869d9a3fb948b7a04bc82f417c4baa647960fe2e4a9b7824fb096 WatchSource:0}: Error finding container 3a0844038ce869d9a3fb948b7a04bc82f417c4baa647960fe2e4a9b7824fb096: Status 404 returned error can't find the container with id 3a0844038ce869d9a3fb948b7a04bc82f417c4baa647960fe2e4a9b7824fb096
Sep 10 21:04:38.861925 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:38.861901    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e7c676d-b4b1-4aab-bce5-4966103f16a1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e7c676d-b4b1-4aab-bce5-4966103f16a1" (UID: "4e7c676d-b4b1-4aab-bce5-4966103f16a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:04:38.862024 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:38.861939    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7c676d-b4b1-4aab-bce5-4966103f16a1-dshm" (OuterVolumeSpecName: "dshm") pod "4e7c676d-b4b1-4aab-bce5-4966103f16a1" (UID: "4e7c676d-b4b1-4aab-bce5-4966103f16a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:04:38.862024 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:38.861965    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e7c676d-b4b1-4aab-bce5-4966103f16a1-kube-api-access-fd6rx" (OuterVolumeSpecName: "kube-api-access-fd6rx") pod "4e7c676d-b4b1-4aab-bce5-4966103f16a1" (UID: "4e7c676d-b4b1-4aab-bce5-4966103f16a1"). InnerVolumeSpecName "kube-api-access-fd6rx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:04:38.961438 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:38.961409    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e7c676d-b4b1-4aab-bce5-4966103f16a1-tls-certs\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:04:38.961438 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:38.961434    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e7c676d-b4b1-4aab-bce5-4966103f16a1-tmp-dir\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:04:38.961438 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:38.961444    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e7c676d-b4b1-4aab-bce5-4966103f16a1-model-cache\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:04:38.961623 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:38.961454    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd6rx\" (UniqueName: \"kubernetes.io/projected/4e7c676d-b4b1-4aab-bce5-4966103f16a1-kube-api-access-fd6rx\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:04:38.961623 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:38.961464    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e7c676d-b4b1-4aab-bce5-4966103f16a1-dshm\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:04:39.243100 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:39.243027    2564 generic.go:356] "Generic (PLEG): container finished" podID="4e7c676d-b4b1-4aab-bce5-4966103f16a1" containerID="8860e20a3e464c2b7207c2e1243bdf5c95941a24de13ec95382b7c57682b4482" exitCode=0
Sep 10 21:04:39.243100 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:39.243074    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw"
Sep 10 21:04:39.243338 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:39.243113    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw" event={"ID":"4e7c676d-b4b1-4aab-bce5-4966103f16a1","Type":"ContainerDied","Data":"8860e20a3e464c2b7207c2e1243bdf5c95941a24de13ec95382b7c57682b4482"}
Sep 10 21:04:39.243338 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:39.243161    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw" event={"ID":"4e7c676d-b4b1-4aab-bce5-4966103f16a1","Type":"ContainerDied","Data":"706b5279987a4c702a1a940970821a5bab1008753937eb99cfd0a0155524e509"}
Sep 10 21:04:39.243338 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:39.243203    2564 scope.go:117] "RemoveContainer" containerID="8860e20a3e464c2b7207c2e1243bdf5c95941a24de13ec95382b7c57682b4482"
Sep 10 21:04:39.244890 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:39.244852    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cd89ac-s67zg" event={"ID":"06522a7d-3df3-431c-be76-78f845ea76ba","Type":"ContainerStarted","Data":"8128d2c21eb089c80603eeb7563a0bb39b4189d01cde23e660c3b87d1ccfc42f"}
Sep 10 21:04:39.245012 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:39.244913    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cd89ac-s67zg" event={"ID":"06522a7d-3df3-431c-be76-78f845ea76ba","Type":"ContainerStarted","Data":"3a0844038ce869d9a3fb948b7a04bc82f417c4baa647960fe2e4a9b7824fb096"}
Sep 10 21:04:39.259221 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:39.259196    2564 scope.go:117] "RemoveContainer" containerID="8860e20a3e464c2b7207c2e1243bdf5c95941a24de13ec95382b7c57682b4482"
Sep 10 21:04:39.259475 ip-10-0-141-185 kubenswrapper[2564]: E0910 21:04:39.259456    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8860e20a3e464c2b7207c2e1243bdf5c95941a24de13ec95382b7c57682b4482\": container with ID starting with 8860e20a3e464c2b7207c2e1243bdf5c95941a24de13ec95382b7c57682b4482 not found: ID does not exist" containerID="8860e20a3e464c2b7207c2e1243bdf5c95941a24de13ec95382b7c57682b4482"
Sep 10 21:04:39.259531 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:39.259482    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8860e20a3e464c2b7207c2e1243bdf5c95941a24de13ec95382b7c57682b4482"} err="failed to get container status \"8860e20a3e464c2b7207c2e1243bdf5c95941a24de13ec95382b7c57682b4482\": rpc error: code = NotFound desc = could not find container \"8860e20a3e464c2b7207c2e1243bdf5c95941a24de13ec95382b7c57682b4482\": container with ID starting with 8860e20a3e464c2b7207c2e1243bdf5c95941a24de13ec95382b7c57682b4482 not found: ID does not exist"
Sep 10 21:04:39.270933 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:39.270891    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cd89ac-s67zg" podStartSLOduration=1.270878191 podStartE2EDuration="1.270878191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:04:39.268101873 +0000 UTC m=+1719.233170067" watchObservedRunningTime="2026-09-10 21:04:39.270878191 +0000 UTC m=+1719.235946386"
Sep 10 21:04:39.285730 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:39.285708    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw"]
Sep 10 21:04:39.289218 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:39.289200    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4fe4f/sub-test-model-f4fe4f-kserve-c4996d6d5-88cvw"]
Sep 10 21:04:40.253369 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:40.253335    2564 generic.go:356] "Generic (PLEG): container finished" podID="06522a7d-3df3-431c-be76-78f845ea76ba" containerID="8128d2c21eb089c80603eeb7563a0bb39b4189d01cde23e660c3b87d1ccfc42f" exitCode=0
Sep 10 21:04:40.253369 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:40.253374    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cd89ac-s67zg" event={"ID":"06522a7d-3df3-431c-be76-78f845ea76ba","Type":"ContainerDied","Data":"8128d2c21eb089c80603eeb7563a0bb39b4189d01cde23e660c3b87d1ccfc42f"}
Sep 10 21:04:40.540833 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:40.540732    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e7c676d-b4b1-4aab-bce5-4966103f16a1" path="/var/lib/kubelet/pods/4e7c676d-b4b1-4aab-bce5-4966103f16a1/volumes"
Sep 10 21:04:41.020339 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:41.020305    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd"
Sep 10 21:04:41.020516 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:41.020366    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd"
Sep 10 21:04:41.032759 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:41.032736    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd"
Sep 10 21:04:41.272417 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:41.272340    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd"
Sep 10 21:04:41.399605 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:41.399582    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cd89ac-s67zg"
Sep 10 21:04:41.483469 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:41.483439    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06522a7d-3df3-431c-be76-78f845ea76ba-maas-api-service-ca\") pod \"06522a7d-3df3-431c-be76-78f845ea76ba\" (UID: \"06522a7d-3df3-431c-be76-78f845ea76ba\") "
Sep 10 21:04:41.483775 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:41.483752    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06522a7d-3df3-431c-be76-78f845ea76ba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "06522a7d-3df3-431c-be76-78f845ea76ba" (UID: "06522a7d-3df3-431c-be76-78f845ea76ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:04:41.584073 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:41.584049    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06522a7d-3df3-431c-be76-78f845ea76ba-maas-api-service-ca\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:04:42.267810 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:42.267778    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cd89ac-s67zg"
Sep 10 21:04:42.267974 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:42.267811    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cd89ac-s67zg" event={"ID":"06522a7d-3df3-431c-be76-78f845ea76ba","Type":"ContainerDied","Data":"3a0844038ce869d9a3fb948b7a04bc82f417c4baa647960fe2e4a9b7824fb096"}
Sep 10 21:04:42.267974 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:42.267856    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a0844038ce869d9a3fb948b7a04bc82f417c4baa647960fe2e4a9b7824fb096"
Sep 10 21:04:48.868224 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:48.868188    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cd89ac-29817900-qftgk"]
Sep 10 21:04:48.870938 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:48.870911    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cd89ac-29817900-qftgk"]
Sep 10 21:04:48.891450 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:48.891422    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cd89ac-cf8d5c4d9-lhgft"]
Sep 10 21:04:48.891657 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:48.891626    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cd89ac-cf8d5c4d9-lhgft" podUID="df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e" containerName="payload-processing" containerID="cri-o://56867c50c50b90d50b1752e4bd8169d2f57a6f67229bbd3f7239ba27ef335e24" gracePeriod=30
Sep 10 21:04:48.900509 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:48.900472    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cd89ac-5497d785db-gksj6"]
Sep 10 21:04:48.900729 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:48.900683    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cd89ac-5497d785db-gksj6" podUID="b832e901-0ff9-457e-ae81-a35d8ab799b6" containerName="payload-pre-processing" containerID="cri-o://12a396ae77df424330deac0827940f32e85c71a479f937cfda23e0d9e8f33527" gracePeriod=30
Sep 10 21:04:49.147448 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:49.147427    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cd89ac-cf8d5c4d9-lhgft"
Sep 10 21:04:49.175201 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:49.175162    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cd89ac-s67zg"]
Sep 10 21:04:49.176118 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:49.176096    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cd89ac-5497d785db-gksj6"
Sep 10 21:04:49.186996 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:49.186973    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cd89ac-s67zg"]
Sep 10 21:04:49.244806 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:49.244770    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b832e901-0ff9-457e-ae81-a35d8ab799b6-plugins-config-volume\") pod \"b832e901-0ff9-457e-ae81-a35d8ab799b6\" (UID: \"b832e901-0ff9-457e-ae81-a35d8ab799b6\") "
Sep 10 21:04:49.244940 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:49.244910    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6zrs\" (UniqueName: \"kubernetes.io/projected/df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e-kube-api-access-q6zrs\") pod \"df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e\" (UID: \"df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e\") "
Sep 10 21:04:49.245004 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:49.244983    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfcxn\" (UniqueName: \"kubernetes.io/projected/b832e901-0ff9-457e-ae81-a35d8ab799b6-kube-api-access-jfcxn\") pod \"b832e901-0ff9-457e-ae81-a35d8ab799b6\" (UID: \"b832e901-0ff9-457e-ae81-a35d8ab799b6\") "
Sep 10 21:04:49.245066 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:49.245012    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e-plugins-config-volume\") pod \"df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e\" (UID: \"df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e\") "
Sep 10 21:04:49.245130 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:49.245105    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b832e901-0ff9-457e-ae81-a35d8ab799b6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b832e901-0ff9-457e-ae81-a35d8ab799b6" (UID: "b832e901-0ff9-457e-ae81-a35d8ab799b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:04:49.245312 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:49.245296    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b832e901-0ff9-457e-ae81-a35d8ab799b6-plugins-config-volume\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:04:49.245477 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:49.245452    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e" (UID: "df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:04:49.247008 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:49.246987    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b832e901-0ff9-457e-ae81-a35d8ab799b6-kube-api-access-jfcxn" (OuterVolumeSpecName: "kube-api-access-jfcxn") pod "b832e901-0ff9-457e-ae81-a35d8ab799b6" (UID: "b832e901-0ff9-457e-ae81-a35d8ab799b6"). InnerVolumeSpecName "kube-api-access-jfcxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:04:49.247080 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:49.247006    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e-kube-api-access-q6zrs" (OuterVolumeSpecName: "kube-api-access-q6zrs") pod "df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e" (UID: "df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e"). InnerVolumeSpecName "kube-api-access-q6zrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:04:49.316993 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:49.316961    2564 generic.go:356] "Generic (PLEG): container finished" podID="df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e" containerID="56867c50c50b90d50b1752e4bd8169d2f57a6f67229bbd3f7239ba27ef335e24" exitCode=0
Sep 10 21:04:49.317198 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:49.317015    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cd89ac-cf8d5c4d9-lhgft"
Sep 10 21:04:49.317198 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:49.317051    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cd89ac-cf8d5c4d9-lhgft" event={"ID":"df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e","Type":"ContainerDied","Data":"56867c50c50b90d50b1752e4bd8169d2f57a6f67229bbd3f7239ba27ef335e24"}
Sep 10 21:04:49.317198 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:49.317133    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cd89ac-cf8d5c4d9-lhgft" event={"ID":"df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e","Type":"ContainerDied","Data":"b642f3e4618f6f1d566987743fd9bc49dd8c3d62300de05e11aff9e9f54a2454"}
Sep 10 21:04:49.317198 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:49.317178    2564 scope.go:117] "RemoveContainer" containerID="56867c50c50b90d50b1752e4bd8169d2f57a6f67229bbd3f7239ba27ef335e24"
Sep 10 21:04:49.318584 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:49.318561    2564 generic.go:356] "Generic (PLEG): container finished" podID="b832e901-0ff9-457e-ae81-a35d8ab799b6" containerID="12a396ae77df424330deac0827940f32e85c71a479f937cfda23e0d9e8f33527" exitCode=0
Sep 10 21:04:49.318706 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:49.318630    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cd89ac-5497d785db-gksj6"
Sep 10 21:04:49.318706 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:49.318648    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cd89ac-5497d785db-gksj6" event={"ID":"b832e901-0ff9-457e-ae81-a35d8ab799b6","Type":"ContainerDied","Data":"12a396ae77df424330deac0827940f32e85c71a479f937cfda23e0d9e8f33527"}
Sep 10 21:04:49.318706 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:49.318688    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cd89ac-5497d785db-gksj6" event={"ID":"b832e901-0ff9-457e-ae81-a35d8ab799b6","Type":"ContainerDied","Data":"1f467621fb7403075a04bf9add47440dc41901c6661a651bd11051832e227a44"}
Sep 10 21:04:49.328350 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:49.328333    2564 scope.go:117] "RemoveContainer" containerID="56867c50c50b90d50b1752e4bd8169d2f57a6f67229bbd3f7239ba27ef335e24"
Sep 10 21:04:49.328655 ip-10-0-141-185 kubenswrapper[2564]: E0910 21:04:49.328632    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56867c50c50b90d50b1752e4bd8169d2f57a6f67229bbd3f7239ba27ef335e24\": container with ID starting with 56867c50c50b90d50b1752e4bd8169d2f57a6f67229bbd3f7239ba27ef335e24 not found: ID does not exist" containerID="56867c50c50b90d50b1752e4bd8169d2f57a6f67229bbd3f7239ba27ef335e24"
Sep 10 21:04:49.328778 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:49.328659    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56867c50c50b90d50b1752e4bd8169d2f57a6f67229bbd3f7239ba27ef335e24"} err="failed to get container status \"56867c50c50b90d50b1752e4bd8169d2f57a6f67229bbd3f7239ba27ef335e24\": rpc error: code = NotFound desc = could not find container \"56867c50c50b90d50b1752e4bd8169d2f57a6f67229bbd3f7239ba27ef335e24\": container with ID starting with 56867c50c50b90d50b1752e4bd8169d2f57a6f67229bbd3f7239ba27ef335e24 not found: ID does not exist"
Sep 10 21:04:49.328778 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:49.328675    2564 scope.go:117] "RemoveContainer" containerID="12a396ae77df424330deac0827940f32e85c71a479f937cfda23e0d9e8f33527"
Sep 10 21:04:49.336700 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:49.336681    2564 scope.go:117] "RemoveContainer" containerID="12a396ae77df424330deac0827940f32e85c71a479f937cfda23e0d9e8f33527"
Sep 10 21:04:49.336954 ip-10-0-141-185 kubenswrapper[2564]: E0910 21:04:49.336936    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12a396ae77df424330deac0827940f32e85c71a479f937cfda23e0d9e8f33527\": container with ID starting with 12a396ae77df424330deac0827940f32e85c71a479f937cfda23e0d9e8f33527 not found: ID does not exist" containerID="12a396ae77df424330deac0827940f32e85c71a479f937cfda23e0d9e8f33527"
Sep 10 21:04:49.337005 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:49.336961    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12a396ae77df424330deac0827940f32e85c71a479f937cfda23e0d9e8f33527"} err="failed to get container status \"12a396ae77df424330deac0827940f32e85c71a479f937cfda23e0d9e8f33527\": rpc error: code = NotFound desc = could not find container \"12a396ae77df424330deac0827940f32e85c71a479f937cfda23e0d9e8f33527\": container with ID starting with 12a396ae77df424330deac0827940f32e85c71a479f937cfda23e0d9e8f33527 not found: ID does not exist"
Sep 10 21:04:49.345057 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:49.345022    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cd89ac-5497d785db-gksj6"]
Sep 10 21:04:49.345916 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:49.345896    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfcxn\" (UniqueName: \"kubernetes.io/projected/b832e901-0ff9-457e-ae81-a35d8ab799b6-kube-api-access-jfcxn\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:04:49.345966 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:49.345919    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e-plugins-config-volume\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:04:49.345966 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:49.345929    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6zrs\" (UniqueName: \"kubernetes.io/projected/df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e-kube-api-access-q6zrs\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:04:49.349951 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:49.349931    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cd89ac-5497d785db-gksj6"]
Sep 10 21:04:49.362957 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:49.362933    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cd89ac-cf8d5c4d9-lhgft"]
Sep 10 21:04:49.365062 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:49.365042    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cd89ac-cf8d5c4d9-lhgft"]
Sep 10 21:04:50.540414 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:50.540378    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06522a7d-3df3-431c-be76-78f845ea76ba" path="/var/lib/kubelet/pods/06522a7d-3df3-431c-be76-78f845ea76ba/volumes"
Sep 10 21:04:50.540792 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:50.540745    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b832e901-0ff9-457e-ae81-a35d8ab799b6" path="/var/lib/kubelet/pods/b832e901-0ff9-457e-ae81-a35d8ab799b6/volumes"
Sep 10 21:04:50.541066 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:50.541052    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e" path="/var/lib/kubelet/pods/df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e/volumes"
Sep 10 21:04:50.541319 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:50.541307    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8bc1141-a53e-4e3c-9abd-85fb096a5c95" path="/var/lib/kubelet/pods/f8bc1141-a53e-4e3c-9abd-85fb096a5c95/volumes"
Sep 10 21:04:52.303340 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.303309    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6"]
Sep 10 21:04:52.303795 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.303643    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06522a7d-3df3-431c-be76-78f845ea76ba" containerName="revoke-keys"
Sep 10 21:04:52.303795 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.303654    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="06522a7d-3df3-431c-be76-78f845ea76ba" containerName="revoke-keys"
Sep 10 21:04:52.303795 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.303665    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b832e901-0ff9-457e-ae81-a35d8ab799b6" containerName="payload-pre-processing"
Sep 10 21:04:52.303795 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.303671    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b832e901-0ff9-457e-ae81-a35d8ab799b6" containerName="payload-pre-processing"
Sep 10 21:04:52.303795 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.303679    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e" containerName="payload-processing"
Sep 10 21:04:52.303795 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.303686    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e" containerName="payload-processing"
Sep 10 21:04:52.303795 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.303693    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e7c676d-b4b1-4aab-bce5-4966103f16a1" containerName="main"
Sep 10 21:04:52.303795 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.303697    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e7c676d-b4b1-4aab-bce5-4966103f16a1" containerName="main"
Sep 10 21:04:52.303795 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.303752    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="df51c2f2-6b1b-4eb9-85ff-7b972ac06f8e" containerName="payload-processing"
Sep 10 21:04:52.303795 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.303760    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b832e901-0ff9-457e-ae81-a35d8ab799b6" containerName="payload-pre-processing"
Sep 10 21:04:52.303795 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.303767    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="06522a7d-3df3-431c-be76-78f845ea76ba" containerName="revoke-keys"
Sep 10 21:04:52.303795 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.303774    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e7c676d-b4b1-4aab-bce5-4966103f16a1" containerName="main"
Sep 10 21:04:52.307967 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.307943    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6"
Sep 10 21:04:52.310900 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.310872    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4fe4f\"/\"openshift-service-ca.crt\""
Sep 10 21:04:52.310900 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.310896    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4fe4f\"/\"rate-test-model-f4fe4f-kserve-self-signed-certs\""
Sep 10 21:04:52.311255 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.310921    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4fe4f\"/\"default-dockercfg-qf9qn\""
Sep 10 21:04:52.311255 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.310948    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4fe4f\"/\"kube-root-ca.crt\""
Sep 10 21:04:52.317291 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.317271    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6"]
Sep 10 21:04:52.468922 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.468875    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd9dde70-4a53-48eb-9699-238b760c58fc-home\") pod \"rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6\" (UID: \"cd9dde70-4a53-48eb-9699-238b760c58fc\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6"
Sep 10 21:04:52.468922 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.468927    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkzjk\" (UniqueName: \"kubernetes.io/projected/cd9dde70-4a53-48eb-9699-238b760c58fc-kube-api-access-fkzjk\") pod \"rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6\" (UID: \"cd9dde70-4a53-48eb-9699-238b760c58fc\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6"
Sep 10 21:04:52.469194 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.468971    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd9dde70-4a53-48eb-9699-238b760c58fc-dshm\") pod \"rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6\" (UID: \"cd9dde70-4a53-48eb-9699-238b760c58fc\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6"
Sep 10 21:04:52.469194 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.469132    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd9dde70-4a53-48eb-9699-238b760c58fc-tls-certs\") pod \"rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6\" (UID: \"cd9dde70-4a53-48eb-9699-238b760c58fc\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6"
Sep 10 21:04:52.469310 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.469289    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd9dde70-4a53-48eb-9699-238b760c58fc-model-cache\") pod \"rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6\" (UID: \"cd9dde70-4a53-48eb-9699-238b760c58fc\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6"
Sep 10 21:04:52.469368 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.469332    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd9dde70-4a53-48eb-9699-238b760c58fc-tmp-dir\") pod \"rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6\" (UID: \"cd9dde70-4a53-48eb-9699-238b760c58fc\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6"
Sep 10 21:04:52.570825 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.570790    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd9dde70-4a53-48eb-9699-238b760c58fc-model-cache\") pod \"rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6\" (UID: \"cd9dde70-4a53-48eb-9699-238b760c58fc\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6"
Sep 10 21:04:52.571005 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.570840    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd9dde70-4a53-48eb-9699-238b760c58fc-tmp-dir\") pod \"rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6\" (UID: \"cd9dde70-4a53-48eb-9699-238b760c58fc\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6"
Sep 10 21:04:52.571005 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.570879    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd9dde70-4a53-48eb-9699-238b760c58fc-home\") pod \"rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6\" (UID: \"cd9dde70-4a53-48eb-9699-238b760c58fc\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6"
Sep 10 21:04:52.571005 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.570912    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkzjk\" (UniqueName: \"kubernetes.io/projected/cd9dde70-4a53-48eb-9699-238b760c58fc-kube-api-access-fkzjk\") pod \"rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6\" (UID: \"cd9dde70-4a53-48eb-9699-238b760c58fc\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6"
Sep 10 21:04:52.571005 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.570935    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd9dde70-4a53-48eb-9699-238b760c58fc-dshm\") pod \"rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6\" (UID: \"cd9dde70-4a53-48eb-9699-238b760c58fc\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6"
Sep 10 21:04:52.571005 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.570959    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd9dde70-4a53-48eb-9699-238b760c58fc-tls-certs\") pod \"rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6\" (UID: \"cd9dde70-4a53-48eb-9699-238b760c58fc\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6"
Sep 10 21:04:52.571292 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.571276    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd9dde70-4a53-48eb-9699-238b760c58fc-model-cache\") pod \"rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6\" (UID: \"cd9dde70-4a53-48eb-9699-238b760c58fc\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6"
Sep 10 21:04:52.571570 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.571542    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd9dde70-4a53-48eb-9699-238b760c58fc-home\") pod \"rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6\" (UID: \"cd9dde70-4a53-48eb-9699-238b760c58fc\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6"
Sep 10 21:04:52.571817 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.571796    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd9dde70-4a53-48eb-9699-238b760c58fc-tmp-dir\") pod \"rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6\" (UID: \"cd9dde70-4a53-48eb-9699-238b760c58fc\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6"
Sep 10 21:04:52.573544 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.573524    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd9dde70-4a53-48eb-9699-238b760c58fc-tls-certs\") pod \"rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6\" (UID: \"cd9dde70-4a53-48eb-9699-238b760c58fc\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6"
Sep 10 21:04:52.573869 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.573850    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd9dde70-4a53-48eb-9699-238b760c58fc-dshm\") pod \"rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6\" (UID: \"cd9dde70-4a53-48eb-9699-238b760c58fc\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6"
Sep 10 21:04:52.579648 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.579630    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkzjk\" (UniqueName: \"kubernetes.io/projected/cd9dde70-4a53-48eb-9699-238b760c58fc-kube-api-access-fkzjk\") pod \"rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6\" (UID: \"cd9dde70-4a53-48eb-9699-238b760c58fc\") " pod="ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6"
Sep 10 21:04:52.618400 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.618369    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6"
Sep 10 21:04:52.888619 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:52.888594    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6"]
Sep 10 21:04:52.890861 ip-10-0-141-185 kubenswrapper[2564]: W0910 21:04:52.890833    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd9dde70_4a53_48eb_9699_238b760c58fc.slice/crio-57e621129cb33d86e6cbb1965998eb6b285ac646afc15388b28869e4928af0fc WatchSource:0}: Error finding container 57e621129cb33d86e6cbb1965998eb6b285ac646afc15388b28869e4928af0fc: Status 404 returned error can't find the container with id 57e621129cb33d86e6cbb1965998eb6b285ac646afc15388b28869e4928af0fc
Sep 10 21:04:53.346458 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:53.346418    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6" event={"ID":"cd9dde70-4a53-48eb-9699-238b760c58fc","Type":"ContainerStarted","Data":"655afeaf2f8227cfe86d2bd4948399e26af7550c9d235feb409518856a5a886b"}
Sep 10 21:04:53.346458 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:53.346468    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6" event={"ID":"cd9dde70-4a53-48eb-9699-238b760c58fc","Type":"ContainerStarted","Data":"57e621129cb33d86e6cbb1965998eb6b285ac646afc15388b28869e4928af0fc"}
Sep 10 21:04:53.369852 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:53.369800    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6" podStartSLOduration=1.369786501 podStartE2EDuration="1.369786501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:04:53.36729349 +0000 UTC m=+1733.332361685" watchObservedRunningTime="2026-09-10 21:04:53.369786501 +0000 UTC m=+1733.334854695"
Sep 10 21:04:56.237800 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:56.237765    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3bbb40-q8mtm"]
Sep 10 21:04:56.240722 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:56.240705    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3bbb40-q8mtm"
Sep 10 21:04:56.245640 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:56.245613    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plv2h\""
Sep 10 21:04:56.245770 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:56.245652    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:04:56.258046 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:56.258021    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3bbb40-q8mtm"]
Sep 10 21:04:56.401593 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:56.401557    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9821fcba-fec7-4cda-b8f6-99f0fc156220-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3bbb40-q8mtm\" (UID: \"9821fcba-fec7-4cda-b8f6-99f0fc156220\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3bbb40-q8mtm"
Sep 10 21:04:56.502034 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:56.501947    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9821fcba-fec7-4cda-b8f6-99f0fc156220-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3bbb40-q8mtm\" (UID: \"9821fcba-fec7-4cda-b8f6-99f0fc156220\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3bbb40-q8mtm"
Sep 10 21:04:56.502576 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:56.502558    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9821fcba-fec7-4cda-b8f6-99f0fc156220-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3bbb40-q8mtm\" (UID: \"9821fcba-fec7-4cda-b8f6-99f0fc156220\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3bbb40-q8mtm"
Sep 10 21:04:56.550699 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:56.550672    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3bbb40-q8mtm"
Sep 10 21:04:56.823542 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:56.823514    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3bbb40-q8mtm"]
Sep 10 21:04:56.825581 ip-10-0-141-185 kubenswrapper[2564]: W0910 21:04:56.825554    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9821fcba_fec7_4cda_b8f6_99f0fc156220.slice/crio-93b59b5aa6130869825450b7958a7e9101c61f2e8802c1319e747b18e42536ce WatchSource:0}: Error finding container 93b59b5aa6130869825450b7958a7e9101c61f2e8802c1319e747b18e42536ce: Status 404 returned error can't find the container with id 93b59b5aa6130869825450b7958a7e9101c61f2e8802c1319e747b18e42536ce
Sep 10 21:04:57.369766 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:57.369733    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3bbb40-q8mtm" event={"ID":"9821fcba-fec7-4cda-b8f6-99f0fc156220","Type":"ContainerStarted","Data":"9f21cd44c047c5c0b432aab111307a98dd760d5ca36da177857a4d5e6045484b"}
Sep 10 21:04:57.370134 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:57.369790    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3bbb40-q8mtm" event={"ID":"9821fcba-fec7-4cda-b8f6-99f0fc156220","Type":"ContainerStarted","Data":"93b59b5aa6130869825450b7958a7e9101c61f2e8802c1319e747b18e42536ce"}
Sep 10 21:04:57.393963 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:57.393914    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3bbb40-q8mtm" podStartSLOduration=1.393898523 podStartE2EDuration="1.393898523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:04:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:04:57.393353959 +0000 UTC m=+1737.358422153" watchObservedRunningTime="2026-09-10 21:04:57.393898523 +0000 UTC m=+1737.358966719"
Sep 10 21:04:58.378047 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:58.378014    2564 generic.go:356] "Generic (PLEG): container finished" podID="9821fcba-fec7-4cda-b8f6-99f0fc156220" containerID="9f21cd44c047c5c0b432aab111307a98dd760d5ca36da177857a4d5e6045484b" exitCode=0
Sep 10 21:04:58.378431 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:58.378109    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3bbb40-q8mtm" event={"ID":"9821fcba-fec7-4cda-b8f6-99f0fc156220","Type":"ContainerDied","Data":"9f21cd44c047c5c0b432aab111307a98dd760d5ca36da177857a4d5e6045484b"}
Sep 10 21:04:59.518435 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:59.518410    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3bbb40-q8mtm"
Sep 10 21:04:59.547550 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:59.547526    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9821fcba-fec7-4cda-b8f6-99f0fc156220-maas-api-service-ca\") pod \"9821fcba-fec7-4cda-b8f6-99f0fc156220\" (UID: \"9821fcba-fec7-4cda-b8f6-99f0fc156220\") "
Sep 10 21:04:59.547837 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:59.547815    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9821fcba-fec7-4cda-b8f6-99f0fc156220-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9821fcba-fec7-4cda-b8f6-99f0fc156220" (UID: "9821fcba-fec7-4cda-b8f6-99f0fc156220"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:04:59.648153 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:04:59.648085    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9821fcba-fec7-4cda-b8f6-99f0fc156220-maas-api-service-ca\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:05:00.393620 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:00.393589    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3bbb40-q8mtm"
Sep 10 21:05:00.393783 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:00.393617    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3bbb40-q8mtm" event={"ID":"9821fcba-fec7-4cda-b8f6-99f0fc156220","Type":"ContainerDied","Data":"93b59b5aa6130869825450b7958a7e9101c61f2e8802c1319e747b18e42536ce"}
Sep 10 21:05:00.393783 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:00.393665    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93b59b5aa6130869825450b7958a7e9101c61f2e8802c1319e747b18e42536ce"
Sep 10 21:05:02.618962 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:02.618929    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6"
Sep 10 21:05:02.619327 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:02.618974    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6"
Sep 10 21:05:02.632473 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:02.632449    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6"
Sep 10 21:05:03.423369 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:03.423338    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6"
Sep 10 21:05:06.858146 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:06.858111    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bbb40-29817900-22lht"]
Sep 10 21:05:06.861035 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:06.861010    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bbb40-29817900-22lht"]
Sep 10 21:05:06.891035 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:06.891005    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bbb40-848bbfbcb6-k4cgb"]
Sep 10 21:05:06.891246 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:06.891222    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bbb40-848bbfbcb6-k4cgb" podUID="7dae9e7e-4e67-415f-a209-47f9fd57281c" containerName="payload-pre-processing" containerID="cri-o://9dba583fd0ab3965adb109d9d4926bcabd77d032de476957f433200c29e2b33b" gracePeriod=30
Sep 10 21:05:07.092362 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:07.092328    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3bbb40-q8mtm"]
Sep 10 21:05:07.095672 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:07.095634    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3bbb40-q8mtm"]
Sep 10 21:05:07.148560 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:07.148536    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bbb40-848bbfbcb6-k4cgb"
Sep 10 21:05:07.202969 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:07.202943    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvw9h\" (UniqueName: \"kubernetes.io/projected/7dae9e7e-4e67-415f-a209-47f9fd57281c-kube-api-access-hvw9h\") pod \"7dae9e7e-4e67-415f-a209-47f9fd57281c\" (UID: \"7dae9e7e-4e67-415f-a209-47f9fd57281c\") "
Sep 10 21:05:07.203094 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:07.202993    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dae9e7e-4e67-415f-a209-47f9fd57281c-plugins-config-volume\") pod \"7dae9e7e-4e67-415f-a209-47f9fd57281c\" (UID: \"7dae9e7e-4e67-415f-a209-47f9fd57281c\") "
Sep 10 21:05:07.203416 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:07.203395    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dae9e7e-4e67-415f-a209-47f9fd57281c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7dae9e7e-4e67-415f-a209-47f9fd57281c" (UID: "7dae9e7e-4e67-415f-a209-47f9fd57281c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:05:07.204940 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:07.204910    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dae9e7e-4e67-415f-a209-47f9fd57281c-kube-api-access-hvw9h" (OuterVolumeSpecName: "kube-api-access-hvw9h") pod "7dae9e7e-4e67-415f-a209-47f9fd57281c" (UID: "7dae9e7e-4e67-415f-a209-47f9fd57281c"). InnerVolumeSpecName "kube-api-access-hvw9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:05:07.304044 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:07.304016    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvw9h\" (UniqueName: \"kubernetes.io/projected/7dae9e7e-4e67-415f-a209-47f9fd57281c-kube-api-access-hvw9h\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:05:07.304044 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:07.304042    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dae9e7e-4e67-415f-a209-47f9fd57281c-plugins-config-volume\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:05:07.435227 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:07.435147    2564 generic.go:356] "Generic (PLEG): container finished" podID="7dae9e7e-4e67-415f-a209-47f9fd57281c" containerID="9dba583fd0ab3965adb109d9d4926bcabd77d032de476957f433200c29e2b33b" exitCode=0
Sep 10 21:05:07.435227 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:07.435219    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bbb40-848bbfbcb6-k4cgb"
Sep 10 21:05:07.435376 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:07.435227    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bbb40-848bbfbcb6-k4cgb" event={"ID":"7dae9e7e-4e67-415f-a209-47f9fd57281c","Type":"ContainerDied","Data":"9dba583fd0ab3965adb109d9d4926bcabd77d032de476957f433200c29e2b33b"}
Sep 10 21:05:07.435376 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:07.435268    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bbb40-848bbfbcb6-k4cgb" event={"ID":"7dae9e7e-4e67-415f-a209-47f9fd57281c","Type":"ContainerDied","Data":"8617bc01f0ccb8bf9b1ed57146aa3270d2b8d93581c038d273d22af34f3a49c7"}
Sep 10 21:05:07.435376 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:07.435289    2564 scope.go:117] "RemoveContainer" containerID="9dba583fd0ab3965adb109d9d4926bcabd77d032de476957f433200c29e2b33b"
Sep 10 21:05:07.444807 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:07.444789    2564 scope.go:117] "RemoveContainer" containerID="9dba583fd0ab3965adb109d9d4926bcabd77d032de476957f433200c29e2b33b"
Sep 10 21:05:07.445048 ip-10-0-141-185 kubenswrapper[2564]: E0910 21:05:07.445025    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dba583fd0ab3965adb109d9d4926bcabd77d032de476957f433200c29e2b33b\": container with ID starting with 9dba583fd0ab3965adb109d9d4926bcabd77d032de476957f433200c29e2b33b not found: ID does not exist" containerID="9dba583fd0ab3965adb109d9d4926bcabd77d032de476957f433200c29e2b33b"
Sep 10 21:05:07.445120 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:07.445060    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dba583fd0ab3965adb109d9d4926bcabd77d032de476957f433200c29e2b33b"} err="failed to get container status \"9dba583fd0ab3965adb109d9d4926bcabd77d032de476957f433200c29e2b33b\": rpc error: code = NotFound desc = could not find container \"9dba583fd0ab3965adb109d9d4926bcabd77d032de476957f433200c29e2b33b\": container with ID starting with 9dba583fd0ab3965adb109d9d4926bcabd77d032de476957f433200c29e2b33b not found: ID does not exist"
Sep 10 21:05:07.468335 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:07.468308    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bbb40-848bbfbcb6-k4cgb"]
Sep 10 21:05:07.489889 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:07.485443    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bbb40-848bbfbcb6-k4cgb"]
Sep 10 21:05:08.539937 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:08.539907    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fa5c29a-1c7a-46a1-809c-269ca28a4057" path="/var/lib/kubelet/pods/2fa5c29a-1c7a-46a1-809c-269ca28a4057/volumes"
Sep 10 21:05:08.540341 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:08.540235    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dae9e7e-4e67-415f-a209-47f9fd57281c" path="/var/lib/kubelet/pods/7dae9e7e-4e67-415f-a209-47f9fd57281c/volumes"
Sep 10 21:05:08.540539 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:08.540526    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9821fcba-fec7-4cda-b8f6-99f0fc156220" path="/var/lib/kubelet/pods/9821fcba-fec7-4cda-b8f6-99f0fc156220/volumes"
Sep 10 21:05:13.916333 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:13.916292    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"]
Sep 10 21:05:13.916766 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:13.916597    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn" podUID="c328c839-e286-49b2-ad8b-3a472c32d624" containerName="istio-proxy" containerID="cri-o://dc7e19fb7a49a523009afc4b620afee54d79f08e3f90c8b838e1fb3998ce1812" gracePeriod=30
Sep 10 21:05:19.156794 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.156770    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 21:05:19.193769 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.193706    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzpc6\" (UniqueName: \"kubernetes.io/projected/c328c839-e286-49b2-ad8b-3a472c32d624-kube-api-access-bzpc6\") pod \"c328c839-e286-49b2-ad8b-3a472c32d624\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") "
Sep 10 21:05:19.193769 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.193738    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c328c839-e286-49b2-ad8b-3a472c32d624-istio-token\") pod \"c328c839-e286-49b2-ad8b-3a472c32d624\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") "
Sep 10 21:05:19.193957 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.193770    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c328c839-e286-49b2-ad8b-3a472c32d624-credential-socket\") pod \"c328c839-e286-49b2-ad8b-3a472c32d624\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") "
Sep 10 21:05:19.193957 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.193797    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c328c839-e286-49b2-ad8b-3a472c32d624-workload-socket\") pod \"c328c839-e286-49b2-ad8b-3a472c32d624\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") "
Sep 10 21:05:19.193957 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.193893    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c328c839-e286-49b2-ad8b-3a472c32d624-istio-podinfo\") pod \"c328c839-e286-49b2-ad8b-3a472c32d624\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") "
Sep 10 21:05:19.193957 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.193939    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c328c839-e286-49b2-ad8b-3a472c32d624-workload-certs\") pod \"c328c839-e286-49b2-ad8b-3a472c32d624\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") "
Sep 10 21:05:19.194199 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.194012    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c328c839-e286-49b2-ad8b-3a472c32d624-istio-data\") pod \"c328c839-e286-49b2-ad8b-3a472c32d624\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") "
Sep 10 21:05:19.194199 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.194092    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c328c839-e286-49b2-ad8b-3a472c32d624-istio-envoy\") pod \"c328c839-e286-49b2-ad8b-3a472c32d624\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") "
Sep 10 21:05:19.194199 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.194078    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c328c839-e286-49b2-ad8b-3a472c32d624-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c328c839-e286-49b2-ad8b-3a472c32d624" (UID: "c328c839-e286-49b2-ad8b-3a472c32d624"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:05:19.194199 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.194116    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c328c839-e286-49b2-ad8b-3a472c32d624-istiod-ca-cert\") pod \"c328c839-e286-49b2-ad8b-3a472c32d624\" (UID: \"c328c839-e286-49b2-ad8b-3a472c32d624\") "
Sep 10 21:05:19.194415 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.194313    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c328c839-e286-49b2-ad8b-3a472c32d624-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c328c839-e286-49b2-ad8b-3a472c32d624" (UID: "c328c839-e286-49b2-ad8b-3a472c32d624"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:05:19.194415 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.194399    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c328c839-e286-49b2-ad8b-3a472c32d624-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c328c839-e286-49b2-ad8b-3a472c32d624" (UID: "c328c839-e286-49b2-ad8b-3a472c32d624"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:05:19.194527 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.194496    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c328c839-e286-49b2-ad8b-3a472c32d624-credential-socket\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:05:19.194527 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.194515    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c328c839-e286-49b2-ad8b-3a472c32d624-workload-socket\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:05:19.194632 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.194531    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c328c839-e286-49b2-ad8b-3a472c32d624-workload-certs\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:05:19.194632 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.194599    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c328c839-e286-49b2-ad8b-3a472c32d624-istio-data" (OuterVolumeSpecName: "istio-data") pod "c328c839-e286-49b2-ad8b-3a472c32d624" (UID: "c328c839-e286-49b2-ad8b-3a472c32d624"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:05:19.194747 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.194696    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c328c839-e286-49b2-ad8b-3a472c32d624-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c328c839-e286-49b2-ad8b-3a472c32d624" (UID: "c328c839-e286-49b2-ad8b-3a472c32d624"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:05:19.196024 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.195999    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c328c839-e286-49b2-ad8b-3a472c32d624-istio-token" (OuterVolumeSpecName: "istio-token") pod "c328c839-e286-49b2-ad8b-3a472c32d624" (UID: "c328c839-e286-49b2-ad8b-3a472c32d624"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:05:19.196024 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.196010    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c328c839-e286-49b2-ad8b-3a472c32d624-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c328c839-e286-49b2-ad8b-3a472c32d624" (UID: "c328c839-e286-49b2-ad8b-3a472c32d624"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:05:19.196230 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.196055    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c328c839-e286-49b2-ad8b-3a472c32d624-kube-api-access-bzpc6" (OuterVolumeSpecName: "kube-api-access-bzpc6") pod "c328c839-e286-49b2-ad8b-3a472c32d624" (UID: "c328c839-e286-49b2-ad8b-3a472c32d624"). InnerVolumeSpecName "kube-api-access-bzpc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:05:19.196783 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.196762    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c328c839-e286-49b2-ad8b-3a472c32d624-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c328c839-e286-49b2-ad8b-3a472c32d624" (UID: "c328c839-e286-49b2-ad8b-3a472c32d624"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:05:19.301029 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.295689    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzpc6\" (UniqueName: \"kubernetes.io/projected/c328c839-e286-49b2-ad8b-3a472c32d624-kube-api-access-bzpc6\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:05:19.301029 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.295720    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c328c839-e286-49b2-ad8b-3a472c32d624-istio-token\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:05:19.301029 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.295736    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c328c839-e286-49b2-ad8b-3a472c32d624-istio-podinfo\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:05:19.301029 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.295749    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c328c839-e286-49b2-ad8b-3a472c32d624-istio-data\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:05:19.301029 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.295768    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c328c839-e286-49b2-ad8b-3a472c32d624-istio-envoy\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:05:19.301029 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.295781    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c328c839-e286-49b2-ad8b-3a472c32d624-istiod-ca-cert\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:05:19.504398 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.504315    2564 generic.go:356] "Generic (PLEG): container finished" podID="c328c839-e286-49b2-ad8b-3a472c32d624" containerID="dc7e19fb7a49a523009afc4b620afee54d79f08e3f90c8b838e1fb3998ce1812" exitCode=0
Sep 10 21:05:19.504398 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.504378    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"
Sep 10 21:05:19.504555 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.504392    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn" event={"ID":"c328c839-e286-49b2-ad8b-3a472c32d624","Type":"ContainerDied","Data":"dc7e19fb7a49a523009afc4b620afee54d79f08e3f90c8b838e1fb3998ce1812"}
Sep 10 21:05:19.504555 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.504466    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn" event={"ID":"c328c839-e286-49b2-ad8b-3a472c32d624","Type":"ContainerDied","Data":"df294fd3f451eff07008f612825cc38a074d7c99ce25e495bc68dfac0c0921e2"}
Sep 10 21:05:19.504555 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.504489    2564 scope.go:117] "RemoveContainer" containerID="dc7e19fb7a49a523009afc4b620afee54d79f08e3f90c8b838e1fb3998ce1812"
Sep 10 21:05:19.514587 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.514569    2564 scope.go:117] "RemoveContainer" containerID="b6cb6654ae6bceed64e2024f25280c7d0798a208561ad3a62b89726823d00d1e"
Sep 10 21:05:19.522496 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.522478    2564 scope.go:117] "RemoveContainer" containerID="dc7e19fb7a49a523009afc4b620afee54d79f08e3f90c8b838e1fb3998ce1812"
Sep 10 21:05:19.522735 ip-10-0-141-185 kubenswrapper[2564]: E0910 21:05:19.522718    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc7e19fb7a49a523009afc4b620afee54d79f08e3f90c8b838e1fb3998ce1812\": container with ID starting with dc7e19fb7a49a523009afc4b620afee54d79f08e3f90c8b838e1fb3998ce1812 not found: ID does not exist" containerID="dc7e19fb7a49a523009afc4b620afee54d79f08e3f90c8b838e1fb3998ce1812"
Sep 10 21:05:19.522770 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.522745    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc7e19fb7a49a523009afc4b620afee54d79f08e3f90c8b838e1fb3998ce1812"} err="failed to get container status \"dc7e19fb7a49a523009afc4b620afee54d79f08e3f90c8b838e1fb3998ce1812\": rpc error: code = NotFound desc = could not find container \"dc7e19fb7a49a523009afc4b620afee54d79f08e3f90c8b838e1fb3998ce1812\": container with ID starting with dc7e19fb7a49a523009afc4b620afee54d79f08e3f90c8b838e1fb3998ce1812 not found: ID does not exist"
Sep 10 21:05:19.522770 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.522762    2564 scope.go:117] "RemoveContainer" containerID="b6cb6654ae6bceed64e2024f25280c7d0798a208561ad3a62b89726823d00d1e"
Sep 10 21:05:19.522997 ip-10-0-141-185 kubenswrapper[2564]: E0910 21:05:19.522981    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6cb6654ae6bceed64e2024f25280c7d0798a208561ad3a62b89726823d00d1e\": container with ID starting with b6cb6654ae6bceed64e2024f25280c7d0798a208561ad3a62b89726823d00d1e not found: ID does not exist" containerID="b6cb6654ae6bceed64e2024f25280c7d0798a208561ad3a62b89726823d00d1e"
Sep 10 21:05:19.523036 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.523003    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6cb6654ae6bceed64e2024f25280c7d0798a208561ad3a62b89726823d00d1e"} err="failed to get container status \"b6cb6654ae6bceed64e2024f25280c7d0798a208561ad3a62b89726823d00d1e\": rpc error: code = NotFound desc = could not find container \"b6cb6654ae6bceed64e2024f25280c7d0798a208561ad3a62b89726823d00d1e\": container with ID starting with b6cb6654ae6bceed64e2024f25280c7d0798a208561ad3a62b89726823d00d1e not found: ID does not exist"
Sep 10 21:05:19.526519 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.526499    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"]
Sep 10 21:05:19.530007 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:19.529984    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3bbb40-openshift-default-7957f8b7c4-dn4mn"]
Sep 10 21:05:20.542155 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:20.542109    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c328c839-e286-49b2-ad8b-3a472c32d624" path="/var/lib/kubelet/pods/c328c839-e286-49b2-ad8b-3a472c32d624/volumes"
Sep 10 21:05:41.923146 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:41.923109    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd"]
Sep 10 21:05:42.979627 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:42.979589    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6"]
Sep 10 21:05:56.941810 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:56.941725    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd" podUID="4581ab35-d3ec-41db-a488-c9515df6afca" containerName="main" containerID="cri-o://ab70c22f1311debfd0a1635898235b54c23317e89030a0ef90ef253b4721557f" gracePeriod=45
Sep 10 21:05:57.186809 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:57.186780    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd"
Sep 10 21:05:57.286097 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:57.286036    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rpw8\" (UniqueName: \"kubernetes.io/projected/4581ab35-d3ec-41db-a488-c9515df6afca-kube-api-access-7rpw8\") pod \"4581ab35-d3ec-41db-a488-c9515df6afca\" (UID: \"4581ab35-d3ec-41db-a488-c9515df6afca\") "
Sep 10 21:05:57.286097 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:57.286079    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4581ab35-d3ec-41db-a488-c9515df6afca-dshm\") pod \"4581ab35-d3ec-41db-a488-c9515df6afca\" (UID: \"4581ab35-d3ec-41db-a488-c9515df6afca\") "
Sep 10 21:05:57.286282 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:57.286104    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4581ab35-d3ec-41db-a488-c9515df6afca-model-cache\") pod \"4581ab35-d3ec-41db-a488-c9515df6afca\" (UID: \"4581ab35-d3ec-41db-a488-c9515df6afca\") "
Sep 10 21:05:57.286282 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:57.286149    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4581ab35-d3ec-41db-a488-c9515df6afca-tls-certs\") pod \"4581ab35-d3ec-41db-a488-c9515df6afca\" (UID: \"4581ab35-d3ec-41db-a488-c9515df6afca\") "
Sep 10 21:05:57.286282 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:57.286211    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4581ab35-d3ec-41db-a488-c9515df6afca-tmp-dir\") pod \"4581ab35-d3ec-41db-a488-c9515df6afca\" (UID: \"4581ab35-d3ec-41db-a488-c9515df6afca\") "
Sep 10 21:05:57.286282 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:57.286252    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4581ab35-d3ec-41db-a488-c9515df6afca-home\") pod \"4581ab35-d3ec-41db-a488-c9515df6afca\" (UID: \"4581ab35-d3ec-41db-a488-c9515df6afca\") "
Sep 10 21:05:57.286549 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:57.286508    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4581ab35-d3ec-41db-a488-c9515df6afca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4581ab35-d3ec-41db-a488-c9515df6afca" (UID: "4581ab35-d3ec-41db-a488-c9515df6afca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:05:57.286672 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:57.286596    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4581ab35-d3ec-41db-a488-c9515df6afca-model-cache" (OuterVolumeSpecName: "model-cache") pod "4581ab35-d3ec-41db-a488-c9515df6afca" (UID: "4581ab35-d3ec-41db-a488-c9515df6afca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:05:57.286672 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:57.286651    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4581ab35-d3ec-41db-a488-c9515df6afca-home" (OuterVolumeSpecName: "home") pod "4581ab35-d3ec-41db-a488-c9515df6afca" (UID: "4581ab35-d3ec-41db-a488-c9515df6afca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:05:57.288362 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:57.288332    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4581ab35-d3ec-41db-a488-c9515df6afca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4581ab35-d3ec-41db-a488-c9515df6afca" (UID: "4581ab35-d3ec-41db-a488-c9515df6afca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:05:57.288449 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:57.288339    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4581ab35-d3ec-41db-a488-c9515df6afca-dshm" (OuterVolumeSpecName: "dshm") pod "4581ab35-d3ec-41db-a488-c9515df6afca" (UID: "4581ab35-d3ec-41db-a488-c9515df6afca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:05:57.288449 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:57.288411    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4581ab35-d3ec-41db-a488-c9515df6afca-kube-api-access-7rpw8" (OuterVolumeSpecName: "kube-api-access-7rpw8") pod "4581ab35-d3ec-41db-a488-c9515df6afca" (UID: "4581ab35-d3ec-41db-a488-c9515df6afca"). InnerVolumeSpecName "kube-api-access-7rpw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:05:57.387805 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:57.387784    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4581ab35-d3ec-41db-a488-c9515df6afca-dshm\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:05:57.387805 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:57.387804    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4581ab35-d3ec-41db-a488-c9515df6afca-model-cache\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:05:57.387931 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:57.387814    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4581ab35-d3ec-41db-a488-c9515df6afca-tls-certs\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:05:57.387931 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:57.387822    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4581ab35-d3ec-41db-a488-c9515df6afca-tmp-dir\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:05:57.387931 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:57.387830    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4581ab35-d3ec-41db-a488-c9515df6afca-home\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:05:57.387931 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:57.387838    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rpw8\" (UniqueName: \"kubernetes.io/projected/4581ab35-d3ec-41db-a488-c9515df6afca-kube-api-access-7rpw8\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:05:57.731883 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:57.731850    2564 generic.go:356] "Generic (PLEG): container finished" podID="4581ab35-d3ec-41db-a488-c9515df6afca" containerID="ab70c22f1311debfd0a1635898235b54c23317e89030a0ef90ef253b4721557f" exitCode=0
Sep 10 21:05:57.732008 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:57.731900    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd"
Sep 10 21:05:57.732008 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:57.731934    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd" event={"ID":"4581ab35-d3ec-41db-a488-c9515df6afca","Type":"ContainerDied","Data":"ab70c22f1311debfd0a1635898235b54c23317e89030a0ef90ef253b4721557f"}
Sep 10 21:05:57.732008 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:57.731986    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd" event={"ID":"4581ab35-d3ec-41db-a488-c9515df6afca","Type":"ContainerDied","Data":"bb0c18ce7c3d9622eb0d6c40bfa16974c15ff97594bf243ef62c2136b5c4ee74"}
Sep 10 21:05:57.732008 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:57.732004    2564 scope.go:117] "RemoveContainer" containerID="ab70c22f1311debfd0a1635898235b54c23317e89030a0ef90ef253b4721557f"
Sep 10 21:05:57.742086 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:57.742065    2564 scope.go:117] "RemoveContainer" containerID="ab70c22f1311debfd0a1635898235b54c23317e89030a0ef90ef253b4721557f"
Sep 10 21:05:57.742404 ip-10-0-141-185 kubenswrapper[2564]: E0910 21:05:57.742378    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab70c22f1311debfd0a1635898235b54c23317e89030a0ef90ef253b4721557f\": container with ID starting with ab70c22f1311debfd0a1635898235b54c23317e89030a0ef90ef253b4721557f not found: ID does not exist" containerID="ab70c22f1311debfd0a1635898235b54c23317e89030a0ef90ef253b4721557f"
Sep 10 21:05:57.742489 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:57.742411    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab70c22f1311debfd0a1635898235b54c23317e89030a0ef90ef253b4721557f"} err="failed to get container status \"ab70c22f1311debfd0a1635898235b54c23317e89030a0ef90ef253b4721557f\": rpc error: code = NotFound desc = could not find container \"ab70c22f1311debfd0a1635898235b54c23317e89030a0ef90ef253b4721557f\": container with ID starting with ab70c22f1311debfd0a1635898235b54c23317e89030a0ef90ef253b4721557f not found: ID does not exist"
Sep 10 21:05:57.753522 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:57.753498    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd"]
Sep 10 21:05:57.757308 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:57.757285    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3fc2b8/rate-test-model-3fc2b8-kserve-bddc577f5-74znd"]
Sep 10 21:05:57.998787 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:57.998707    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6" podUID="cd9dde70-4a53-48eb-9699-238b760c58fc" containerName="main" containerID="cri-o://655afeaf2f8227cfe86d2bd4948399e26af7550c9d235feb409518856a5a886b" gracePeriod=45
Sep 10 21:05:58.250349 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:58.250290    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6"
Sep 10 21:05:58.394966 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:58.394937    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd9dde70-4a53-48eb-9699-238b760c58fc-model-cache\") pod \"cd9dde70-4a53-48eb-9699-238b760c58fc\" (UID: \"cd9dde70-4a53-48eb-9699-238b760c58fc\") "
Sep 10 21:05:58.395100 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:58.394981    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd9dde70-4a53-48eb-9699-238b760c58fc-dshm\") pod \"cd9dde70-4a53-48eb-9699-238b760c58fc\" (UID: \"cd9dde70-4a53-48eb-9699-238b760c58fc\") "
Sep 10 21:05:58.395100 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:58.395004    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd9dde70-4a53-48eb-9699-238b760c58fc-tmp-dir\") pod \"cd9dde70-4a53-48eb-9699-238b760c58fc\" (UID: \"cd9dde70-4a53-48eb-9699-238b760c58fc\") "
Sep 10 21:05:58.395100 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:58.395054    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd9dde70-4a53-48eb-9699-238b760c58fc-home\") pod \"cd9dde70-4a53-48eb-9699-238b760c58fc\" (UID: \"cd9dde70-4a53-48eb-9699-238b760c58fc\") "
Sep 10 21:05:58.395292 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:58.395125    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkzjk\" (UniqueName: \"kubernetes.io/projected/cd9dde70-4a53-48eb-9699-238b760c58fc-kube-api-access-fkzjk\") pod \"cd9dde70-4a53-48eb-9699-238b760c58fc\" (UID: \"cd9dde70-4a53-48eb-9699-238b760c58fc\") "
Sep 10 21:05:58.395292 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:58.395162    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd9dde70-4a53-48eb-9699-238b760c58fc-tls-certs\") pod \"cd9dde70-4a53-48eb-9699-238b760c58fc\" (UID: \"cd9dde70-4a53-48eb-9699-238b760c58fc\") "
Sep 10 21:05:58.395292 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:58.395225    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd9dde70-4a53-48eb-9699-238b760c58fc-model-cache" (OuterVolumeSpecName: "model-cache") pod "cd9dde70-4a53-48eb-9699-238b760c58fc" (UID: "cd9dde70-4a53-48eb-9699-238b760c58fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:05:58.395455 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:58.395313    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd9dde70-4a53-48eb-9699-238b760c58fc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cd9dde70-4a53-48eb-9699-238b760c58fc" (UID: "cd9dde70-4a53-48eb-9699-238b760c58fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:05:58.395455 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:58.395403    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd9dde70-4a53-48eb-9699-238b760c58fc-home" (OuterVolumeSpecName: "home") pod "cd9dde70-4a53-48eb-9699-238b760c58fc" (UID: "cd9dde70-4a53-48eb-9699-238b760c58fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:05:58.395455 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:58.395424    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd9dde70-4a53-48eb-9699-238b760c58fc-model-cache\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:05:58.395606 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:58.395460    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd9dde70-4a53-48eb-9699-238b760c58fc-tmp-dir\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:05:58.397068 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:58.397040    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd9dde70-4a53-48eb-9699-238b760c58fc-dshm" (OuterVolumeSpecName: "dshm") pod "cd9dde70-4a53-48eb-9699-238b760c58fc" (UID: "cd9dde70-4a53-48eb-9699-238b760c58fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:05:58.397194 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:58.397099    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd9dde70-4a53-48eb-9699-238b760c58fc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cd9dde70-4a53-48eb-9699-238b760c58fc" (UID: "cd9dde70-4a53-48eb-9699-238b760c58fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:05:58.397194 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:58.397136    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd9dde70-4a53-48eb-9699-238b760c58fc-kube-api-access-fkzjk" (OuterVolumeSpecName: "kube-api-access-fkzjk") pod "cd9dde70-4a53-48eb-9699-238b760c58fc" (UID: "cd9dde70-4a53-48eb-9699-238b760c58fc"). InnerVolumeSpecName "kube-api-access-fkzjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:05:58.496711 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:58.496688    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd9dde70-4a53-48eb-9699-238b760c58fc-dshm\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:05:58.496711 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:58.496707    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd9dde70-4a53-48eb-9699-238b760c58fc-home\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:05:58.496849 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:58.496717    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkzjk\" (UniqueName: \"kubernetes.io/projected/cd9dde70-4a53-48eb-9699-238b760c58fc-kube-api-access-fkzjk\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:05:58.496849 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:58.496728    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd9dde70-4a53-48eb-9699-238b760c58fc-tls-certs\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:05:58.540916 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:58.540861    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4581ab35-d3ec-41db-a488-c9515df6afca" path="/var/lib/kubelet/pods/4581ab35-d3ec-41db-a488-c9515df6afca/volumes"
Sep 10 21:05:58.740106 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:58.740079    2564 generic.go:356] "Generic (PLEG): container finished" podID="cd9dde70-4a53-48eb-9699-238b760c58fc" containerID="655afeaf2f8227cfe86d2bd4948399e26af7550c9d235feb409518856a5a886b" exitCode=0
Sep 10 21:05:58.740249 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:58.740125    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6"
Sep 10 21:05:58.740249 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:58.740162    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6" event={"ID":"cd9dde70-4a53-48eb-9699-238b760c58fc","Type":"ContainerDied","Data":"655afeaf2f8227cfe86d2bd4948399e26af7550c9d235feb409518856a5a886b"}
Sep 10 21:05:58.740249 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:58.740226    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6" event={"ID":"cd9dde70-4a53-48eb-9699-238b760c58fc","Type":"ContainerDied","Data":"57e621129cb33d86e6cbb1965998eb6b285ac646afc15388b28869e4928af0fc"}
Sep 10 21:05:58.740416 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:58.740252    2564 scope.go:117] "RemoveContainer" containerID="655afeaf2f8227cfe86d2bd4948399e26af7550c9d235feb409518856a5a886b"
Sep 10 21:05:58.751296 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:58.751275    2564 scope.go:117] "RemoveContainer" containerID="655afeaf2f8227cfe86d2bd4948399e26af7550c9d235feb409518856a5a886b"
Sep 10 21:05:58.751620 ip-10-0-141-185 kubenswrapper[2564]: E0910 21:05:58.751598    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"655afeaf2f8227cfe86d2bd4948399e26af7550c9d235feb409518856a5a886b\": container with ID starting with 655afeaf2f8227cfe86d2bd4948399e26af7550c9d235feb409518856a5a886b not found: ID does not exist" containerID="655afeaf2f8227cfe86d2bd4948399e26af7550c9d235feb409518856a5a886b"
Sep 10 21:05:58.751708 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:58.751628    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"655afeaf2f8227cfe86d2bd4948399e26af7550c9d235feb409518856a5a886b"} err="failed to get container status \"655afeaf2f8227cfe86d2bd4948399e26af7550c9d235feb409518856a5a886b\": rpc error: code = NotFound desc = could not find container \"655afeaf2f8227cfe86d2bd4948399e26af7550c9d235feb409518856a5a886b\": container with ID starting with 655afeaf2f8227cfe86d2bd4948399e26af7550c9d235feb409518856a5a886b not found: ID does not exist"
Sep 10 21:05:58.759241 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:58.759217    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6"]
Sep 10 21:05:58.761709 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:05:58.761688    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4fe4f/rate-test-model-f4fe4f-kserve-7c65f7dcd7-nnpz6"]
Sep 10 21:06:00.540273 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:00.540234    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd9dde70-4a53-48eb-9699-238b760c58fc" path="/var/lib/kubelet/pods/cd9dde70-4a53-48eb-9699-238b760c58fc/volumes"
Sep 10 21:06:00.763393 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:00.763361    2564 scope.go:117] "RemoveContainer" containerID="78f206d30c974fc4803652e0d8c8280115cd191f0b22488c8d26a1b23d9c610b"
Sep 10 21:06:00.771850 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:00.771834    2564 scope.go:117] "RemoveContainer" containerID="6ef1160eba92ce11cdeca69cb0585fc5b3bab23ce1a649216a1e19cf29357869"
Sep 10 21:06:12.541327 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.541292    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"]
Sep 10 21:06:12.541936 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.541654    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4581ab35-d3ec-41db-a488-c9515df6afca" containerName="main"
Sep 10 21:06:12.541936 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.541667    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4581ab35-d3ec-41db-a488-c9515df6afca" containerName="main"
Sep 10 21:06:12.541936 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.541676    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c328c839-e286-49b2-ad8b-3a472c32d624" containerName="istio-proxy"
Sep 10 21:06:12.541936 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.541682    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c328c839-e286-49b2-ad8b-3a472c32d624" containerName="istio-proxy"
Sep 10 21:06:12.541936 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.541695    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c328c839-e286-49b2-ad8b-3a472c32d624" containerName="istio-proxy"
Sep 10 21:06:12.541936 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.541705    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c328c839-e286-49b2-ad8b-3a472c32d624" containerName="istio-proxy"
Sep 10 21:06:12.541936 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.541716    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd9dde70-4a53-48eb-9699-238b760c58fc" containerName="main"
Sep 10 21:06:12.541936 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.541723    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd9dde70-4a53-48eb-9699-238b760c58fc" containerName="main"
Sep 10 21:06:12.541936 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.541739    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9821fcba-fec7-4cda-b8f6-99f0fc156220" containerName="revoke-keys"
Sep 10 21:06:12.541936 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.541744    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9821fcba-fec7-4cda-b8f6-99f0fc156220" containerName="revoke-keys"
Sep 10 21:06:12.541936 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.541751    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dae9e7e-4e67-415f-a209-47f9fd57281c" containerName="payload-pre-processing"
Sep 10 21:06:12.541936 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.541756    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dae9e7e-4e67-415f-a209-47f9fd57281c" containerName="payload-pre-processing"
Sep 10 21:06:12.541936 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.541816    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd9dde70-4a53-48eb-9699-238b760c58fc" containerName="main"
Sep 10 21:06:12.541936 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.541824    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9821fcba-fec7-4cda-b8f6-99f0fc156220" containerName="revoke-keys"
Sep 10 21:06:12.541936 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.541829    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c328c839-e286-49b2-ad8b-3a472c32d624" containerName="istio-proxy"
Sep 10 21:06:12.541936 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.541835    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dae9e7e-4e67-415f-a209-47f9fd57281c" containerName="payload-pre-processing"
Sep 10 21:06:12.541936 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.541841    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4581ab35-d3ec-41db-a488-c9515df6afca" containerName="main"
Sep 10 21:06:12.541936 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.541848    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c328c839-e286-49b2-ad8b-3a472c32d624" containerName="istio-proxy"
Sep 10 21:06:12.544466 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.544451    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:12.546640 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.546619    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-18b9ac-openshift-default-dockercfg-bhkz4\""
Sep 10 21:06:12.556208 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.556184    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"]
Sep 10 21:06:12.703860 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.703825    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e013fecc-7d3c-48f6-a7ef-c5695f284b39-istiod-ca-cert\") pod \"e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") " pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:12.703860 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.703865    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e013fecc-7d3c-48f6-a7ef-c5695f284b39-istio-podinfo\") pod \"e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") " pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:12.704074 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.703893    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e013fecc-7d3c-48f6-a7ef-c5695f284b39-istio-envoy\") pod \"e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") " pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:12.704074 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.703963    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52kvh\" (UniqueName: \"kubernetes.io/projected/e013fecc-7d3c-48f6-a7ef-c5695f284b39-kube-api-access-52kvh\") pod \"e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") " pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:12.704074 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.703995    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e013fecc-7d3c-48f6-a7ef-c5695f284b39-workload-socket\") pod \"e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") " pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:12.704074 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.704023    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e013fecc-7d3c-48f6-a7ef-c5695f284b39-istio-token\") pod \"e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") " pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:12.704074 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.704040    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e013fecc-7d3c-48f6-a7ef-c5695f284b39-credential-socket\") pod \"e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") " pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:12.704286 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.704137    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e013fecc-7d3c-48f6-a7ef-c5695f284b39-workload-certs\") pod \"e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") " pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:12.704286 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.704215    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e013fecc-7d3c-48f6-a7ef-c5695f284b39-istio-data\") pod \"e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") " pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:12.805326 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.805251    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e013fecc-7d3c-48f6-a7ef-c5695f284b39-workload-socket\") pod \"e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") " pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:12.805326 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.805289    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e013fecc-7d3c-48f6-a7ef-c5695f284b39-istio-token\") pod \"e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") " pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:12.805326 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.805308    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e013fecc-7d3c-48f6-a7ef-c5695f284b39-credential-socket\") pod \"e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") " pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:12.805583 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.805329    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e013fecc-7d3c-48f6-a7ef-c5695f284b39-workload-certs\") pod \"e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") " pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:12.805583 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.805351    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e013fecc-7d3c-48f6-a7ef-c5695f284b39-istio-data\") pod \"e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") " pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:12.805583 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.805385    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e013fecc-7d3c-48f6-a7ef-c5695f284b39-istiod-ca-cert\") pod \"e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") " pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:12.805583 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.805408    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e013fecc-7d3c-48f6-a7ef-c5695f284b39-istio-podinfo\") pod \"e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") " pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:12.805583 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.805446    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e013fecc-7d3c-48f6-a7ef-c5695f284b39-istio-envoy\") pod \"e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") " pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:12.805583 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.805485    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52kvh\" (UniqueName: \"kubernetes.io/projected/e013fecc-7d3c-48f6-a7ef-c5695f284b39-kube-api-access-52kvh\") pod \"e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") " pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:12.805880 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.805672    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e013fecc-7d3c-48f6-a7ef-c5695f284b39-credential-socket\") pod \"e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") " pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:12.805880 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.805823    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e013fecc-7d3c-48f6-a7ef-c5695f284b39-istio-data\") pod \"e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") " pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:12.805987 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.805889    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e013fecc-7d3c-48f6-a7ef-c5695f284b39-workload-certs\") pod \"e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") " pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:12.806030 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.805990    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e013fecc-7d3c-48f6-a7ef-c5695f284b39-workload-socket\") pod \"e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") " pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:12.806180 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.806142    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e013fecc-7d3c-48f6-a7ef-c5695f284b39-istiod-ca-cert\") pod \"e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") " pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:12.807770 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.807735    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e013fecc-7d3c-48f6-a7ef-c5695f284b39-istio-envoy\") pod \"e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") " pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:12.807995 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.807972    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e013fecc-7d3c-48f6-a7ef-c5695f284b39-istio-podinfo\") pod \"e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") " pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:12.815037 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.815015    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e013fecc-7d3c-48f6-a7ef-c5695f284b39-istio-token\") pod \"e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") " pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:12.817555 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.817536    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52kvh\" (UniqueName: \"kubernetes.io/projected/e013fecc-7d3c-48f6-a7ef-c5695f284b39-kube-api-access-52kvh\") pod \"e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") " pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:12.857256 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:12.857226    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:13.148040 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:13.148015    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"]
Sep 10 21:06:13.150425 ip-10-0-141-185 kubenswrapper[2564]: W0910 21:06:13.150397    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode013fecc_7d3c_48f6_a7ef_c5695f284b39.slice/crio-923b561ecb424b7d790da5343b6137da354f748703df674ef937419999a8111e WatchSource:0}: Error finding container 923b561ecb424b7d790da5343b6137da354f748703df674ef937419999a8111e: Status 404 returned error can't find the container with id 923b561ecb424b7d790da5343b6137da354f748703df674ef937419999a8111e
Sep 10 21:06:13.152488 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:13.152456    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:06:13.152568 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:13.152519    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:06:13.152568 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:13.152551    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:06:13.830475 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:13.830435    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z" event={"ID":"e013fecc-7d3c-48f6-a7ef-c5695f284b39","Type":"ContainerStarted","Data":"425d4f89c49ccead979d48446fc28034ae9e2c489360e28f4b17e97ab18ec058"}
Sep 10 21:06:13.830863 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:13.830497    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z" event={"ID":"e013fecc-7d3c-48f6-a7ef-c5695f284b39","Type":"ContainerStarted","Data":"923b561ecb424b7d790da5343b6137da354f748703df674ef937419999a8111e"}
Sep 10 21:06:13.851201 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:13.851137    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z" podStartSLOduration=1.851121589 podStartE2EDuration="1.851121589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:06:13.84991748 +0000 UTC m=+1813.814985676" watchObservedRunningTime="2026-09-10 21:06:13.851121589 +0000 UTC m=+1813.816189784"
Sep 10 21:06:13.857829 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:13.857807    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:14.862586 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:14.862556    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:15.842994 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:15.842957    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:15.843891 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:15.843844    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:17.787751 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:17.787712    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-18b9ac-6cf5b5c9f-2whf7"]
Sep 10 21:06:17.791432 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:17.791409    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-18b9ac-6cf5b5c9f-2whf7"
Sep 10 21:06:17.793605 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:17.793586    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-18b9ac\""
Sep 10 21:06:17.793815 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:17.793799    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-18b9ac-dockercfg-58b76\""
Sep 10 21:06:17.801995 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:17.801975    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-18b9ac-6cf5b5c9f-2whf7"]
Sep 10 21:06:17.827193 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:17.827156    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-18b9ac-5f79c684b-6hxxz"]
Sep 10 21:06:17.830348 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:17.830334    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18b9ac-5f79c684b-6hxxz"
Sep 10 21:06:17.840289 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:17.840268    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-18b9ac-5f79c684b-6hxxz"]
Sep 10 21:06:17.948229 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:17.948188    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9gwr\" (UniqueName: \"kubernetes.io/projected/e10c2e9e-56e7-4639-bda8-b5c982287310-kube-api-access-w9gwr\") pod \"payload-pre-processing-e2e-ipp-b-18b9ac-5f79c684b-6hxxz\" (UID: \"e10c2e9e-56e7-4639-bda8-b5c982287310\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18b9ac-5f79c684b-6hxxz"
Sep 10 21:06:17.948419 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:17.948237    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/170ee806-8562-4304-8eb4-9ce73100f449-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-18b9ac-6cf5b5c9f-2whf7\" (UID: \"170ee806-8562-4304-8eb4-9ce73100f449\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-18b9ac-6cf5b5c9f-2whf7"
Sep 10 21:06:17.948419 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:17.948395    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sklq9\" (UniqueName: \"kubernetes.io/projected/170ee806-8562-4304-8eb4-9ce73100f449-kube-api-access-sklq9\") pod \"payload-processing-e2e-ipp-b-18b9ac-6cf5b5c9f-2whf7\" (UID: \"170ee806-8562-4304-8eb4-9ce73100f449\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-18b9ac-6cf5b5c9f-2whf7"
Sep 10 21:06:17.948547 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:17.948484    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e10c2e9e-56e7-4639-bda8-b5c982287310-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-18b9ac-5f79c684b-6hxxz\" (UID: \"e10c2e9e-56e7-4639-bda8-b5c982287310\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18b9ac-5f79c684b-6hxxz"
Sep 10 21:06:18.049560 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:18.049492    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sklq9\" (UniqueName: \"kubernetes.io/projected/170ee806-8562-4304-8eb4-9ce73100f449-kube-api-access-sklq9\") pod \"payload-processing-e2e-ipp-b-18b9ac-6cf5b5c9f-2whf7\" (UID: \"170ee806-8562-4304-8eb4-9ce73100f449\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-18b9ac-6cf5b5c9f-2whf7"
Sep 10 21:06:18.049560 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:18.049525    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e10c2e9e-56e7-4639-bda8-b5c982287310-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-18b9ac-5f79c684b-6hxxz\" (UID: \"e10c2e9e-56e7-4639-bda8-b5c982287310\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18b9ac-5f79c684b-6hxxz"
Sep 10 21:06:18.049560 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:18.049560    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9gwr\" (UniqueName: \"kubernetes.io/projected/e10c2e9e-56e7-4639-bda8-b5c982287310-kube-api-access-w9gwr\") pod \"payload-pre-processing-e2e-ipp-b-18b9ac-5f79c684b-6hxxz\" (UID: \"e10c2e9e-56e7-4639-bda8-b5c982287310\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18b9ac-5f79c684b-6hxxz"
Sep 10 21:06:18.049784 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:18.049590    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/170ee806-8562-4304-8eb4-9ce73100f449-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-18b9ac-6cf5b5c9f-2whf7\" (UID: \"170ee806-8562-4304-8eb4-9ce73100f449\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-18b9ac-6cf5b5c9f-2whf7"
Sep 10 21:06:18.050468 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:18.050447    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e10c2e9e-56e7-4639-bda8-b5c982287310-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-18b9ac-5f79c684b-6hxxz\" (UID: \"e10c2e9e-56e7-4639-bda8-b5c982287310\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18b9ac-5f79c684b-6hxxz"
Sep 10 21:06:18.050514 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:18.050447    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/170ee806-8562-4304-8eb4-9ce73100f449-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-18b9ac-6cf5b5c9f-2whf7\" (UID: \"170ee806-8562-4304-8eb4-9ce73100f449\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-18b9ac-6cf5b5c9f-2whf7"
Sep 10 21:06:18.057605 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:18.057577    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sklq9\" (UniqueName: \"kubernetes.io/projected/170ee806-8562-4304-8eb4-9ce73100f449-kube-api-access-sklq9\") pod \"payload-processing-e2e-ipp-b-18b9ac-6cf5b5c9f-2whf7\" (UID: \"170ee806-8562-4304-8eb4-9ce73100f449\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-18b9ac-6cf5b5c9f-2whf7"
Sep 10 21:06:18.057802 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:18.057786    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9gwr\" (UniqueName: \"kubernetes.io/projected/e10c2e9e-56e7-4639-bda8-b5c982287310-kube-api-access-w9gwr\") pod \"payload-pre-processing-e2e-ipp-b-18b9ac-5f79c684b-6hxxz\" (UID: \"e10c2e9e-56e7-4639-bda8-b5c982287310\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18b9ac-5f79c684b-6hxxz"
Sep 10 21:06:18.103317 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:18.103287    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-18b9ac-6cf5b5c9f-2whf7"
Sep 10 21:06:18.140225 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:18.140200    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18b9ac-5f79c684b-6hxxz"
Sep 10 21:06:18.376995 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:18.376967    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-18b9ac-6cf5b5c9f-2whf7"]
Sep 10 21:06:18.378522 ip-10-0-141-185 kubenswrapper[2564]: W0910 21:06:18.378499    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod170ee806_8562_4304_8eb4_9ce73100f449.slice/crio-e75e7c71d2a40dc85b7741e684782f72764ccfb2d7988086c6360518076614de WatchSource:0}: Error finding container e75e7c71d2a40dc85b7741e684782f72764ccfb2d7988086c6360518076614de: Status 404 returned error can't find the container with id e75e7c71d2a40dc85b7741e684782f72764ccfb2d7988086c6360518076614de
Sep 10 21:06:18.418089 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:18.418066    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-18b9ac-5f79c684b-6hxxz"]
Sep 10 21:06:18.421089 ip-10-0-141-185 kubenswrapper[2564]: W0910 21:06:18.421066    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode10c2e9e_56e7_4639_bda8_b5c982287310.slice/crio-0588db51a9e4a13c2800fcb59e12642967874ab214478e68d92a8dcbd2d544bf WatchSource:0}: Error finding container 0588db51a9e4a13c2800fcb59e12642967874ab214478e68d92a8dcbd2d544bf: Status 404 returned error can't find the container with id 0588db51a9e4a13c2800fcb59e12642967874ab214478e68d92a8dcbd2d544bf
Sep 10 21:06:18.860912 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:18.860879    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-18b9ac-6cf5b5c9f-2whf7" event={"ID":"170ee806-8562-4304-8eb4-9ce73100f449","Type":"ContainerStarted","Data":"6f0f3b40ecdbff57622f697c531dc6ab6a1df6ef0c9bd572cd6b79dfc9952b1c"}
Sep 10 21:06:18.861386 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:18.860940    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-18b9ac-6cf5b5c9f-2whf7" event={"ID":"170ee806-8562-4304-8eb4-9ce73100f449","Type":"ContainerStarted","Data":"e75e7c71d2a40dc85b7741e684782f72764ccfb2d7988086c6360518076614de"}
Sep 10 21:06:18.861386 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:18.860967    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-18b9ac-6cf5b5c9f-2whf7"
Sep 10 21:06:18.862508 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:18.862487    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18b9ac-5f79c684b-6hxxz" event={"ID":"e10c2e9e-56e7-4639-bda8-b5c982287310","Type":"ContainerStarted","Data":"7872bdd2b02054afcba6242d5fe16643619ffba92dfb9f1a905685cdecf30961"}
Sep 10 21:06:18.862611 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:18.862526    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18b9ac-5f79c684b-6hxxz" event={"ID":"e10c2e9e-56e7-4639-bda8-b5c982287310","Type":"ContainerStarted","Data":"0588db51a9e4a13c2800fcb59e12642967874ab214478e68d92a8dcbd2d544bf"}
Sep 10 21:06:18.862611 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:18.862592    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18b9ac-5f79c684b-6hxxz"
Sep 10 21:06:18.876780 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:18.876742    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-18b9ac-6cf5b5c9f-2whf7" podStartSLOduration=1.876730479 podStartE2EDuration="1.876730479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:06:18.875921646 +0000 UTC m=+1818.840989844" watchObservedRunningTime="2026-09-10 21:06:18.876730479 +0000 UTC m=+1818.841798674"
Sep 10 21:06:18.890779 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:18.890736    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18b9ac-5f79c684b-6hxxz" podStartSLOduration=1.890723667 podStartE2EDuration="1.890723667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:06:18.889592694 +0000 UTC m=+1818.854660889" watchObservedRunningTime="2026-09-10 21:06:18.890723667 +0000 UTC m=+1818.855791862"
Sep 10 21:06:29.872113 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:29.872080    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18b9ac-5f79c684b-6hxxz"
Sep 10 21:06:29.875151 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:29.875127    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-18b9ac-6cf5b5c9f-2whf7"
Sep 10 21:06:31.884296 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:31.884262    2564 patch_prober.go:28] interesting pod/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 21:06:31.884667 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:31.884322    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z" podUID="e013fecc-7d3c-48f6-a7ef-c5695f284b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 21:06:33.952073 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:33.952032    2564 generic.go:356] "Generic (PLEG): container finished" podID="e013fecc-7d3c-48f6-a7ef-c5695f284b39" containerID="425d4f89c49ccead979d48446fc28034ae9e2c489360e28f4b17e97ab18ec058" exitCode=0
Sep 10 21:06:33.952525 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:33.952112    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z" event={"ID":"e013fecc-7d3c-48f6-a7ef-c5695f284b39","Type":"ContainerDied","Data":"425d4f89c49ccead979d48446fc28034ae9e2c489360e28f4b17e97ab18ec058"}
Sep 10 21:06:33.952525 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:33.952521    2564 scope.go:117] "RemoveContainer" containerID="425d4f89c49ccead979d48446fc28034ae9e2c489360e28f4b17e97ab18ec058"
Sep 10 21:06:33.953710 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:33.953677    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:06:33.953797 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:33.953755    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:06:33.953797 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:33.953788    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:06:34.963090 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:34.963043    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z" event={"ID":"e013fecc-7d3c-48f6-a7ef-c5695f284b39","Type":"ContainerStarted","Data":"8fbf9f362bb0869e1a75890cf0777f5e6992e2463edd97b019635874cc848e2e"}
Sep 10 21:06:35.858497 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:35.858461    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:35.858705 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:35.858574    2564 patch_prober.go:28] interesting pod/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 10 21:06:35.858705 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:35.858614    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z" podUID="e013fecc-7d3c-48f6-a7ef-c5695f284b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 10 21:06:36.858968 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:36.858519    2564 patch_prober.go:28] interesting pod/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 10 21:06:36.858968 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:36.858583    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z" podUID="e013fecc-7d3c-48f6-a7ef-c5695f284b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 10 21:06:37.858288 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:37.858250    2564 patch_prober.go:28] interesting pod/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 10 21:06:37.858551 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:37.858320    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z" podUID="e013fecc-7d3c-48f6-a7ef-c5695f284b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 10 21:06:38.857683 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:38.857649    2564 patch_prober.go:28] interesting pod/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 10 21:06:38.858245 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:38.858213    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z" podUID="e013fecc-7d3c-48f6-a7ef-c5695f284b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 10 21:06:40.007492 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:40.007457    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:40.007956 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:40.007870    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:40.008562 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:40.008545    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:06:45.945884 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:45.945850    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-18b9ac-6cf5b5c9f-2whf7_170ee806-8562-4304-8eb4-9ce73100f449/payload-processing/0.log"
Sep 10 21:06:52.882707 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:52.882664    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv"]
Sep 10 21:06:52.887375 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:52.887352    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv"
Sep 10 21:06:52.889256 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:52.889236    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b73da3\"/\"default-dockercfg-2z9pb\""
Sep 10 21:06:52.890049 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:52.890021    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b73da3\"/\"openshift-service-ca.crt\""
Sep 10 21:06:52.890049 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:52.890060    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b73da3\"/\"ipp-route-b73da3-kserve-self-signed-certs\""
Sep 10 21:06:52.890317 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:52.890153    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b73da3\"/\"kube-root-ca.crt\""
Sep 10 21:06:52.894901 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:52.894875    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv"]
Sep 10 21:06:52.902239 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:52.902212    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75c345d0-ebca-4d74-a567-a3076060d816-tmp-dir\") pod \"ipp-route-b73da3-kserve-597d989c9-8hhkv\" (UID: \"75c345d0-ebca-4d74-a567-a3076060d816\") " pod="ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv"
Sep 10 21:06:52.902348 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:52.902248    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75c345d0-ebca-4d74-a567-a3076060d816-tls-certs\") pod \"ipp-route-b73da3-kserve-597d989c9-8hhkv\" (UID: \"75c345d0-ebca-4d74-a567-a3076060d816\") " pod="ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv"
Sep 10 21:06:52.902348 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:52.902269    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75c345d0-ebca-4d74-a567-a3076060d816-home\") pod \"ipp-route-b73da3-kserve-597d989c9-8hhkv\" (UID: \"75c345d0-ebca-4d74-a567-a3076060d816\") " pod="ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv"
Sep 10 21:06:52.902348 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:52.902298    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75c345d0-ebca-4d74-a567-a3076060d816-dshm\") pod \"ipp-route-b73da3-kserve-597d989c9-8hhkv\" (UID: \"75c345d0-ebca-4d74-a567-a3076060d816\") " pod="ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv"
Sep 10 21:06:52.902468 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:52.902405    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddl5p\" (UniqueName: \"kubernetes.io/projected/75c345d0-ebca-4d74-a567-a3076060d816-kube-api-access-ddl5p\") pod \"ipp-route-b73da3-kserve-597d989c9-8hhkv\" (UID: \"75c345d0-ebca-4d74-a567-a3076060d816\") " pod="ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv"
Sep 10 21:06:52.902468 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:52.902457    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75c345d0-ebca-4d74-a567-a3076060d816-model-cache\") pod \"ipp-route-b73da3-kserve-597d989c9-8hhkv\" (UID: \"75c345d0-ebca-4d74-a567-a3076060d816\") " pod="ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv"
Sep 10 21:06:53.002957 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:53.002930    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75c345d0-ebca-4d74-a567-a3076060d816-tmp-dir\") pod \"ipp-route-b73da3-kserve-597d989c9-8hhkv\" (UID: \"75c345d0-ebca-4d74-a567-a3076060d816\") " pod="ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv"
Sep 10 21:06:53.003101 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:53.002963    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75c345d0-ebca-4d74-a567-a3076060d816-tls-certs\") pod \"ipp-route-b73da3-kserve-597d989c9-8hhkv\" (UID: \"75c345d0-ebca-4d74-a567-a3076060d816\") " pod="ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv"
Sep 10 21:06:53.003101 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:53.002984    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75c345d0-ebca-4d74-a567-a3076060d816-home\") pod \"ipp-route-b73da3-kserve-597d989c9-8hhkv\" (UID: \"75c345d0-ebca-4d74-a567-a3076060d816\") " pod="ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv"
Sep 10 21:06:53.003101 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:53.003004    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75c345d0-ebca-4d74-a567-a3076060d816-dshm\") pod \"ipp-route-b73da3-kserve-597d989c9-8hhkv\" (UID: \"75c345d0-ebca-4d74-a567-a3076060d816\") " pod="ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv"
Sep 10 21:06:53.003101 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:53.003050    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddl5p\" (UniqueName: \"kubernetes.io/projected/75c345d0-ebca-4d74-a567-a3076060d816-kube-api-access-ddl5p\") pod \"ipp-route-b73da3-kserve-597d989c9-8hhkv\" (UID: \"75c345d0-ebca-4d74-a567-a3076060d816\") " pod="ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv"
Sep 10 21:06:53.003101 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:53.003093    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75c345d0-ebca-4d74-a567-a3076060d816-model-cache\") pod \"ipp-route-b73da3-kserve-597d989c9-8hhkv\" (UID: \"75c345d0-ebca-4d74-a567-a3076060d816\") " pod="ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv"
Sep 10 21:06:53.003440 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:53.003417    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75c345d0-ebca-4d74-a567-a3076060d816-tmp-dir\") pod \"ipp-route-b73da3-kserve-597d989c9-8hhkv\" (UID: \"75c345d0-ebca-4d74-a567-a3076060d816\") " pod="ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv"
Sep 10 21:06:53.003502 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:53.003427    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75c345d0-ebca-4d74-a567-a3076060d816-home\") pod \"ipp-route-b73da3-kserve-597d989c9-8hhkv\" (UID: \"75c345d0-ebca-4d74-a567-a3076060d816\") " pod="ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv"
Sep 10 21:06:53.003502 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:53.003489    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75c345d0-ebca-4d74-a567-a3076060d816-model-cache\") pod \"ipp-route-b73da3-kserve-597d989c9-8hhkv\" (UID: \"75c345d0-ebca-4d74-a567-a3076060d816\") " pod="ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv"
Sep 10 21:06:53.005201 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:53.005157    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75c345d0-ebca-4d74-a567-a3076060d816-dshm\") pod \"ipp-route-b73da3-kserve-597d989c9-8hhkv\" (UID: \"75c345d0-ebca-4d74-a567-a3076060d816\") " pod="ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv"
Sep 10 21:06:53.005486 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:53.005469    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75c345d0-ebca-4d74-a567-a3076060d816-tls-certs\") pod \"ipp-route-b73da3-kserve-597d989c9-8hhkv\" (UID: \"75c345d0-ebca-4d74-a567-a3076060d816\") " pod="ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv"
Sep 10 21:06:53.014769 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:53.014748    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddl5p\" (UniqueName: \"kubernetes.io/projected/75c345d0-ebca-4d74-a567-a3076060d816-kube-api-access-ddl5p\") pod \"ipp-route-b73da3-kserve-597d989c9-8hhkv\" (UID: \"75c345d0-ebca-4d74-a567-a3076060d816\") " pod="ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv"
Sep 10 21:06:53.199597 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:53.199522    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv"
Sep 10 21:06:53.472353 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:53.472328    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv"]
Sep 10 21:06:53.474059 ip-10-0-141-185 kubenswrapper[2564]: W0910 21:06:53.474035    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75c345d0_ebca_4d74_a567_a3076060d816.slice/crio-e0265e09ddaadb13c40b9c0d6a9e46f0218ffa09183649e394dd5d3ad1f23877 WatchSource:0}: Error finding container e0265e09ddaadb13c40b9c0d6a9e46f0218ffa09183649e394dd5d3ad1f23877: Status 404 returned error can't find the container with id e0265e09ddaadb13c40b9c0d6a9e46f0218ffa09183649e394dd5d3ad1f23877
Sep 10 21:06:54.078846 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:54.078808    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv" event={"ID":"75c345d0-ebca-4d74-a567-a3076060d816","Type":"ContainerStarted","Data":"0e2f0f651d98b634b1a8a77f8331ea499ceae4917145073ab0520a6c02cb41e6"}
Sep 10 21:06:54.079212 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:54.078863    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv" event={"ID":"75c345d0-ebca-4d74-a567-a3076060d816","Type":"ContainerStarted","Data":"e0265e09ddaadb13c40b9c0d6a9e46f0218ffa09183649e394dd5d3ad1f23877"}
Sep 10 21:06:54.097321 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:06:54.097276    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv" podStartSLOduration=2.097262966 podStartE2EDuration="2.097262966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:06:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:06:54.09448946 +0000 UTC m=+1854.059557656" watchObservedRunningTime="2026-09-10 21:06:54.097262966 +0000 UTC m=+1854.062331160"
Sep 10 21:07:00.809469 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:00.809434    2564 scope.go:117] "RemoveContainer" containerID="1bed497035860851e0c7bebf855dd44c77ad2ff5018728ef9588e37fad20d52a"
Sep 10 21:07:03.200068 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:03.200033    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv"
Sep 10 21:07:03.200557 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:03.200089    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv"
Sep 10 21:07:03.212835 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:03.212809    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv"
Sep 10 21:07:04.148331 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:04.148290    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv"
Sep 10 21:07:26.870439 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:26.870363    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-18b9ac-6cf5b5c9f-2whf7_170ee806-8562-4304-8eb4-9ce73100f449/payload-processing/0.log"
Sep 10 21:07:32.092751 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:32.092716    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-eb2a90-d9cf8bcd5-l5zw5"]
Sep 10 21:07:32.096335 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:32.096311    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb2a90-d9cf8bcd5-l5zw5"
Sep 10 21:07:32.098433 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:32.098405    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-eb2a90-dockercfg-z49bb\""
Sep 10 21:07:32.098533 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:32.098438    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-eb2a90\""
Sep 10 21:07:32.106059 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:32.106038    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-eb2a90-d9cf8bcd5-l5zw5"]
Sep 10 21:07:32.201515 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:32.201472    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f98d88b0-9e67-4c6e-b50c-fb9e77bed537-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-eb2a90-d9cf8bcd5-l5zw5\" (UID: \"f98d88b0-9e67-4c6e-b50c-fb9e77bed537\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb2a90-d9cf8bcd5-l5zw5"
Sep 10 21:07:32.201685 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:32.201535    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zksgv\" (UniqueName: \"kubernetes.io/projected/f98d88b0-9e67-4c6e-b50c-fb9e77bed537-kube-api-access-zksgv\") pod \"payload-processing-e2e-ipp-cleanup-eb2a90-d9cf8bcd5-l5zw5\" (UID: \"f98d88b0-9e67-4c6e-b50c-fb9e77bed537\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb2a90-d9cf8bcd5-l5zw5"
Sep 10 21:07:32.302883 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:32.302850    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f98d88b0-9e67-4c6e-b50c-fb9e77bed537-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-eb2a90-d9cf8bcd5-l5zw5\" (UID: \"f98d88b0-9e67-4c6e-b50c-fb9e77bed537\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb2a90-d9cf8bcd5-l5zw5"
Sep 10 21:07:32.303036 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:32.302895    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zksgv\" (UniqueName: \"kubernetes.io/projected/f98d88b0-9e67-4c6e-b50c-fb9e77bed537-kube-api-access-zksgv\") pod \"payload-processing-e2e-ipp-cleanup-eb2a90-d9cf8bcd5-l5zw5\" (UID: \"f98d88b0-9e67-4c6e-b50c-fb9e77bed537\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb2a90-d9cf8bcd5-l5zw5"
Sep 10 21:07:32.303569 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:32.303548    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f98d88b0-9e67-4c6e-b50c-fb9e77bed537-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-eb2a90-d9cf8bcd5-l5zw5\" (UID: \"f98d88b0-9e67-4c6e-b50c-fb9e77bed537\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb2a90-d9cf8bcd5-l5zw5"
Sep 10 21:07:32.311053 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:32.311026    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zksgv\" (UniqueName: \"kubernetes.io/projected/f98d88b0-9e67-4c6e-b50c-fb9e77bed537-kube-api-access-zksgv\") pod \"payload-processing-e2e-ipp-cleanup-eb2a90-d9cf8bcd5-l5zw5\" (UID: \"f98d88b0-9e67-4c6e-b50c-fb9e77bed537\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb2a90-d9cf8bcd5-l5zw5"
Sep 10 21:07:32.407316 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:32.407254    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb2a90-d9cf8bcd5-l5zw5"
Sep 10 21:07:32.683599 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:32.683572    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-eb2a90-d9cf8bcd5-l5zw5"]
Sep 10 21:07:32.685347 ip-10-0-141-185 kubenswrapper[2564]: W0910 21:07:32.685321    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf98d88b0_9e67_4c6e_b50c_fb9e77bed537.slice/crio-221f0461f7f1733ecc0841d39485e37395bf2c907649959612a7db372cde49a0 WatchSource:0}: Error finding container 221f0461f7f1733ecc0841d39485e37395bf2c907649959612a7db372cde49a0: Status 404 returned error can't find the container with id 221f0461f7f1733ecc0841d39485e37395bf2c907649959612a7db372cde49a0
Sep 10 21:07:33.307132 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:33.307096    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb2a90-d9cf8bcd5-l5zw5" event={"ID":"f98d88b0-9e67-4c6e-b50c-fb9e77bed537","Type":"ContainerStarted","Data":"d34213dd926b71f4ab2babee9068440dad7fdd1704205cd8115eb012d6d322f2"}
Sep 10 21:07:33.307506 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:33.307159    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb2a90-d9cf8bcd5-l5zw5"
Sep 10 21:07:33.307506 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:33.307206    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb2a90-d9cf8bcd5-l5zw5" event={"ID":"f98d88b0-9e67-4c6e-b50c-fb9e77bed537","Type":"ContainerStarted","Data":"221f0461f7f1733ecc0841d39485e37395bf2c907649959612a7db372cde49a0"}
Sep 10 21:07:33.326972 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:33.326919    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb2a90-d9cf8bcd5-l5zw5" podStartSLOduration=1.326905477 podStartE2EDuration="1.326905477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:07:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:07:33.324645487 +0000 UTC m=+1893.289713681" watchObservedRunningTime="2026-09-10 21:07:33.326905477 +0000 UTC m=+1893.291973672"
Sep 10 21:07:44.315989 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:44.315958    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb2a90-d9cf8bcd5-l5zw5"
Sep 10 21:07:55.742430 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:55.742395    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eb2a90-gdxlx"]
Sep 10 21:07:55.746143 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:55.746123    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eb2a90-gdxlx"
Sep 10 21:07:55.748460 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:55.748433    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:07:55.748591 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:55.748461    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plv2h\""
Sep 10 21:07:55.752526 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:55.752497    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eb2a90-gdxlx"]
Sep 10 21:07:55.884869 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:55.884843    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdde0124-b7ae-4064-8538-147d0f7d78bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-eb2a90-gdxlx\" (UID: \"bdde0124-b7ae-4064-8538-147d0f7d78bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eb2a90-gdxlx"
Sep 10 21:07:55.986212 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:55.986160    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdde0124-b7ae-4064-8538-147d0f7d78bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-eb2a90-gdxlx\" (UID: \"bdde0124-b7ae-4064-8538-147d0f7d78bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eb2a90-gdxlx"
Sep 10 21:07:55.986759 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:55.986738    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdde0124-b7ae-4064-8538-147d0f7d78bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-eb2a90-gdxlx\" (UID: \"bdde0124-b7ae-4064-8538-147d0f7d78bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eb2a90-gdxlx"
Sep 10 21:07:56.058621 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:56.058569    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eb2a90-gdxlx"
Sep 10 21:07:56.338699 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:56.338671    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eb2a90-gdxlx"]
Sep 10 21:07:56.339959 ip-10-0-141-185 kubenswrapper[2564]: W0910 21:07:56.339935    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdde0124_b7ae_4064_8538_147d0f7d78bb.slice/crio-267f6f81d9295ef6366fb81ac730135f0ef38b17cb99e9bc7d0e09c8899b5b1c WatchSource:0}: Error finding container 267f6f81d9295ef6366fb81ac730135f0ef38b17cb99e9bc7d0e09c8899b5b1c: Status 404 returned error can't find the container with id 267f6f81d9295ef6366fb81ac730135f0ef38b17cb99e9bc7d0e09c8899b5b1c
Sep 10 21:07:56.449934 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:56.449905    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eb2a90-gdxlx" event={"ID":"bdde0124-b7ae-4064-8538-147d0f7d78bb","Type":"ContainerStarted","Data":"267f6f81d9295ef6366fb81ac730135f0ef38b17cb99e9bc7d0e09c8899b5b1c"}
Sep 10 21:07:57.458664 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:57.458631    2564 generic.go:356] "Generic (PLEG): container finished" podID="bdde0124-b7ae-4064-8538-147d0f7d78bb" containerID="81df71c2b619999ea58acc025695991cdd63776239b6042ea871618edcabb499" exitCode=0
Sep 10 21:07:57.459035 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:57.458680    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eb2a90-gdxlx" event={"ID":"bdde0124-b7ae-4064-8538-147d0f7d78bb","Type":"ContainerDied","Data":"81df71c2b619999ea58acc025695991cdd63776239b6042ea871618edcabb499"}
Sep 10 21:07:58.601766 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:58.601739    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eb2a90-gdxlx"
Sep 10 21:07:58.709721 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:58.709695    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdde0124-b7ae-4064-8538-147d0f7d78bb-maas-api-service-ca\") pod \"bdde0124-b7ae-4064-8538-147d0f7d78bb\" (UID: \"bdde0124-b7ae-4064-8538-147d0f7d78bb\") "
Sep 10 21:07:58.710008 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:58.709987    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdde0124-b7ae-4064-8538-147d0f7d78bb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bdde0124-b7ae-4064-8538-147d0f7d78bb" (UID: "bdde0124-b7ae-4064-8538-147d0f7d78bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:07:58.811262 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:58.811204    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdde0124-b7ae-4064-8538-147d0f7d78bb-maas-api-service-ca\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:07:59.472228 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:59.472194    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eb2a90-gdxlx"
Sep 10 21:07:59.472430 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:59.472221    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eb2a90-gdxlx" event={"ID":"bdde0124-b7ae-4064-8538-147d0f7d78bb","Type":"ContainerDied","Data":"267f6f81d9295ef6366fb81ac730135f0ef38b17cb99e9bc7d0e09c8899b5b1c"}
Sep 10 21:07:59.472430 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:07:59.472277    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="267f6f81d9295ef6366fb81ac730135f0ef38b17cb99e9bc7d0e09c8899b5b1c"
Sep 10 21:08:02.069553 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:02.069511    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-eb2a90-d9cf8bcd5-l5zw5"]
Sep 10 21:08:02.070053 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:02.069765    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb2a90-d9cf8bcd5-l5zw5" podUID="f98d88b0-9e67-4c6e-b50c-fb9e77bed537" containerName="payload-processing" containerID="cri-o://d34213dd926b71f4ab2babee9068440dad7fdd1704205cd8115eb012d6d322f2" gracePeriod=30
Sep 10 21:08:02.252363 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:02.252292    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eb2a90-gdxlx"]
Sep 10 21:08:02.255711 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:02.255678    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eb2a90-gdxlx"]
Sep 10 21:08:02.310850 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:02.310824    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb2a90-d9cf8bcd5-l5zw5"
Sep 10 21:08:02.437893 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:02.437812    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zksgv\" (UniqueName: \"kubernetes.io/projected/f98d88b0-9e67-4c6e-b50c-fb9e77bed537-kube-api-access-zksgv\") pod \"f98d88b0-9e67-4c6e-b50c-fb9e77bed537\" (UID: \"f98d88b0-9e67-4c6e-b50c-fb9e77bed537\") "
Sep 10 21:08:02.437893 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:02.437867    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f98d88b0-9e67-4c6e-b50c-fb9e77bed537-plugins-config-volume\") pod \"f98d88b0-9e67-4c6e-b50c-fb9e77bed537\" (UID: \"f98d88b0-9e67-4c6e-b50c-fb9e77bed537\") "
Sep 10 21:08:02.438272 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:02.438241    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f98d88b0-9e67-4c6e-b50c-fb9e77bed537-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f98d88b0-9e67-4c6e-b50c-fb9e77bed537" (UID: "f98d88b0-9e67-4c6e-b50c-fb9e77bed537"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:08:02.439905 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:02.439889    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f98d88b0-9e67-4c6e-b50c-fb9e77bed537-kube-api-access-zksgv" (OuterVolumeSpecName: "kube-api-access-zksgv") pod "f98d88b0-9e67-4c6e-b50c-fb9e77bed537" (UID: "f98d88b0-9e67-4c6e-b50c-fb9e77bed537"). InnerVolumeSpecName "kube-api-access-zksgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:08:02.490973 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:02.490948    2564 generic.go:356] "Generic (PLEG): container finished" podID="f98d88b0-9e67-4c6e-b50c-fb9e77bed537" containerID="d34213dd926b71f4ab2babee9068440dad7fdd1704205cd8115eb012d6d322f2" exitCode=0
Sep 10 21:08:02.491089 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:02.490991    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb2a90-d9cf8bcd5-l5zw5"
Sep 10 21:08:02.491089 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:02.491034    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb2a90-d9cf8bcd5-l5zw5" event={"ID":"f98d88b0-9e67-4c6e-b50c-fb9e77bed537","Type":"ContainerDied","Data":"d34213dd926b71f4ab2babee9068440dad7fdd1704205cd8115eb012d6d322f2"}
Sep 10 21:08:02.491089 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:02.491081    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eb2a90-d9cf8bcd5-l5zw5" event={"ID":"f98d88b0-9e67-4c6e-b50c-fb9e77bed537","Type":"ContainerDied","Data":"221f0461f7f1733ecc0841d39485e37395bf2c907649959612a7db372cde49a0"}
Sep 10 21:08:02.491285 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:02.491104    2564 scope.go:117] "RemoveContainer" containerID="d34213dd926b71f4ab2babee9068440dad7fdd1704205cd8115eb012d6d322f2"
Sep 10 21:08:02.500698 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:02.500682    2564 scope.go:117] "RemoveContainer" containerID="d34213dd926b71f4ab2babee9068440dad7fdd1704205cd8115eb012d6d322f2"
Sep 10 21:08:02.500946 ip-10-0-141-185 kubenswrapper[2564]: E0910 21:08:02.500928    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d34213dd926b71f4ab2babee9068440dad7fdd1704205cd8115eb012d6d322f2\": container with ID starting with d34213dd926b71f4ab2babee9068440dad7fdd1704205cd8115eb012d6d322f2 not found: ID does not exist" containerID="d34213dd926b71f4ab2babee9068440dad7fdd1704205cd8115eb012d6d322f2"
Sep 10 21:08:02.501003 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:02.500954    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d34213dd926b71f4ab2babee9068440dad7fdd1704205cd8115eb012d6d322f2"} err="failed to get container status \"d34213dd926b71f4ab2babee9068440dad7fdd1704205cd8115eb012d6d322f2\": rpc error: code = NotFound desc = could not find container \"d34213dd926b71f4ab2babee9068440dad7fdd1704205cd8115eb012d6d322f2\": container with ID starting with d34213dd926b71f4ab2babee9068440dad7fdd1704205cd8115eb012d6d322f2 not found: ID does not exist"
Sep 10 21:08:02.512244 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:02.512221    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-eb2a90-d9cf8bcd5-l5zw5"]
Sep 10 21:08:02.516556 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:02.516536    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-eb2a90-d9cf8bcd5-l5zw5"]
Sep 10 21:08:02.538383 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:02.538366    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zksgv\" (UniqueName: \"kubernetes.io/projected/f98d88b0-9e67-4c6e-b50c-fb9e77bed537-kube-api-access-zksgv\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:08:02.538383 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:02.538384    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f98d88b0-9e67-4c6e-b50c-fb9e77bed537-plugins-config-volume\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:08:02.540716 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:02.540696    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdde0124-b7ae-4064-8538-147d0f7d78bb" path="/var/lib/kubelet/pods/bdde0124-b7ae-4064-8538-147d0f7d78bb/volumes"
Sep 10 21:08:02.541050 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:02.541037    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f98d88b0-9e67-4c6e-b50c-fb9e77bed537" path="/var/lib/kubelet/pods/f98d88b0-9e67-4c6e-b50c-fb9e77bed537/volumes"
Sep 10 21:08:12.670825 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:12.670791    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73da3-hfnll"]
Sep 10 21:08:12.671271 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:12.671123    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdde0124-b7ae-4064-8538-147d0f7d78bb" containerName="revoke-keys"
Sep 10 21:08:12.671271 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:12.671134    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdde0124-b7ae-4064-8538-147d0f7d78bb" containerName="revoke-keys"
Sep 10 21:08:12.671271 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:12.671158    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f98d88b0-9e67-4c6e-b50c-fb9e77bed537" containerName="payload-processing"
Sep 10 21:08:12.671271 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:12.671180    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f98d88b0-9e67-4c6e-b50c-fb9e77bed537" containerName="payload-processing"
Sep 10 21:08:12.671271 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:12.671246    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f98d88b0-9e67-4c6e-b50c-fb9e77bed537" containerName="payload-processing"
Sep 10 21:08:12.671271 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:12.671263    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdde0124-b7ae-4064-8538-147d0f7d78bb" containerName="revoke-keys"
Sep 10 21:08:12.674176 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:12.674150    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73da3-hfnll"
Sep 10 21:08:12.676534 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:12.676505    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:08:12.676681 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:12.676553    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plv2h\""
Sep 10 21:08:12.684295 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:12.684267    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73da3-hfnll"]
Sep 10 21:08:12.820699 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:12.820668    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7642d8-16da-4d54-a0aa-e13d870acf9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b73da3-hfnll\" (UID: \"ef7642d8-16da-4d54-a0aa-e13d870acf9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73da3-hfnll"
Sep 10 21:08:12.922142 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:12.922064    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7642d8-16da-4d54-a0aa-e13d870acf9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b73da3-hfnll\" (UID: \"ef7642d8-16da-4d54-a0aa-e13d870acf9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73da3-hfnll"
Sep 10 21:08:12.922699 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:12.922677    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7642d8-16da-4d54-a0aa-e13d870acf9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b73da3-hfnll\" (UID: \"ef7642d8-16da-4d54-a0aa-e13d870acf9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73da3-hfnll"
Sep 10 21:08:12.984598 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:12.984562    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73da3-hfnll"
Sep 10 21:08:13.256599 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:13.256574    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73da3-hfnll"]
Sep 10 21:08:13.258453 ip-10-0-141-185 kubenswrapper[2564]: W0910 21:08:13.258427    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef7642d8_16da_4d54_a0aa_e13d870acf9a.slice/crio-7d13f551b0c240d395e1e4e456136c930ca97b2cfc899dc9508e62a972aecb1c WatchSource:0}: Error finding container 7d13f551b0c240d395e1e4e456136c930ca97b2cfc899dc9508e62a972aecb1c: Status 404 returned error can't find the container with id 7d13f551b0c240d395e1e4e456136c930ca97b2cfc899dc9508e62a972aecb1c
Sep 10 21:08:13.556903 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:13.556832    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73da3-hfnll" event={"ID":"ef7642d8-16da-4d54-a0aa-e13d870acf9a","Type":"ContainerStarted","Data":"dcfd10dda0cb92478beef3f530a3b36d4e5204950e4bbefde4d1fde6c13d0c9c"}
Sep 10 21:08:13.556903 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:13.556886    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73da3-hfnll" event={"ID":"ef7642d8-16da-4d54-a0aa-e13d870acf9a","Type":"ContainerStarted","Data":"7d13f551b0c240d395e1e4e456136c930ca97b2cfc899dc9508e62a972aecb1c"}
Sep 10 21:08:13.573841 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:13.573800    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73da3-hfnll" podStartSLOduration=1.573783634 podStartE2EDuration="1.573783634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:08:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:08:13.572070802 +0000 UTC m=+1933.537138998" watchObservedRunningTime="2026-09-10 21:08:13.573783634 +0000 UTC m=+1933.538851829"
Sep 10 21:08:14.564259 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:14.564224    2564 generic.go:356] "Generic (PLEG): container finished" podID="ef7642d8-16da-4d54-a0aa-e13d870acf9a" containerID="dcfd10dda0cb92478beef3f530a3b36d4e5204950e4bbefde4d1fde6c13d0c9c" exitCode=0
Sep 10 21:08:14.564656 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:14.564325    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73da3-hfnll" event={"ID":"ef7642d8-16da-4d54-a0aa-e13d870acf9a","Type":"ContainerDied","Data":"dcfd10dda0cb92478beef3f530a3b36d4e5204950e4bbefde4d1fde6c13d0c9c"}
Sep 10 21:08:15.698927 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:15.698901    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73da3-hfnll"
Sep 10 21:08:15.743859 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:15.743830    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7642d8-16da-4d54-a0aa-e13d870acf9a-maas-api-service-ca\") pod \"ef7642d8-16da-4d54-a0aa-e13d870acf9a\" (UID: \"ef7642d8-16da-4d54-a0aa-e13d870acf9a\") "
Sep 10 21:08:15.744150 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:15.744126    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef7642d8-16da-4d54-a0aa-e13d870acf9a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ef7642d8-16da-4d54-a0aa-e13d870acf9a" (UID: "ef7642d8-16da-4d54-a0aa-e13d870acf9a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:08:15.844930 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:15.844877    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7642d8-16da-4d54-a0aa-e13d870acf9a-maas-api-service-ca\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:08:16.577035 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:16.577011    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73da3-hfnll"
Sep 10 21:08:16.577200 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:16.577032    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73da3-hfnll" event={"ID":"ef7642d8-16da-4d54-a0aa-e13d870acf9a","Type":"ContainerDied","Data":"7d13f551b0c240d395e1e4e456136c930ca97b2cfc899dc9508e62a972aecb1c"}
Sep 10 21:08:16.577200 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:16.577075    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d13f551b0c240d395e1e4e456136c930ca97b2cfc899dc9508e62a972aecb1c"
Sep 10 21:08:28.450107 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:28.450074    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73da3-hfnll"]
Sep 10 21:08:28.453248 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:28.453223    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73da3-hfnll"]
Sep 10 21:08:28.540659 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:28.540632    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef7642d8-16da-4d54-a0aa-e13d870acf9a" path="/var/lib/kubelet/pods/ef7642d8-16da-4d54-a0aa-e13d870acf9a/volumes"
Sep 10 21:08:33.793922 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:33.793880    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv"]
Sep 10 21:08:48.813871 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:48.813826    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv" podUID="75c345d0-ebca-4d74-a567-a3076060d816" containerName="main" containerID="cri-o://0e2f0f651d98b634b1a8a77f8331ea499ceae4917145073ab0520a6c02cb41e6" gracePeriod=45
Sep 10 21:08:49.051948 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:49.051925    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv"
Sep 10 21:08:49.095417 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:49.095391    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75c345d0-ebca-4d74-a567-a3076060d816-dshm\") pod \"75c345d0-ebca-4d74-a567-a3076060d816\" (UID: \"75c345d0-ebca-4d74-a567-a3076060d816\") "
Sep 10 21:08:49.095572 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:49.095443    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddl5p\" (UniqueName: \"kubernetes.io/projected/75c345d0-ebca-4d74-a567-a3076060d816-kube-api-access-ddl5p\") pod \"75c345d0-ebca-4d74-a567-a3076060d816\" (UID: \"75c345d0-ebca-4d74-a567-a3076060d816\") "
Sep 10 21:08:49.095572 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:49.095498    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75c345d0-ebca-4d74-a567-a3076060d816-tmp-dir\") pod \"75c345d0-ebca-4d74-a567-a3076060d816\" (UID: \"75c345d0-ebca-4d74-a567-a3076060d816\") "
Sep 10 21:08:49.095572 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:49.095541    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75c345d0-ebca-4d74-a567-a3076060d816-home\") pod \"75c345d0-ebca-4d74-a567-a3076060d816\" (UID: \"75c345d0-ebca-4d74-a567-a3076060d816\") "
Sep 10 21:08:49.095572 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:49.095562    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75c345d0-ebca-4d74-a567-a3076060d816-model-cache\") pod \"75c345d0-ebca-4d74-a567-a3076060d816\" (UID: \"75c345d0-ebca-4d74-a567-a3076060d816\") "
Sep 10 21:08:49.095807 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:49.095604    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75c345d0-ebca-4d74-a567-a3076060d816-tls-certs\") pod \"75c345d0-ebca-4d74-a567-a3076060d816\" (UID: \"75c345d0-ebca-4d74-a567-a3076060d816\") "
Sep 10 21:08:49.095867 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:49.095812    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c345d0-ebca-4d74-a567-a3076060d816-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "75c345d0-ebca-4d74-a567-a3076060d816" (UID: "75c345d0-ebca-4d74-a567-a3076060d816"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:08:49.095928 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:49.095909    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75c345d0-ebca-4d74-a567-a3076060d816-tmp-dir\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:08:49.096093 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:49.096050    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c345d0-ebca-4d74-a567-a3076060d816-model-cache" (OuterVolumeSpecName: "model-cache") pod "75c345d0-ebca-4d74-a567-a3076060d816" (UID: "75c345d0-ebca-4d74-a567-a3076060d816"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:08:49.096093 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:49.096069    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c345d0-ebca-4d74-a567-a3076060d816-home" (OuterVolumeSpecName: "home") pod "75c345d0-ebca-4d74-a567-a3076060d816" (UID: "75c345d0-ebca-4d74-a567-a3076060d816"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:08:49.097712 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:49.097681    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75c345d0-ebca-4d74-a567-a3076060d816-kube-api-access-ddl5p" (OuterVolumeSpecName: "kube-api-access-ddl5p") pod "75c345d0-ebca-4d74-a567-a3076060d816" (UID: "75c345d0-ebca-4d74-a567-a3076060d816"). InnerVolumeSpecName "kube-api-access-ddl5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:08:49.097840 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:49.097818    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75c345d0-ebca-4d74-a567-a3076060d816-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "75c345d0-ebca-4d74-a567-a3076060d816" (UID: "75c345d0-ebca-4d74-a567-a3076060d816"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:08:49.097913 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:49.097900    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c345d0-ebca-4d74-a567-a3076060d816-dshm" (OuterVolumeSpecName: "dshm") pod "75c345d0-ebca-4d74-a567-a3076060d816" (UID: "75c345d0-ebca-4d74-a567-a3076060d816"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:08:49.197331 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:49.197297    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75c345d0-ebca-4d74-a567-a3076060d816-home\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:08:49.197331 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:49.197326    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75c345d0-ebca-4d74-a567-a3076060d816-model-cache\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:08:49.197464 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:49.197337    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75c345d0-ebca-4d74-a567-a3076060d816-tls-certs\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:08:49.197464 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:49.197345    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75c345d0-ebca-4d74-a567-a3076060d816-dshm\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:08:49.197464 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:49.197354    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddl5p\" (UniqueName: \"kubernetes.io/projected/75c345d0-ebca-4d74-a567-a3076060d816-kube-api-access-ddl5p\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:08:49.770743 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:49.770711    2564 generic.go:356] "Generic (PLEG): container finished" podID="75c345d0-ebca-4d74-a567-a3076060d816" containerID="0e2f0f651d98b634b1a8a77f8331ea499ceae4917145073ab0520a6c02cb41e6" exitCode=0
Sep 10 21:08:49.770887 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:49.770753    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv"
Sep 10 21:08:49.770887 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:49.770788    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv" event={"ID":"75c345d0-ebca-4d74-a567-a3076060d816","Type":"ContainerDied","Data":"0e2f0f651d98b634b1a8a77f8331ea499ceae4917145073ab0520a6c02cb41e6"}
Sep 10 21:08:49.770887 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:49.770830    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv" event={"ID":"75c345d0-ebca-4d74-a567-a3076060d816","Type":"ContainerDied","Data":"e0265e09ddaadb13c40b9c0d6a9e46f0218ffa09183649e394dd5d3ad1f23877"}
Sep 10 21:08:49.770887 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:49.770847    2564 scope.go:117] "RemoveContainer" containerID="0e2f0f651d98b634b1a8a77f8331ea499ceae4917145073ab0520a6c02cb41e6"
Sep 10 21:08:49.780695 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:49.780677    2564 scope.go:117] "RemoveContainer" containerID="0e2f0f651d98b634b1a8a77f8331ea499ceae4917145073ab0520a6c02cb41e6"
Sep 10 21:08:49.781005 ip-10-0-141-185 kubenswrapper[2564]: E0910 21:08:49.780981    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e2f0f651d98b634b1a8a77f8331ea499ceae4917145073ab0520a6c02cb41e6\": container with ID starting with 0e2f0f651d98b634b1a8a77f8331ea499ceae4917145073ab0520a6c02cb41e6 not found: ID does not exist" containerID="0e2f0f651d98b634b1a8a77f8331ea499ceae4917145073ab0520a6c02cb41e6"
Sep 10 21:08:49.781106 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:49.781009    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e2f0f651d98b634b1a8a77f8331ea499ceae4917145073ab0520a6c02cb41e6"} err="failed to get container status \"0e2f0f651d98b634b1a8a77f8331ea499ceae4917145073ab0520a6c02cb41e6\": rpc error: code = NotFound desc = could not find container \"0e2f0f651d98b634b1a8a77f8331ea499ceae4917145073ab0520a6c02cb41e6\": container with ID starting with 0e2f0f651d98b634b1a8a77f8331ea499ceae4917145073ab0520a6c02cb41e6 not found: ID does not exist"
Sep 10 21:08:49.792537 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:49.792514    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv"]
Sep 10 21:08:49.796331 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:49.796303    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-b73da3/ipp-route-b73da3-kserve-597d989c9-8hhkv"]
Sep 10 21:08:50.547284 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:08:50.547249    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75c345d0-ebca-4d74-a567-a3076060d816" path="/var/lib/kubelet/pods/75c345d0-ebca-4d74-a567-a3076060d816/volumes"
Sep 10 21:09:18.127767 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:18.127728    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-18b9ac-2x2db"]
Sep 10 21:09:18.128186 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:18.128072    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75c345d0-ebca-4d74-a567-a3076060d816" containerName="main"
Sep 10 21:09:18.128186 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:18.128083    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="75c345d0-ebca-4d74-a567-a3076060d816" containerName="main"
Sep 10 21:09:18.128186 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:18.128101    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef7642d8-16da-4d54-a0aa-e13d870acf9a" containerName="revoke-keys"
Sep 10 21:09:18.128186 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:18.128108    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef7642d8-16da-4d54-a0aa-e13d870acf9a" containerName="revoke-keys"
Sep 10 21:09:18.128382 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:18.128189    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef7642d8-16da-4d54-a0aa-e13d870acf9a" containerName="revoke-keys"
Sep 10 21:09:18.128382 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:18.128203    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="75c345d0-ebca-4d74-a567-a3076060d816" containerName="main"
Sep 10 21:09:18.131158 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:18.131140    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-18b9ac-2x2db"
Sep 10 21:09:18.133365 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:18.133338    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:09:18.133519 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:18.133426    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plv2h\""
Sep 10 21:09:18.139919 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:18.139891    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-18b9ac-2x2db"]
Sep 10 21:09:18.223258 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:18.223230    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1520346b-4b62-44fd-b61b-888a3789895a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-18b9ac-2x2db\" (UID: \"1520346b-4b62-44fd-b61b-888a3789895a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-18b9ac-2x2db"
Sep 10 21:09:18.324518 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:18.324495    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1520346b-4b62-44fd-b61b-888a3789895a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-18b9ac-2x2db\" (UID: \"1520346b-4b62-44fd-b61b-888a3789895a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-18b9ac-2x2db"
Sep 10 21:09:18.325019 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:18.325003    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1520346b-4b62-44fd-b61b-888a3789895a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-18b9ac-2x2db\" (UID: \"1520346b-4b62-44fd-b61b-888a3789895a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-18b9ac-2x2db"
Sep 10 21:09:18.442797 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:18.442735    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-18b9ac-2x2db"
Sep 10 21:09:18.713795 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:18.713773    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-18b9ac-2x2db"]
Sep 10 21:09:18.716533 ip-10-0-141-185 kubenswrapper[2564]: W0910 21:09:18.716506    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1520346b_4b62_44fd_b61b_888a3789895a.slice/crio-c4207af394aa56d0bfec13a3dc31663819452dda933129415ecf48d24c2d89b8 WatchSource:0}: Error finding container c4207af394aa56d0bfec13a3dc31663819452dda933129415ecf48d24c2d89b8: Status 404 returned error can't find the container with id c4207af394aa56d0bfec13a3dc31663819452dda933129415ecf48d24c2d89b8
Sep 10 21:09:18.938985 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:18.938956    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-18b9ac-2x2db" event={"ID":"1520346b-4b62-44fd-b61b-888a3789895a","Type":"ContainerStarted","Data":"c4207af394aa56d0bfec13a3dc31663819452dda933129415ecf48d24c2d89b8"}
Sep 10 21:09:19.946466 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:19.946435    2564 generic.go:356] "Generic (PLEG): container finished" podID="1520346b-4b62-44fd-b61b-888a3789895a" containerID="616c87c511604f3cdb8cbbe70ec8015795306452ab57a387860a52b54dde07fc" exitCode=0
Sep 10 21:09:19.946949 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:19.946529    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-18b9ac-2x2db" event={"ID":"1520346b-4b62-44fd-b61b-888a3789895a","Type":"ContainerDied","Data":"616c87c511604f3cdb8cbbe70ec8015795306452ab57a387860a52b54dde07fc"}
Sep 10 21:09:21.084089 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:21.084062    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-18b9ac-2x2db"
Sep 10 21:09:21.251630 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:21.251560    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1520346b-4b62-44fd-b61b-888a3789895a-maas-api-service-ca\") pod \"1520346b-4b62-44fd-b61b-888a3789895a\" (UID: \"1520346b-4b62-44fd-b61b-888a3789895a\") "
Sep 10 21:09:21.251924 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:21.251903    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1520346b-4b62-44fd-b61b-888a3789895a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1520346b-4b62-44fd-b61b-888a3789895a" (UID: "1520346b-4b62-44fd-b61b-888a3789895a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:09:21.352663 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:21.352626    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1520346b-4b62-44fd-b61b-888a3789895a-maas-api-service-ca\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:09:21.959239 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:21.959201    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-18b9ac-2x2db" event={"ID":"1520346b-4b62-44fd-b61b-888a3789895a","Type":"ContainerDied","Data":"c4207af394aa56d0bfec13a3dc31663819452dda933129415ecf48d24c2d89b8"}
Sep 10 21:09:21.959239 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:21.959235    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-18b9ac-2x2db"
Sep 10 21:09:21.959444 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:21.959285    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4207af394aa56d0bfec13a3dc31663819452dda933129415ecf48d24c2d89b8"
Sep 10 21:09:28.962752 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:28.962714    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-18b9ac-6cf5b5c9f-2whf7"]
Sep 10 21:09:28.963231 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:28.962944    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-18b9ac-6cf5b5c9f-2whf7" podUID="170ee806-8562-4304-8eb4-9ce73100f449" containerName="payload-processing" containerID="cri-o://6f0f3b40ecdbff57622f697c531dc6ab6a1df6ef0c9bd572cd6b79dfc9952b1c" gracePeriod=30
Sep 10 21:09:28.965415 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:28.965386    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-18b9ac-5f79c684b-6hxxz"]
Sep 10 21:09:28.965599 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:28.965565    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18b9ac-5f79c684b-6hxxz" podUID="e10c2e9e-56e7-4639-bda8-b5c982287310" containerName="payload-pre-processing" containerID="cri-o://7872bdd2b02054afcba6242d5fe16643619ffba92dfb9f1a905685cdecf30961" gracePeriod=30
Sep 10 21:09:29.134051 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:29.133991    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-18b9ac-2x2db"]
Sep 10 21:09:29.137650 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:29.137623    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-18b9ac-2x2db"]
Sep 10 21:09:29.234696 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:29.234673    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-18b9ac-6cf5b5c9f-2whf7"
Sep 10 21:09:29.237458 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:29.237436    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18b9ac-5f79c684b-6hxxz"
Sep 10 21:09:29.410868 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:29.410831    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e10c2e9e-56e7-4639-bda8-b5c982287310-plugins-config-volume\") pod \"e10c2e9e-56e7-4639-bda8-b5c982287310\" (UID: \"e10c2e9e-56e7-4639-bda8-b5c982287310\") "
Sep 10 21:09:29.411049 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:29.410917    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9gwr\" (UniqueName: \"kubernetes.io/projected/e10c2e9e-56e7-4639-bda8-b5c982287310-kube-api-access-w9gwr\") pod \"e10c2e9e-56e7-4639-bda8-b5c982287310\" (UID: \"e10c2e9e-56e7-4639-bda8-b5c982287310\") "
Sep 10 21:09:29.411049 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:29.410944    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/170ee806-8562-4304-8eb4-9ce73100f449-plugins-config-volume\") pod \"170ee806-8562-4304-8eb4-9ce73100f449\" (UID: \"170ee806-8562-4304-8eb4-9ce73100f449\") "
Sep 10 21:09:29.411049 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:29.410961    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sklq9\" (UniqueName: \"kubernetes.io/projected/170ee806-8562-4304-8eb4-9ce73100f449-kube-api-access-sklq9\") pod \"170ee806-8562-4304-8eb4-9ce73100f449\" (UID: \"170ee806-8562-4304-8eb4-9ce73100f449\") "
Sep 10 21:09:29.411361 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:29.411330    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/170ee806-8562-4304-8eb4-9ce73100f449-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "170ee806-8562-4304-8eb4-9ce73100f449" (UID: "170ee806-8562-4304-8eb4-9ce73100f449"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:09:29.411452 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:29.411330    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e10c2e9e-56e7-4639-bda8-b5c982287310-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e10c2e9e-56e7-4639-bda8-b5c982287310" (UID: "e10c2e9e-56e7-4639-bda8-b5c982287310"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:09:29.413121 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:29.413092    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/170ee806-8562-4304-8eb4-9ce73100f449-kube-api-access-sklq9" (OuterVolumeSpecName: "kube-api-access-sklq9") pod "170ee806-8562-4304-8eb4-9ce73100f449" (UID: "170ee806-8562-4304-8eb4-9ce73100f449"). InnerVolumeSpecName "kube-api-access-sklq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:09:29.413229 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:29.413129    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e10c2e9e-56e7-4639-bda8-b5c982287310-kube-api-access-w9gwr" (OuterVolumeSpecName: "kube-api-access-w9gwr") pod "e10c2e9e-56e7-4639-bda8-b5c982287310" (UID: "e10c2e9e-56e7-4639-bda8-b5c982287310"). InnerVolumeSpecName "kube-api-access-w9gwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:09:29.512475 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:29.512416    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e10c2e9e-56e7-4639-bda8-b5c982287310-plugins-config-volume\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:09:29.512475 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:29.512439    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9gwr\" (UniqueName: \"kubernetes.io/projected/e10c2e9e-56e7-4639-bda8-b5c982287310-kube-api-access-w9gwr\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:09:29.512475 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:29.512449    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/170ee806-8562-4304-8eb4-9ce73100f449-plugins-config-volume\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:09:29.512475 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:29.512459    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sklq9\" (UniqueName: \"kubernetes.io/projected/170ee806-8562-4304-8eb4-9ce73100f449-kube-api-access-sklq9\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:09:30.008761 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:30.008730    2564 generic.go:356] "Generic (PLEG): container finished" podID="e10c2e9e-56e7-4639-bda8-b5c982287310" containerID="7872bdd2b02054afcba6242d5fe16643619ffba92dfb9f1a905685cdecf30961" exitCode=0
Sep 10 21:09:30.009119 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:30.008776    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18b9ac-5f79c684b-6hxxz"
Sep 10 21:09:30.009119 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:30.008823    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18b9ac-5f79c684b-6hxxz" event={"ID":"e10c2e9e-56e7-4639-bda8-b5c982287310","Type":"ContainerDied","Data":"7872bdd2b02054afcba6242d5fe16643619ffba92dfb9f1a905685cdecf30961"}
Sep 10 21:09:30.009119 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:30.008882    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-18b9ac-5f79c684b-6hxxz" event={"ID":"e10c2e9e-56e7-4639-bda8-b5c982287310","Type":"ContainerDied","Data":"0588db51a9e4a13c2800fcb59e12642967874ab214478e68d92a8dcbd2d544bf"}
Sep 10 21:09:30.009119 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:30.008908    2564 scope.go:117] "RemoveContainer" containerID="7872bdd2b02054afcba6242d5fe16643619ffba92dfb9f1a905685cdecf30961"
Sep 10 21:09:30.010332 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:30.010308    2564 generic.go:356] "Generic (PLEG): container finished" podID="170ee806-8562-4304-8eb4-9ce73100f449" containerID="6f0f3b40ecdbff57622f697c531dc6ab6a1df6ef0c9bd572cd6b79dfc9952b1c" exitCode=0
Sep 10 21:09:30.010400 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:30.010363    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-18b9ac-6cf5b5c9f-2whf7"
Sep 10 21:09:30.010400 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:30.010374    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-18b9ac-6cf5b5c9f-2whf7" event={"ID":"170ee806-8562-4304-8eb4-9ce73100f449","Type":"ContainerDied","Data":"6f0f3b40ecdbff57622f697c531dc6ab6a1df6ef0c9bd572cd6b79dfc9952b1c"}
Sep 10 21:09:30.010498 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:30.010412    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-18b9ac-6cf5b5c9f-2whf7" event={"ID":"170ee806-8562-4304-8eb4-9ce73100f449","Type":"ContainerDied","Data":"e75e7c71d2a40dc85b7741e684782f72764ccfb2d7988086c6360518076614de"}
Sep 10 21:09:30.018325 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:30.018304    2564 scope.go:117] "RemoveContainer" containerID="7872bdd2b02054afcba6242d5fe16643619ffba92dfb9f1a905685cdecf30961"
Sep 10 21:09:30.018639 ip-10-0-141-185 kubenswrapper[2564]: E0910 21:09:30.018614    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7872bdd2b02054afcba6242d5fe16643619ffba92dfb9f1a905685cdecf30961\": container with ID starting with 7872bdd2b02054afcba6242d5fe16643619ffba92dfb9f1a905685cdecf30961 not found: ID does not exist" containerID="7872bdd2b02054afcba6242d5fe16643619ffba92dfb9f1a905685cdecf30961"
Sep 10 21:09:30.018728 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:30.018647    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7872bdd2b02054afcba6242d5fe16643619ffba92dfb9f1a905685cdecf30961"} err="failed to get container status \"7872bdd2b02054afcba6242d5fe16643619ffba92dfb9f1a905685cdecf30961\": rpc error: code = NotFound desc = could not find container \"7872bdd2b02054afcba6242d5fe16643619ffba92dfb9f1a905685cdecf30961\": container with ID starting with 7872bdd2b02054afcba6242d5fe16643619ffba92dfb9f1a905685cdecf30961 not found: ID does not exist"
Sep 10 21:09:30.018728 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:30.018666    2564 scope.go:117] "RemoveContainer" containerID="6f0f3b40ecdbff57622f697c531dc6ab6a1df6ef0c9bd572cd6b79dfc9952b1c"
Sep 10 21:09:30.027531 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:30.027514    2564 scope.go:117] "RemoveContainer" containerID="6f0f3b40ecdbff57622f697c531dc6ab6a1df6ef0c9bd572cd6b79dfc9952b1c"
Sep 10 21:09:30.027804 ip-10-0-141-185 kubenswrapper[2564]: E0910 21:09:30.027784    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f0f3b40ecdbff57622f697c531dc6ab6a1df6ef0c9bd572cd6b79dfc9952b1c\": container with ID starting with 6f0f3b40ecdbff57622f697c531dc6ab6a1df6ef0c9bd572cd6b79dfc9952b1c not found: ID does not exist" containerID="6f0f3b40ecdbff57622f697c531dc6ab6a1df6ef0c9bd572cd6b79dfc9952b1c"
Sep 10 21:09:30.027848 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:30.027811    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f0f3b40ecdbff57622f697c531dc6ab6a1df6ef0c9bd572cd6b79dfc9952b1c"} err="failed to get container status \"6f0f3b40ecdbff57622f697c531dc6ab6a1df6ef0c9bd572cd6b79dfc9952b1c\": rpc error: code = NotFound desc = could not find container \"6f0f3b40ecdbff57622f697c531dc6ab6a1df6ef0c9bd572cd6b79dfc9952b1c\": container with ID starting with 6f0f3b40ecdbff57622f697c531dc6ab6a1df6ef0c9bd572cd6b79dfc9952b1c not found: ID does not exist"
Sep 10 21:09:30.036431 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:30.036409    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-18b9ac-5f79c684b-6hxxz"]
Sep 10 21:09:30.041332 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:30.041312    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-18b9ac-5f79c684b-6hxxz"]
Sep 10 21:09:30.051536 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:30.051515    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-18b9ac-6cf5b5c9f-2whf7"]
Sep 10 21:09:30.055077 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:30.055058    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-18b9ac-6cf5b5c9f-2whf7"]
Sep 10 21:09:30.541502 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:30.541465    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1520346b-4b62-44fd-b61b-888a3789895a" path="/var/lib/kubelet/pods/1520346b-4b62-44fd-b61b-888a3789895a/volumes"
Sep 10 21:09:30.541820 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:30.541805    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="170ee806-8562-4304-8eb4-9ce73100f449" path="/var/lib/kubelet/pods/170ee806-8562-4304-8eb4-9ce73100f449/volumes"
Sep 10 21:09:30.542132 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:30.542119    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e10c2e9e-56e7-4639-bda8-b5c982287310" path="/var/lib/kubelet/pods/e10c2e9e-56e7-4639-bda8-b5c982287310/volumes"
Sep 10 21:09:35.926748 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:35.926716    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"]
Sep 10 21:09:35.927240 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:35.926923    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z" podUID="e013fecc-7d3c-48f6-a7ef-c5695f284b39" containerName="istio-proxy" containerID="cri-o://8fbf9f362bb0869e1a75890cf0777f5e6992e2463edd97b019635874cc848e2e" gracePeriod=30
Sep 10 21:09:36.278317 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:36.278232    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3fc2b8-tgtmr"]
Sep 10 21:09:36.278632 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:36.278612    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e10c2e9e-56e7-4639-bda8-b5c982287310" containerName="payload-pre-processing"
Sep 10 21:09:36.278678 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:36.278636    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e10c2e9e-56e7-4639-bda8-b5c982287310" containerName="payload-pre-processing"
Sep 10 21:09:36.278678 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:36.278651    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1520346b-4b62-44fd-b61b-888a3789895a" containerName="revoke-keys"
Sep 10 21:09:36.278678 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:36.278662    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1520346b-4b62-44fd-b61b-888a3789895a" containerName="revoke-keys"
Sep 10 21:09:36.278792 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:36.278687    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="170ee806-8562-4304-8eb4-9ce73100f449" containerName="payload-processing"
Sep 10 21:09:36.278792 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:36.278697    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="170ee806-8562-4304-8eb4-9ce73100f449" containerName="payload-processing"
Sep 10 21:09:36.278792 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:36.278780    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1520346b-4b62-44fd-b61b-888a3789895a" containerName="revoke-keys"
Sep 10 21:09:36.278931 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:36.278794    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="170ee806-8562-4304-8eb4-9ce73100f449" containerName="payload-processing"
Sep 10 21:09:36.278931 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:36.278803    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e10c2e9e-56e7-4639-bda8-b5c982287310" containerName="payload-pre-processing"
Sep 10 21:09:36.281834 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:36.281815    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3fc2b8-tgtmr"
Sep 10 21:09:36.284278 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:36.284253    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:09:36.284415 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:36.284326    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plv2h\""
Sep 10 21:09:36.289918 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:36.289715    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3fc2b8-tgtmr"]
Sep 10 21:09:36.364033 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:36.364005    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b80f1c8-4260-4571-8d5c-5aa869b71bb7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-3fc2b8-tgtmr\" (UID: \"3b80f1c8-4260-4571-8d5c-5aa869b71bb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3fc2b8-tgtmr"
Sep 10 21:09:36.465438 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:36.465399    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b80f1c8-4260-4571-8d5c-5aa869b71bb7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-3fc2b8-tgtmr\" (UID: \"3b80f1c8-4260-4571-8d5c-5aa869b71bb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3fc2b8-tgtmr"
Sep 10 21:09:36.466022 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:36.465996    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b80f1c8-4260-4571-8d5c-5aa869b71bb7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-3fc2b8-tgtmr\" (UID: \"3b80f1c8-4260-4571-8d5c-5aa869b71bb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3fc2b8-tgtmr"
Sep 10 21:09:36.594472 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:36.594450    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3fc2b8-tgtmr"
Sep 10 21:09:36.868917 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:36.868892    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3fc2b8-tgtmr"]
Sep 10 21:09:36.870528 ip-10-0-141-185 kubenswrapper[2564]: W0910 21:09:36.870496    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b80f1c8_4260_4571_8d5c_5aa869b71bb7.slice/crio-f777dd4826a04424b943f651a6fd3677f1e8467bf81e3cfe70c7748053172ac3 WatchSource:0}: Error finding container f777dd4826a04424b943f651a6fd3677f1e8467bf81e3cfe70c7748053172ac3: Status 404 returned error can't find the container with id f777dd4826a04424b943f651a6fd3677f1e8467bf81e3cfe70c7748053172ac3
Sep 10 21:09:37.054607 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:37.054578    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3fc2b8-tgtmr" event={"ID":"3b80f1c8-4260-4571-8d5c-5aa869b71bb7","Type":"ContainerStarted","Data":"f777dd4826a04424b943f651a6fd3677f1e8467bf81e3cfe70c7748053172ac3"}
Sep 10 21:09:38.061965 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:38.061928    2564 generic.go:356] "Generic (PLEG): container finished" podID="3b80f1c8-4260-4571-8d5c-5aa869b71bb7" containerID="a02f99d01d3617236a8786d791eedb5e089a363dce6d9ec38516e62909afac99" exitCode=0
Sep 10 21:09:38.062379 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:38.062022    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3fc2b8-tgtmr" event={"ID":"3b80f1c8-4260-4571-8d5c-5aa869b71bb7","Type":"ContainerDied","Data":"a02f99d01d3617236a8786d791eedb5e089a363dce6d9ec38516e62909afac99"}
Sep 10 21:09:39.197973 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:39.197946    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3fc2b8-tgtmr"
Sep 10 21:09:39.285998 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:39.285965    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b80f1c8-4260-4571-8d5c-5aa869b71bb7-maas-api-service-ca\") pod \"3b80f1c8-4260-4571-8d5c-5aa869b71bb7\" (UID: \"3b80f1c8-4260-4571-8d5c-5aa869b71bb7\") "
Sep 10 21:09:39.286308 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:39.286286    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b80f1c8-4260-4571-8d5c-5aa869b71bb7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3b80f1c8-4260-4571-8d5c-5aa869b71bb7" (UID: "3b80f1c8-4260-4571-8d5c-5aa869b71bb7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:09:39.387200 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:39.387126    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b80f1c8-4260-4571-8d5c-5aa869b71bb7-maas-api-service-ca\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:09:40.009823 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:40.009777    2564 patch_prober.go:28] interesting pod/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 21:09:40.009981 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:40.009857    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z" podUID="e013fecc-7d3c-48f6-a7ef-c5695f284b39" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 21:09:40.076742 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:40.076711    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3fc2b8-tgtmr" event={"ID":"3b80f1c8-4260-4571-8d5c-5aa869b71bb7","Type":"ContainerDied","Data":"f777dd4826a04424b943f651a6fd3677f1e8467bf81e3cfe70c7748053172ac3"}
Sep 10 21:09:40.076742 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:40.076734    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3fc2b8-tgtmr"
Sep 10 21:09:40.076927 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:40.076764    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f777dd4826a04424b943f651a6fd3677f1e8467bf81e3cfe70c7748053172ac3"
Sep 10 21:09:41.085080 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:41.085046    2564 generic.go:356] "Generic (PLEG): container finished" podID="e013fecc-7d3c-48f6-a7ef-c5695f284b39" containerID="8fbf9f362bb0869e1a75890cf0777f5e6992e2463edd97b019635874cc848e2e" exitCode=0
Sep 10 21:09:41.085440 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:41.085112    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z" event={"ID":"e013fecc-7d3c-48f6-a7ef-c5695f284b39","Type":"ContainerDied","Data":"8fbf9f362bb0869e1a75890cf0777f5e6992e2463edd97b019635874cc848e2e"}
Sep 10 21:09:41.085440 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:41.085197    2564 scope.go:117] "RemoveContainer" containerID="425d4f89c49ccead979d48446fc28034ae9e2c489360e28f4b17e97ab18ec058"
Sep 10 21:09:41.178377 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:41.178357    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:09:41.200755 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:41.200695    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e013fecc-7d3c-48f6-a7ef-c5695f284b39-istiod-ca-cert\") pod \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") "
Sep 10 21:09:41.200755 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:41.200733    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e013fecc-7d3c-48f6-a7ef-c5695f284b39-istio-podinfo\") pod \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") "
Sep 10 21:09:41.200891 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:41.200781    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e013fecc-7d3c-48f6-a7ef-c5695f284b39-workload-certs\") pod \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") "
Sep 10 21:09:41.200965 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:41.200943    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e013fecc-7d3c-48f6-a7ef-c5695f284b39-istio-envoy\") pod \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") "
Sep 10 21:09:41.201024 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:41.200985    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e013fecc-7d3c-48f6-a7ef-c5695f284b39-istio-data\") pod \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") "
Sep 10 21:09:41.201079 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:41.201050    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e013fecc-7d3c-48f6-a7ef-c5695f284b39-workload-socket\") pod \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") "
Sep 10 21:09:41.201079 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:41.201062    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e013fecc-7d3c-48f6-a7ef-c5695f284b39-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e013fecc-7d3c-48f6-a7ef-c5695f284b39" (UID: "e013fecc-7d3c-48f6-a7ef-c5695f284b39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:09:41.201228 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:41.201081    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e013fecc-7d3c-48f6-a7ef-c5695f284b39-istio-token\") pod \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") "
Sep 10 21:09:41.201228 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:41.201108    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52kvh\" (UniqueName: \"kubernetes.io/projected/e013fecc-7d3c-48f6-a7ef-c5695f284b39-kube-api-access-52kvh\") pod \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") "
Sep 10 21:09:41.201228 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:41.201110    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e013fecc-7d3c-48f6-a7ef-c5695f284b39-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e013fecc-7d3c-48f6-a7ef-c5695f284b39" (UID: "e013fecc-7d3c-48f6-a7ef-c5695f284b39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:09:41.201228 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:41.201201    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e013fecc-7d3c-48f6-a7ef-c5695f284b39-credential-socket\") pod \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\" (UID: \"e013fecc-7d3c-48f6-a7ef-c5695f284b39\") "
Sep 10 21:09:41.201436 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:41.201330    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e013fecc-7d3c-48f6-a7ef-c5695f284b39-istio-data" (OuterVolumeSpecName: "istio-data") pod "e013fecc-7d3c-48f6-a7ef-c5695f284b39" (UID: "e013fecc-7d3c-48f6-a7ef-c5695f284b39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:09:41.201550 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:41.201513    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e013fecc-7d3c-48f6-a7ef-c5695f284b39-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e013fecc-7d3c-48f6-a7ef-c5695f284b39" (UID: "e013fecc-7d3c-48f6-a7ef-c5695f284b39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:09:41.201650 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:41.201556    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e013fecc-7d3c-48f6-a7ef-c5695f284b39-istiod-ca-cert\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:09:41.201650 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:41.201576    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e013fecc-7d3c-48f6-a7ef-c5695f284b39-workload-certs\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:09:41.201650 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:41.201588    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e013fecc-7d3c-48f6-a7ef-c5695f284b39-istio-data\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:09:41.201810 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:41.201657    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e013fecc-7d3c-48f6-a7ef-c5695f284b39-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e013fecc-7d3c-48f6-a7ef-c5695f284b39" (UID: "e013fecc-7d3c-48f6-a7ef-c5695f284b39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:09:41.203514 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:41.203488    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e013fecc-7d3c-48f6-a7ef-c5695f284b39-istio-token" (OuterVolumeSpecName: "istio-token") pod "e013fecc-7d3c-48f6-a7ef-c5695f284b39" (UID: "e013fecc-7d3c-48f6-a7ef-c5695f284b39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:09:41.203656 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:41.203637    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e013fecc-7d3c-48f6-a7ef-c5695f284b39-kube-api-access-52kvh" (OuterVolumeSpecName: "kube-api-access-52kvh") pod "e013fecc-7d3c-48f6-a7ef-c5695f284b39" (UID: "e013fecc-7d3c-48f6-a7ef-c5695f284b39"). InnerVolumeSpecName "kube-api-access-52kvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:09:41.203777 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:41.203755    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e013fecc-7d3c-48f6-a7ef-c5695f284b39-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e013fecc-7d3c-48f6-a7ef-c5695f284b39" (UID: "e013fecc-7d3c-48f6-a7ef-c5695f284b39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:09:41.203777 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:41.203767    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e013fecc-7d3c-48f6-a7ef-c5695f284b39-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e013fecc-7d3c-48f6-a7ef-c5695f284b39" (UID: "e013fecc-7d3c-48f6-a7ef-c5695f284b39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:09:41.302428 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:41.302403    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e013fecc-7d3c-48f6-a7ef-c5695f284b39-istio-podinfo\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:09:41.302428 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:41.302428    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e013fecc-7d3c-48f6-a7ef-c5695f284b39-istio-envoy\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:09:41.302569 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:41.302443    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e013fecc-7d3c-48f6-a7ef-c5695f284b39-workload-socket\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:09:41.302569 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:41.302456    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e013fecc-7d3c-48f6-a7ef-c5695f284b39-istio-token\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:09:41.302569 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:41.302469    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52kvh\" (UniqueName: \"kubernetes.io/projected/e013fecc-7d3c-48f6-a7ef-c5695f284b39-kube-api-access-52kvh\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:09:41.302569 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:41.302481    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e013fecc-7d3c-48f6-a7ef-c5695f284b39-credential-socket\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:09:42.092635 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:42.092601    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"
Sep 10 21:09:42.093132 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:42.092627    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z" event={"ID":"e013fecc-7d3c-48f6-a7ef-c5695f284b39","Type":"ContainerDied","Data":"923b561ecb424b7d790da5343b6137da354f748703df674ef937419999a8111e"}
Sep 10 21:09:42.093132 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:42.092699    2564 scope.go:117] "RemoveContainer" containerID="8fbf9f362bb0869e1a75890cf0777f5e6992e2463edd97b019635874cc848e2e"
Sep 10 21:09:42.116533 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:42.116503    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"]
Sep 10 21:09:42.120225 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:42.120200    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-18b9ac-openshift-default-d7d8dc98d-pgz2z"]
Sep 10 21:09:42.547969 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:42.547884    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e013fecc-7d3c-48f6-a7ef-c5695f284b39" path="/var/lib/kubelet/pods/e013fecc-7d3c-48f6-a7ef-c5695f284b39/volumes"
Sep 10 21:09:46.857473 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:46.857438    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3fc2b8-29817900-dh9fz"]
Sep 10 21:09:46.860150 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:46.860119    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3fc2b8-29817900-dh9fz"]
Sep 10 21:09:47.176200 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:47.176093    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3fc2b8-tgtmr"]
Sep 10 21:09:47.178812 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:47.178784    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3fc2b8-tgtmr"]
Sep 10 21:09:48.539979 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:48.539937    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09ae561c-a2b9-4a2c-a95d-9c1125fc2d8e" path="/var/lib/kubelet/pods/09ae561c-a2b9-4a2c-a95d-9c1125fc2d8e/volumes"
Sep 10 21:09:48.540520 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:48.540498    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b80f1c8-4260-4571-8d5c-5aa869b71bb7" path="/var/lib/kubelet/pods/3b80f1c8-4260-4571-8d5c-5aa869b71bb7/volumes"
Sep 10 21:09:54.409945 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:54.409900    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4fe4f-wxw5h"]
Sep 10 21:09:54.410563 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:54.410447    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e013fecc-7d3c-48f6-a7ef-c5695f284b39" containerName="istio-proxy"
Sep 10 21:09:54.410563 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:54.410471    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e013fecc-7d3c-48f6-a7ef-c5695f284b39" containerName="istio-proxy"
Sep 10 21:09:54.410563 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:54.410483    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e013fecc-7d3c-48f6-a7ef-c5695f284b39" containerName="istio-proxy"
Sep 10 21:09:54.410563 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:54.410494    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e013fecc-7d3c-48f6-a7ef-c5695f284b39" containerName="istio-proxy"
Sep 10 21:09:54.410563 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:54.410509    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b80f1c8-4260-4571-8d5c-5aa869b71bb7" containerName="revoke-keys"
Sep 10 21:09:54.410563 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:54.410518    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b80f1c8-4260-4571-8d5c-5aa869b71bb7" containerName="revoke-keys"
Sep 10 21:09:54.410878 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:54.410626    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e013fecc-7d3c-48f6-a7ef-c5695f284b39" containerName="istio-proxy"
Sep 10 21:09:54.410878 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:54.410641    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b80f1c8-4260-4571-8d5c-5aa869b71bb7" containerName="revoke-keys"
Sep 10 21:09:54.410878 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:54.410650    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e013fecc-7d3c-48f6-a7ef-c5695f284b39" containerName="istio-proxy"
Sep 10 21:09:54.415503 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:54.415479    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4fe4f-wxw5h"
Sep 10 21:09:54.417871 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:54.417828    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:09:54.418037 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:54.417934    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-plv2h\""
Sep 10 21:09:54.421772 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:54.421747    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4fe4f-wxw5h"]
Sep 10 21:09:54.502959 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:54.502929    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38a29dd5-81c1-45b8-a2ef-f66648e15258-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f4fe4f-wxw5h\" (UID: \"38a29dd5-81c1-45b8-a2ef-f66648e15258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4fe4f-wxw5h"
Sep 10 21:09:54.604100 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:54.604067    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38a29dd5-81c1-45b8-a2ef-f66648e15258-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f4fe4f-wxw5h\" (UID: \"38a29dd5-81c1-45b8-a2ef-f66648e15258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4fe4f-wxw5h"
Sep 10 21:09:54.604700 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:54.604679    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38a29dd5-81c1-45b8-a2ef-f66648e15258-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f4fe4f-wxw5h\" (UID: \"38a29dd5-81c1-45b8-a2ef-f66648e15258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4fe4f-wxw5h"
Sep 10 21:09:54.730046 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:54.729987    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4fe4f-wxw5h"
Sep 10 21:09:55.001371 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:55.001340    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4fe4f-wxw5h"]
Sep 10 21:09:55.003803 ip-10-0-141-185 kubenswrapper[2564]: W0910 21:09:55.003777    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a29dd5_81c1_45b8_a2ef_f66648e15258.slice/crio-c0028bdc41d5d96e6144cde24d8cb2e04cbdc32e6cc522797a7e8d9d50e6bc38 WatchSource:0}: Error finding container c0028bdc41d5d96e6144cde24d8cb2e04cbdc32e6cc522797a7e8d9d50e6bc38: Status 404 returned error can't find the container with id c0028bdc41d5d96e6144cde24d8cb2e04cbdc32e6cc522797a7e8d9d50e6bc38
Sep 10 21:09:55.171081 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:55.171052    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4fe4f-wxw5h" event={"ID":"38a29dd5-81c1-45b8-a2ef-f66648e15258","Type":"ContainerStarted","Data":"c0028bdc41d5d96e6144cde24d8cb2e04cbdc32e6cc522797a7e8d9d50e6bc38"}
Sep 10 21:09:56.179656 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:56.179616    2564 generic.go:356] "Generic (PLEG): container finished" podID="38a29dd5-81c1-45b8-a2ef-f66648e15258" containerID="56ee05822db047af167064f4b9194b515f9d7694816260f5e95806e0fadd943b" exitCode=0
Sep 10 21:09:56.180093 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:56.179670    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4fe4f-wxw5h" event={"ID":"38a29dd5-81c1-45b8-a2ef-f66648e15258","Type":"ContainerDied","Data":"56ee05822db047af167064f4b9194b515f9d7694816260f5e95806e0fadd943b"}
Sep 10 21:09:57.321592 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:57.321568    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4fe4f-wxw5h"
Sep 10 21:09:57.424007 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:57.423982    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38a29dd5-81c1-45b8-a2ef-f66648e15258-maas-api-service-ca\") pod \"38a29dd5-81c1-45b8-a2ef-f66648e15258\" (UID: \"38a29dd5-81c1-45b8-a2ef-f66648e15258\") "
Sep 10 21:09:57.424349 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:57.424326    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38a29dd5-81c1-45b8-a2ef-f66648e15258-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "38a29dd5-81c1-45b8-a2ef-f66648e15258" (UID: "38a29dd5-81c1-45b8-a2ef-f66648e15258"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:09:57.525093 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:57.525039    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38a29dd5-81c1-45b8-a2ef-f66648e15258-maas-api-service-ca\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:09:58.193858 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:58.193829    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4fe4f-wxw5h"
Sep 10 21:09:58.194010 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:58.193859    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4fe4f-wxw5h" event={"ID":"38a29dd5-81c1-45b8-a2ef-f66648e15258","Type":"ContainerDied","Data":"c0028bdc41d5d96e6144cde24d8cb2e04cbdc32e6cc522797a7e8d9d50e6bc38"}
Sep 10 21:09:58.194010 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:09:58.193909    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0028bdc41d5d96e6144cde24d8cb2e04cbdc32e6cc522797a7e8d9d50e6bc38"
Sep 10 21:10:00.858591 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:00.858558    2564 scope.go:117] "RemoveContainer" containerID="d355a172c4e00bc8be735cdcb265b1858440eb5e9df267e43b35bfa02e28d7a6"
Sep 10 21:10:00.867053 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:00.867033    2564 scope.go:117] "RemoveContainer" containerID="f219ca5bb8eb84fca80857e6f2b5df20f99c5d4e199e0a741afa2d552917dc61"
Sep 10 21:10:05.128560 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:05.128524    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f4fe4f-698489dd7f-jq8jc"]
Sep 10 21:10:05.129481 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:05.128779    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4fe4f-698489dd7f-jq8jc" podUID="415e549a-63ca-459a-ad45-452a01d6b98a" containerName="payload-processing" containerID="cri-o://eb2d560db59642071db059e4c2c55a50bbb70acfe9c0535481893c6b3699aff8" gracePeriod=30
Sep 10 21:10:05.297761 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:05.297726    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4fe4f-wxw5h"]
Sep 10 21:10:05.299810 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:05.299783    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4fe4f-wxw5h"]
Sep 10 21:10:05.386289 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:05.386232    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4fe4f-698489dd7f-jq8jc"
Sep 10 21:10:05.482936 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:05.482908    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/415e549a-63ca-459a-ad45-452a01d6b98a-plugins-config-volume\") pod \"415e549a-63ca-459a-ad45-452a01d6b98a\" (UID: \"415e549a-63ca-459a-ad45-452a01d6b98a\") "
Sep 10 21:10:05.483109 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:05.483002    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrjwg\" (UniqueName: \"kubernetes.io/projected/415e549a-63ca-459a-ad45-452a01d6b98a-kube-api-access-wrjwg\") pod \"415e549a-63ca-459a-ad45-452a01d6b98a\" (UID: \"415e549a-63ca-459a-ad45-452a01d6b98a\") "
Sep 10 21:10:05.483287 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:05.483263    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/415e549a-63ca-459a-ad45-452a01d6b98a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "415e549a-63ca-459a-ad45-452a01d6b98a" (UID: "415e549a-63ca-459a-ad45-452a01d6b98a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:10:05.485034 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:05.485014    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/415e549a-63ca-459a-ad45-452a01d6b98a-kube-api-access-wrjwg" (OuterVolumeSpecName: "kube-api-access-wrjwg") pod "415e549a-63ca-459a-ad45-452a01d6b98a" (UID: "415e549a-63ca-459a-ad45-452a01d6b98a"). InnerVolumeSpecName "kube-api-access-wrjwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:10:05.583652 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:05.583624    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrjwg\" (UniqueName: \"kubernetes.io/projected/415e549a-63ca-459a-ad45-452a01d6b98a-kube-api-access-wrjwg\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:10:05.583652 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:05.583651    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/415e549a-63ca-459a-ad45-452a01d6b98a-plugins-config-volume\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:10:06.244631 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:06.244594    2564 generic.go:356] "Generic (PLEG): container finished" podID="415e549a-63ca-459a-ad45-452a01d6b98a" containerID="eb2d560db59642071db059e4c2c55a50bbb70acfe9c0535481893c6b3699aff8" exitCode=0
Sep 10 21:10:06.245046 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:06.244640    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4fe4f-698489dd7f-jq8jc"
Sep 10 21:10:06.245046 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:06.244681    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4fe4f-698489dd7f-jq8jc" event={"ID":"415e549a-63ca-459a-ad45-452a01d6b98a","Type":"ContainerDied","Data":"eb2d560db59642071db059e4c2c55a50bbb70acfe9c0535481893c6b3699aff8"}
Sep 10 21:10:06.245046 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:06.244739    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4fe4f-698489dd7f-jq8jc" event={"ID":"415e549a-63ca-459a-ad45-452a01d6b98a","Type":"ContainerDied","Data":"2fb9ed9085f6488abb988e56cf2239da1862cb43d12bc6cbfe3e9ad1730ad513"}
Sep 10 21:10:06.245046 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:06.244768    2564 scope.go:117] "RemoveContainer" containerID="eb2d560db59642071db059e4c2c55a50bbb70acfe9c0535481893c6b3699aff8"
Sep 10 21:10:06.254333 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:06.254314    2564 scope.go:117] "RemoveContainer" containerID="eb2d560db59642071db059e4c2c55a50bbb70acfe9c0535481893c6b3699aff8"
Sep 10 21:10:06.254612 ip-10-0-141-185 kubenswrapper[2564]: E0910 21:10:06.254594    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb2d560db59642071db059e4c2c55a50bbb70acfe9c0535481893c6b3699aff8\": container with ID starting with eb2d560db59642071db059e4c2c55a50bbb70acfe9c0535481893c6b3699aff8 not found: ID does not exist" containerID="eb2d560db59642071db059e4c2c55a50bbb70acfe9c0535481893c6b3699aff8"
Sep 10 21:10:06.254666 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:06.254623    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb2d560db59642071db059e4c2c55a50bbb70acfe9c0535481893c6b3699aff8"} err="failed to get container status \"eb2d560db59642071db059e4c2c55a50bbb70acfe9c0535481893c6b3699aff8\": rpc error: code = NotFound desc = could not find container \"eb2d560db59642071db059e4c2c55a50bbb70acfe9c0535481893c6b3699aff8\": container with ID starting with eb2d560db59642071db059e4c2c55a50bbb70acfe9c0535481893c6b3699aff8 not found: ID does not exist"
Sep 10 21:10:06.265312 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:06.265287    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f4fe4f-698489dd7f-jq8jc"]
Sep 10 21:10:06.270908 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:06.270887    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f4fe4f-698489dd7f-jq8jc"]
Sep 10 21:10:06.540826 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:06.540769    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38a29dd5-81c1-45b8-a2ef-f66648e15258" path="/var/lib/kubelet/pods/38a29dd5-81c1-45b8-a2ef-f66648e15258/volumes"
Sep 10 21:10:06.541116 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:06.541102    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="415e549a-63ca-459a-ad45-452a01d6b98a" path="/var/lib/kubelet/pods/415e549a-63ca-459a-ad45-452a01d6b98a/volumes"
Sep 10 21:10:12.247152 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:12.247119    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"]
Sep 10 21:10:12.247512 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:12.247418    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k" podUID="bf04e82d-33fb-488d-950a-00e92d693cbc" containerName="istio-proxy" containerID="cri-o://4520907ee4e7c68acdf3998032323f2b10e86f668ba3d802e67e96da072a088d" gracePeriod=30
Sep 10 21:10:12.287117 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:12.287083    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 21:10:12.287266 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:12.287149    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k" podUID="bf04e82d-33fb-488d-950a-00e92d693cbc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 21:10:17.496892 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:17.496868    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 21:10:17.577061 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:17.577031    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf04e82d-33fb-488d-950a-00e92d693cbc-workload-certs\") pod \"bf04e82d-33fb-488d-950a-00e92d693cbc\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") "
Sep 10 21:10:17.577187 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:17.577096    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5cpn\" (UniqueName: \"kubernetes.io/projected/bf04e82d-33fb-488d-950a-00e92d693cbc-kube-api-access-j5cpn\") pod \"bf04e82d-33fb-488d-950a-00e92d693cbc\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") "
Sep 10 21:10:17.577187 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:17.577120    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf04e82d-33fb-488d-950a-00e92d693cbc-workload-socket\") pod \"bf04e82d-33fb-488d-950a-00e92d693cbc\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") "
Sep 10 21:10:17.577187 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:17.577143    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf04e82d-33fb-488d-950a-00e92d693cbc-istiod-ca-cert\") pod \"bf04e82d-33fb-488d-950a-00e92d693cbc\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") "
Sep 10 21:10:17.577187 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:17.577162    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf04e82d-33fb-488d-950a-00e92d693cbc-istio-token\") pod \"bf04e82d-33fb-488d-950a-00e92d693cbc\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") "
Sep 10 21:10:17.577424 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:17.577210    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf04e82d-33fb-488d-950a-00e92d693cbc-istio-envoy\") pod \"bf04e82d-33fb-488d-950a-00e92d693cbc\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") "
Sep 10 21:10:17.577424 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:17.577322    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf04e82d-33fb-488d-950a-00e92d693cbc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bf04e82d-33fb-488d-950a-00e92d693cbc" (UID: "bf04e82d-33fb-488d-950a-00e92d693cbc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:10:17.577424 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:17.577332    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf04e82d-33fb-488d-950a-00e92d693cbc-istio-data\") pod \"bf04e82d-33fb-488d-950a-00e92d693cbc\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") "
Sep 10 21:10:17.577588 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:17.577424    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf04e82d-33fb-488d-950a-00e92d693cbc-credential-socket\") pod \"bf04e82d-33fb-488d-950a-00e92d693cbc\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") "
Sep 10 21:10:17.577588 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:17.577462    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf04e82d-33fb-488d-950a-00e92d693cbc-istio-podinfo\") pod \"bf04e82d-33fb-488d-950a-00e92d693cbc\" (UID: \"bf04e82d-33fb-488d-950a-00e92d693cbc\") "
Sep 10 21:10:17.577727 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:17.577707    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf04e82d-33fb-488d-950a-00e92d693cbc-workload-certs\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:10:17.577784 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:17.577455    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf04e82d-33fb-488d-950a-00e92d693cbc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bf04e82d-33fb-488d-950a-00e92d693cbc" (UID: "bf04e82d-33fb-488d-950a-00e92d693cbc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:10:17.577784 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:17.577593    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf04e82d-33fb-488d-950a-00e92d693cbc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bf04e82d-33fb-488d-950a-00e92d693cbc" (UID: "bf04e82d-33fb-488d-950a-00e92d693cbc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:10:17.577784 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:17.577667    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf04e82d-33fb-488d-950a-00e92d693cbc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bf04e82d-33fb-488d-950a-00e92d693cbc" (UID: "bf04e82d-33fb-488d-950a-00e92d693cbc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:10:17.577784 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:17.577751    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf04e82d-33fb-488d-950a-00e92d693cbc-istio-data" (OuterVolumeSpecName: "istio-data") pod "bf04e82d-33fb-488d-950a-00e92d693cbc" (UID: "bf04e82d-33fb-488d-950a-00e92d693cbc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:10:17.579234 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:17.579205    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf04e82d-33fb-488d-950a-00e92d693cbc-kube-api-access-j5cpn" (OuterVolumeSpecName: "kube-api-access-j5cpn") pod "bf04e82d-33fb-488d-950a-00e92d693cbc" (UID: "bf04e82d-33fb-488d-950a-00e92d693cbc"). InnerVolumeSpecName "kube-api-access-j5cpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:10:17.579468 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:17.579436    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf04e82d-33fb-488d-950a-00e92d693cbc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bf04e82d-33fb-488d-950a-00e92d693cbc" (UID: "bf04e82d-33fb-488d-950a-00e92d693cbc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:10:17.579710 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:17.579688    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf04e82d-33fb-488d-950a-00e92d693cbc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bf04e82d-33fb-488d-950a-00e92d693cbc" (UID: "bf04e82d-33fb-488d-950a-00e92d693cbc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:10:17.579788 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:17.579690    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf04e82d-33fb-488d-950a-00e92d693cbc-istio-token" (OuterVolumeSpecName: "istio-token") pod "bf04e82d-33fb-488d-950a-00e92d693cbc" (UID: "bf04e82d-33fb-488d-950a-00e92d693cbc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:10:17.678389 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:17.678366    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf04e82d-33fb-488d-950a-00e92d693cbc-credential-socket\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:10:17.678486 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:17.678393    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf04e82d-33fb-488d-950a-00e92d693cbc-istio-podinfo\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:10:17.678486 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:17.678408    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5cpn\" (UniqueName: \"kubernetes.io/projected/bf04e82d-33fb-488d-950a-00e92d693cbc-kube-api-access-j5cpn\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:10:17.678486 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:17.678422    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf04e82d-33fb-488d-950a-00e92d693cbc-workload-socket\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:10:17.678486 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:17.678435    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf04e82d-33fb-488d-950a-00e92d693cbc-istiod-ca-cert\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:10:17.678486 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:17.678448    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf04e82d-33fb-488d-950a-00e92d693cbc-istio-token\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:10:17.678486 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:17.678460    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf04e82d-33fb-488d-950a-00e92d693cbc-istio-envoy\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:10:17.678486 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:17.678471    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf04e82d-33fb-488d-950a-00e92d693cbc-istio-data\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:10:18.317160 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:18.317123    2564 generic.go:356] "Generic (PLEG): container finished" podID="bf04e82d-33fb-488d-950a-00e92d693cbc" containerID="4520907ee4e7c68acdf3998032323f2b10e86f668ba3d802e67e96da072a088d" exitCode=0
Sep 10 21:10:18.317338 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:18.317195    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"
Sep 10 21:10:18.317338 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:18.317222    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k" event={"ID":"bf04e82d-33fb-488d-950a-00e92d693cbc","Type":"ContainerDied","Data":"4520907ee4e7c68acdf3998032323f2b10e86f668ba3d802e67e96da072a088d"}
Sep 10 21:10:18.317338 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:18.317288    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k" event={"ID":"bf04e82d-33fb-488d-950a-00e92d693cbc","Type":"ContainerDied","Data":"373553354f58ef9b39cfb0fcbfe883eefc8d10ee18a1e8519ab43e9c59e29d26"}
Sep 10 21:10:18.317338 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:18.317312    2564 scope.go:117] "RemoveContainer" containerID="4520907ee4e7c68acdf3998032323f2b10e86f668ba3d802e67e96da072a088d"
Sep 10 21:10:18.327742 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:18.327718    2564 scope.go:117] "RemoveContainer" containerID="439aaeaadfd6c24d0689894ccf27460b2c64c87f67b92b7910b22c7554a427bb"
Sep 10 21:10:18.336082 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:18.336059    2564 scope.go:117] "RemoveContainer" containerID="4520907ee4e7c68acdf3998032323f2b10e86f668ba3d802e67e96da072a088d"
Sep 10 21:10:18.336429 ip-10-0-141-185 kubenswrapper[2564]: E0910 21:10:18.336406    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4520907ee4e7c68acdf3998032323f2b10e86f668ba3d802e67e96da072a088d\": container with ID starting with 4520907ee4e7c68acdf3998032323f2b10e86f668ba3d802e67e96da072a088d not found: ID does not exist" containerID="4520907ee4e7c68acdf3998032323f2b10e86f668ba3d802e67e96da072a088d"
Sep 10 21:10:18.336517 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:18.336459    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4520907ee4e7c68acdf3998032323f2b10e86f668ba3d802e67e96da072a088d"} err="failed to get container status \"4520907ee4e7c68acdf3998032323f2b10e86f668ba3d802e67e96da072a088d\": rpc error: code = NotFound desc = could not find container \"4520907ee4e7c68acdf3998032323f2b10e86f668ba3d802e67e96da072a088d\": container with ID starting with 4520907ee4e7c68acdf3998032323f2b10e86f668ba3d802e67e96da072a088d not found: ID does not exist"
Sep 10 21:10:18.336517 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:18.336480    2564 scope.go:117] "RemoveContainer" containerID="439aaeaadfd6c24d0689894ccf27460b2c64c87f67b92b7910b22c7554a427bb"
Sep 10 21:10:18.336793 ip-10-0-141-185 kubenswrapper[2564]: E0910 21:10:18.336768    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"439aaeaadfd6c24d0689894ccf27460b2c64c87f67b92b7910b22c7554a427bb\": container with ID starting with 439aaeaadfd6c24d0689894ccf27460b2c64c87f67b92b7910b22c7554a427bb not found: ID does not exist" containerID="439aaeaadfd6c24d0689894ccf27460b2c64c87f67b92b7910b22c7554a427bb"
Sep 10 21:10:18.336881 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:18.336796    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"439aaeaadfd6c24d0689894ccf27460b2c64c87f67b92b7910b22c7554a427bb"} err="failed to get container status \"439aaeaadfd6c24d0689894ccf27460b2c64c87f67b92b7910b22c7554a427bb\": rpc error: code = NotFound desc = could not find container \"439aaeaadfd6c24d0689894ccf27460b2c64c87f67b92b7910b22c7554a427bb\": container with ID starting with 439aaeaadfd6c24d0689894ccf27460b2c64c87f67b92b7910b22c7554a427bb not found: ID does not exist"
Sep 10 21:10:18.339442 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:18.339416    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"]
Sep 10 21:10:18.343865 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:18.343842    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f4fe4f-openshift-default-6544cdf44-zfv6k"]
Sep 10 21:10:18.541407 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:10:18.541383    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf04e82d-33fb-488d-950a-00e92d693cbc" path="/var/lib/kubelet/pods/bf04e82d-33fb-488d-950a-00e92d693cbc/volumes"
Sep 10 21:11:00.901206 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:11:00.901153    2564 scope.go:117] "RemoveContainer" containerID="9f21cd44c047c5c0b432aab111307a98dd760d5ca36da177857a4d5e6045484b"
Sep 10 21:11:00.909108 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:11:00.909087    2564 scope.go:117] "RemoveContainer" containerID="8128d2c21eb089c80603eeb7563a0bb39b4189d01cde23e660c3b87d1ccfc42f"
Sep 10 21:11:00.916226 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:11:00.916209    2564 scope.go:117] "RemoveContainer" containerID="b1c9d8501ed66fd7ccac6a2d6d0281117777c10f8ac65a44dfe1d5d39461b69f"
Sep 10 21:14:00.963658 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:00.963619    2564 scope.go:117] "RemoveContainer" containerID="81df71c2b619999ea58acc025695991cdd63776239b6042ea871618edcabb499"
Sep 10 21:14:33.097564 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.097520    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s"]
Sep 10 21:14:33.098071 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.098026    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38a29dd5-81c1-45b8-a2ef-f66648e15258" containerName="revoke-keys"
Sep 10 21:14:33.098071 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.098047    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="38a29dd5-81c1-45b8-a2ef-f66648e15258" containerName="revoke-keys"
Sep 10 21:14:33.098071 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.098064    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="415e549a-63ca-459a-ad45-452a01d6b98a" containerName="payload-processing"
Sep 10 21:14:33.098222 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.098074    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="415e549a-63ca-459a-ad45-452a01d6b98a" containerName="payload-processing"
Sep 10 21:14:33.098222 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.098086    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf04e82d-33fb-488d-950a-00e92d693cbc" containerName="istio-proxy"
Sep 10 21:14:33.098222 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.098095    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf04e82d-33fb-488d-950a-00e92d693cbc" containerName="istio-proxy"
Sep 10 21:14:33.098222 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.098108    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf04e82d-33fb-488d-950a-00e92d693cbc" containerName="istio-proxy"
Sep 10 21:14:33.098222 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.098117    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf04e82d-33fb-488d-950a-00e92d693cbc" containerName="istio-proxy"
Sep 10 21:14:33.098395 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.098231    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf04e82d-33fb-488d-950a-00e92d693cbc" containerName="istio-proxy"
Sep 10 21:14:33.098395 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.098247    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf04e82d-33fb-488d-950a-00e92d693cbc" containerName="istio-proxy"
Sep 10 21:14:33.098395 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.098259    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="38a29dd5-81c1-45b8-a2ef-f66648e15258" containerName="revoke-keys"
Sep 10 21:14:33.098395 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.098271    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="415e549a-63ca-459a-ad45-452a01d6b98a" containerName="payload-processing"
Sep 10 21:14:33.101675 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.101653    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s"
Sep 10 21:14:33.103925 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.103898    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-4c60742f-kserve-self-signed-certs\""
Sep 10 21:14:33.110194 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.110144    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s"]
Sep 10 21:14:33.144794 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.144768    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f08c940f-490f-41e2-be8b-873e4d082f78-tls-certs\") pod \"e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s\" (UID: \"f08c940f-490f-41e2-be8b-873e4d082f78\") " pod="llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s"
Sep 10 21:14:33.144881 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.144807    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f08c940f-490f-41e2-be8b-873e4d082f78-home\") pod \"e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s\" (UID: \"f08c940f-490f-41e2-be8b-873e4d082f78\") " pod="llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s"
Sep 10 21:14:33.144881 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.144830    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f08c940f-490f-41e2-be8b-873e4d082f78-dshm\") pod \"e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s\" (UID: \"f08c940f-490f-41e2-be8b-873e4d082f78\") " pod="llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s"
Sep 10 21:14:33.144881 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.144859    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f08c940f-490f-41e2-be8b-873e4d082f78-model-cache\") pod \"e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s\" (UID: \"f08c940f-490f-41e2-be8b-873e4d082f78\") " pod="llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s"
Sep 10 21:14:33.145016 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.144896    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f08c940f-490f-41e2-be8b-873e4d082f78-tmp-dir\") pod \"e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s\" (UID: \"f08c940f-490f-41e2-be8b-873e4d082f78\") " pod="llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s"
Sep 10 21:14:33.145016 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.144973    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7fkd\" (UniqueName: \"kubernetes.io/projected/f08c940f-490f-41e2-be8b-873e4d082f78-kube-api-access-d7fkd\") pod \"e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s\" (UID: \"f08c940f-490f-41e2-be8b-873e4d082f78\") " pod="llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s"
Sep 10 21:14:33.245521 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.245497    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f08c940f-490f-41e2-be8b-873e4d082f78-tmp-dir\") pod \"e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s\" (UID: \"f08c940f-490f-41e2-be8b-873e4d082f78\") " pod="llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s"
Sep 10 21:14:33.245627 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.245534    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7fkd\" (UniqueName: \"kubernetes.io/projected/f08c940f-490f-41e2-be8b-873e4d082f78-kube-api-access-d7fkd\") pod \"e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s\" (UID: \"f08c940f-490f-41e2-be8b-873e4d082f78\") " pod="llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s"
Sep 10 21:14:33.245627 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.245564    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f08c940f-490f-41e2-be8b-873e4d082f78-tls-certs\") pod \"e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s\" (UID: \"f08c940f-490f-41e2-be8b-873e4d082f78\") " pod="llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s"
Sep 10 21:14:33.245627 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.245597    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f08c940f-490f-41e2-be8b-873e4d082f78-home\") pod \"e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s\" (UID: \"f08c940f-490f-41e2-be8b-873e4d082f78\") " pod="llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s"
Sep 10 21:14:33.245627 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.245613    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f08c940f-490f-41e2-be8b-873e4d082f78-dshm\") pod \"e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s\" (UID: \"f08c940f-490f-41e2-be8b-873e4d082f78\") " pod="llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s"
Sep 10 21:14:33.245851 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.245634    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f08c940f-490f-41e2-be8b-873e4d082f78-model-cache\") pod \"e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s\" (UID: \"f08c940f-490f-41e2-be8b-873e4d082f78\") " pod="llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s"
Sep 10 21:14:33.245913 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.245866    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f08c940f-490f-41e2-be8b-873e4d082f78-tmp-dir\") pod \"e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s\" (UID: \"f08c940f-490f-41e2-be8b-873e4d082f78\") " pod="llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s"
Sep 10 21:14:33.245987 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.245965    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f08c940f-490f-41e2-be8b-873e4d082f78-home\") pod \"e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s\" (UID: \"f08c940f-490f-41e2-be8b-873e4d082f78\") " pod="llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s"
Sep 10 21:14:33.246037 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.245997    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f08c940f-490f-41e2-be8b-873e4d082f78-model-cache\") pod \"e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s\" (UID: \"f08c940f-490f-41e2-be8b-873e4d082f78\") " pod="llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s"
Sep 10 21:14:33.248041 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.248019    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f08c940f-490f-41e2-be8b-873e4d082f78-dshm\") pod \"e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s\" (UID: \"f08c940f-490f-41e2-be8b-873e4d082f78\") " pod="llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s"
Sep 10 21:14:33.248130 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.248109    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f08c940f-490f-41e2-be8b-873e4d082f78-tls-certs\") pod \"e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s\" (UID: \"f08c940f-490f-41e2-be8b-873e4d082f78\") " pod="llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s"
Sep 10 21:14:33.255819 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.255794    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7fkd\" (UniqueName: \"kubernetes.io/projected/f08c940f-490f-41e2-be8b-873e4d082f78-kube-api-access-d7fkd\") pod \"e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s\" (UID: \"f08c940f-490f-41e2-be8b-873e4d082f78\") " pod="llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s"
Sep 10 21:14:33.412333 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.412265    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s"
Sep 10 21:14:33.685020 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.684996    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s"]
Sep 10 21:14:33.686327 ip-10-0-141-185 kubenswrapper[2564]: W0910 21:14:33.686301    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf08c940f_490f_41e2_be8b_873e4d082f78.slice/crio-25987bfdc9e911b218fc39f0d8fef9dd7567f0c098a3f8a48be01b1e18fde7c9 WatchSource:0}: Error finding container 25987bfdc9e911b218fc39f0d8fef9dd7567f0c098a3f8a48be01b1e18fde7c9: Status 404 returned error can't find the container with id 25987bfdc9e911b218fc39f0d8fef9dd7567f0c098a3f8a48be01b1e18fde7c9
Sep 10 21:14:33.731203 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:33.731155    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s" event={"ID":"f08c940f-490f-41e2-be8b-873e4d082f78","Type":"ContainerStarted","Data":"25987bfdc9e911b218fc39f0d8fef9dd7567f0c098a3f8a48be01b1e18fde7c9"}
Sep 10 21:14:34.737863 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:34.737819    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s" event={"ID":"f08c940f-490f-41e2-be8b-873e4d082f78","Type":"ContainerStarted","Data":"eaa8118f584e0a279e816d63042b73a57627aa2d834ac72e340bea05f1f80eb0"}
Sep 10 21:14:34.756271 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:34.756226    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s" podStartSLOduration=1.756212481 podStartE2EDuration="1.756212481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:14:34.755048531 +0000 UTC m=+2314.720116728" watchObservedRunningTime="2026-09-10 21:14:34.756212481 +0000 UTC m=+2314.721280675"
Sep 10 21:14:43.412753 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:43.412710    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s"
Sep 10 21:14:43.413245 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:43.412770    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s"
Sep 10 21:14:43.425567 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:43.425541    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s"
Sep 10 21:14:43.802886 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:43.802803    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s"
Sep 10 21:14:50.704688 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:50.704642    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk"]
Sep 10 21:14:50.708440 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:50.708411    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk"
Sep 10 21:14:50.711815 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:50.711788    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-4c60742f-kserve-self-signed-certs\""
Sep 10 21:14:50.717896 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:50.717866    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk"]
Sep 10 21:14:50.775455 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:50.775421    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/701683f0-955f-45f1-9f7e-feab82d17833-home\") pod \"e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk\" (UID: \"701683f0-955f-45f1-9f7e-feab82d17833\") " pod="llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk"
Sep 10 21:14:50.775575 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:50.775490    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/701683f0-955f-45f1-9f7e-feab82d17833-model-cache\") pod \"e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk\" (UID: \"701683f0-955f-45f1-9f7e-feab82d17833\") " pod="llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk"
Sep 10 21:14:50.775575 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:50.775525    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/701683f0-955f-45f1-9f7e-feab82d17833-tmp-dir\") pod \"e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk\" (UID: \"701683f0-955f-45f1-9f7e-feab82d17833\") " pod="llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk"
Sep 10 21:14:50.775704 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:50.775576    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/701683f0-955f-45f1-9f7e-feab82d17833-dshm\") pod \"e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk\" (UID: \"701683f0-955f-45f1-9f7e-feab82d17833\") " pod="llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk"
Sep 10 21:14:50.775704 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:50.775642    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/701683f0-955f-45f1-9f7e-feab82d17833-tls-certs\") pod \"e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk\" (UID: \"701683f0-955f-45f1-9f7e-feab82d17833\") " pod="llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk"
Sep 10 21:14:50.775704 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:50.775674    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94cs7\" (UniqueName: \"kubernetes.io/projected/701683f0-955f-45f1-9f7e-feab82d17833-kube-api-access-94cs7\") pod \"e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk\" (UID: \"701683f0-955f-45f1-9f7e-feab82d17833\") " pod="llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk"
Sep 10 21:14:50.876139 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:50.876110    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/701683f0-955f-45f1-9f7e-feab82d17833-tls-certs\") pod \"e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk\" (UID: \"701683f0-955f-45f1-9f7e-feab82d17833\") " pod="llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk"
Sep 10 21:14:50.876318 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:50.876148    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94cs7\" (UniqueName: \"kubernetes.io/projected/701683f0-955f-45f1-9f7e-feab82d17833-kube-api-access-94cs7\") pod \"e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk\" (UID: \"701683f0-955f-45f1-9f7e-feab82d17833\") " pod="llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk"
Sep 10 21:14:50.876405 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:50.876330    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/701683f0-955f-45f1-9f7e-feab82d17833-home\") pod \"e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk\" (UID: \"701683f0-955f-45f1-9f7e-feab82d17833\") " pod="llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk"
Sep 10 21:14:50.876405 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:50.876399    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/701683f0-955f-45f1-9f7e-feab82d17833-model-cache\") pod \"e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk\" (UID: \"701683f0-955f-45f1-9f7e-feab82d17833\") " pod="llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk"
Sep 10 21:14:50.876666 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:50.876418    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/701683f0-955f-45f1-9f7e-feab82d17833-tmp-dir\") pod \"e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk\" (UID: \"701683f0-955f-45f1-9f7e-feab82d17833\") " pod="llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk"
Sep 10 21:14:50.876666 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:50.876455    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/701683f0-955f-45f1-9f7e-feab82d17833-dshm\") pod \"e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk\" (UID: \"701683f0-955f-45f1-9f7e-feab82d17833\") " pod="llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk"
Sep 10 21:14:50.876875 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:50.876848    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/701683f0-955f-45f1-9f7e-feab82d17833-model-cache\") pod \"e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk\" (UID: \"701683f0-955f-45f1-9f7e-feab82d17833\") " pod="llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk"
Sep 10 21:14:50.876962 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:50.876924    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/701683f0-955f-45f1-9f7e-feab82d17833-tmp-dir\") pod \"e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk\" (UID: \"701683f0-955f-45f1-9f7e-feab82d17833\") " pod="llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk"
Sep 10 21:14:50.877026 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:50.877012    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/701683f0-955f-45f1-9f7e-feab82d17833-home\") pod \"e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk\" (UID: \"701683f0-955f-45f1-9f7e-feab82d17833\") " pod="llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk"
Sep 10 21:14:50.878718 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:50.878695    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/701683f0-955f-45f1-9f7e-feab82d17833-dshm\") pod \"e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk\" (UID: \"701683f0-955f-45f1-9f7e-feab82d17833\") " pod="llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk"
Sep 10 21:14:50.878799 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:50.878704    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/701683f0-955f-45f1-9f7e-feab82d17833-tls-certs\") pod \"e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk\" (UID: \"701683f0-955f-45f1-9f7e-feab82d17833\") " pod="llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk"
Sep 10 21:14:50.884442 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:50.884419    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94cs7\" (UniqueName: \"kubernetes.io/projected/701683f0-955f-45f1-9f7e-feab82d17833-kube-api-access-94cs7\") pod \"e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk\" (UID: \"701683f0-955f-45f1-9f7e-feab82d17833\") " pod="llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk"
Sep 10 21:14:51.020258 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:51.020195    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk"
Sep 10 21:14:51.293876 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:51.293847    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk"]
Sep 10 21:14:51.295033 ip-10-0-141-185 kubenswrapper[2564]: W0910 21:14:51.295006    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod701683f0_955f_45f1_9f7e_feab82d17833.slice/crio-04f1d9b0092807e63a1ce45399e4eca8121db05a6ffc0e64e630a8dd85dd820f WatchSource:0}: Error finding container 04f1d9b0092807e63a1ce45399e4eca8121db05a6ffc0e64e630a8dd85dd820f: Status 404 returned error can't find the container with id 04f1d9b0092807e63a1ce45399e4eca8121db05a6ffc0e64e630a8dd85dd820f
Sep 10 21:14:51.844639 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:51.844594    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk" event={"ID":"701683f0-955f-45f1-9f7e-feab82d17833","Type":"ContainerStarted","Data":"a1d3e7e249a436f6328340482a0921c0ffb51a71047b9d67fdda6085208bcf99"}
Sep 10 21:14:51.845018 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:51.844646    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk" event={"ID":"701683f0-955f-45f1-9f7e-feab82d17833","Type":"ContainerStarted","Data":"04f1d9b0092807e63a1ce45399e4eca8121db05a6ffc0e64e630a8dd85dd820f"}
Sep 10 21:14:51.862205 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:14:51.862135    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk" podStartSLOduration=1.8621180000000002 podStartE2EDuration="1.862118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:14:51.860501454 +0000 UTC m=+2331.825569651" watchObservedRunningTime="2026-09-10 21:14:51.862118 +0000 UTC m=+2331.827186195"
Sep 10 21:15:00.987992 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:00.987903    2564 scope.go:117] "RemoveContainer" containerID="dcfd10dda0cb92478beef3f530a3b36d4e5204950e4bbefde4d1fde6c13d0c9c"
Sep 10 21:15:01.021313 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:01.021284    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk"
Sep 10 21:15:01.021443 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:01.021328    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk"
Sep 10 21:15:01.033573 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:01.033550    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk"
Sep 10 21:15:01.910791 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:01.910762    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk"
Sep 10 21:15:16.483303 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:16.483271    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s"]
Sep 10 21:15:18.591479 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:18.591446    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk"]
Sep 10 21:15:31.501990 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:31.501934    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s" podUID="f08c940f-490f-41e2-be8b-873e4d082f78" containerName="main" containerID="cri-o://eaa8118f584e0a279e816d63042b73a57627aa2d834ac72e340bea05f1f80eb0" gracePeriod=45
Sep 10 21:15:31.745595 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:31.745571    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s"
Sep 10 21:15:31.889154 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:31.889127    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f08c940f-490f-41e2-be8b-873e4d082f78-dshm\") pod \"f08c940f-490f-41e2-be8b-873e4d082f78\" (UID: \"f08c940f-490f-41e2-be8b-873e4d082f78\") "
Sep 10 21:15:31.889323 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:31.889206    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f08c940f-490f-41e2-be8b-873e4d082f78-model-cache\") pod \"f08c940f-490f-41e2-be8b-873e4d082f78\" (UID: \"f08c940f-490f-41e2-be8b-873e4d082f78\") "
Sep 10 21:15:31.889323 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:31.889251    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f08c940f-490f-41e2-be8b-873e4d082f78-home\") pod \"f08c940f-490f-41e2-be8b-873e4d082f78\" (UID: \"f08c940f-490f-41e2-be8b-873e4d082f78\") "
Sep 10 21:15:31.889323 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:31.889269    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f08c940f-490f-41e2-be8b-873e4d082f78-tmp-dir\") pod \"f08c940f-490f-41e2-be8b-873e4d082f78\" (UID: \"f08c940f-490f-41e2-be8b-873e4d082f78\") "
Sep 10 21:15:31.889323 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:31.889295    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f08c940f-490f-41e2-be8b-873e4d082f78-tls-certs\") pod \"f08c940f-490f-41e2-be8b-873e4d082f78\" (UID: \"f08c940f-490f-41e2-be8b-873e4d082f78\") "
Sep 10 21:15:31.889323 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:31.889314    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7fkd\" (UniqueName: \"kubernetes.io/projected/f08c940f-490f-41e2-be8b-873e4d082f78-kube-api-access-d7fkd\") pod \"f08c940f-490f-41e2-be8b-873e4d082f78\" (UID: \"f08c940f-490f-41e2-be8b-873e4d082f78\") "
Sep 10 21:15:31.889621 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:31.889515    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08c940f-490f-41e2-be8b-873e4d082f78-model-cache" (OuterVolumeSpecName: "model-cache") pod "f08c940f-490f-41e2-be8b-873e4d082f78" (UID: "f08c940f-490f-41e2-be8b-873e4d082f78"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:31.889621 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:31.889524    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08c940f-490f-41e2-be8b-873e4d082f78-home" (OuterVolumeSpecName: "home") pod "f08c940f-490f-41e2-be8b-873e4d082f78" (UID: "f08c940f-490f-41e2-be8b-873e4d082f78"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:31.889621 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:31.889545    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08c940f-490f-41e2-be8b-873e4d082f78-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f08c940f-490f-41e2-be8b-873e4d082f78" (UID: "f08c940f-490f-41e2-be8b-873e4d082f78"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:31.891143 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:31.891122    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08c940f-490f-41e2-be8b-873e4d082f78-dshm" (OuterVolumeSpecName: "dshm") pod "f08c940f-490f-41e2-be8b-873e4d082f78" (UID: "f08c940f-490f-41e2-be8b-873e4d082f78"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:31.891357 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:31.891334    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f08c940f-490f-41e2-be8b-873e4d082f78-kube-api-access-d7fkd" (OuterVolumeSpecName: "kube-api-access-d7fkd") pod "f08c940f-490f-41e2-be8b-873e4d082f78" (UID: "f08c940f-490f-41e2-be8b-873e4d082f78"). InnerVolumeSpecName "kube-api-access-d7fkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:15:31.891399 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:31.891372    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f08c940f-490f-41e2-be8b-873e4d082f78-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f08c940f-490f-41e2-be8b-873e4d082f78" (UID: "f08c940f-490f-41e2-be8b-873e4d082f78"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:15:31.989975 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:31.989949    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f08c940f-490f-41e2-be8b-873e4d082f78-tls-certs\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:15:31.989975 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:31.989972    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7fkd\" (UniqueName: \"kubernetes.io/projected/f08c940f-490f-41e2-be8b-873e4d082f78-kube-api-access-d7fkd\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:15:31.990104 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:31.989983    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f08c940f-490f-41e2-be8b-873e4d082f78-dshm\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:15:31.990104 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:31.989992    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f08c940f-490f-41e2-be8b-873e4d082f78-model-cache\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:15:31.990104 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:31.990000    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f08c940f-490f-41e2-be8b-873e4d082f78-home\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:15:31.990104 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:31.990007    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f08c940f-490f-41e2-be8b-873e4d082f78-tmp-dir\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:15:32.065954 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:32.065924    2564 generic.go:356] "Generic (PLEG): container finished" podID="f08c940f-490f-41e2-be8b-873e4d082f78" containerID="eaa8118f584e0a279e816d63042b73a57627aa2d834ac72e340bea05f1f80eb0" exitCode=0
Sep 10 21:15:32.066049 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:32.065975    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s"
Sep 10 21:15:32.066049 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:32.065989    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s" event={"ID":"f08c940f-490f-41e2-be8b-873e4d082f78","Type":"ContainerDied","Data":"eaa8118f584e0a279e816d63042b73a57627aa2d834ac72e340bea05f1f80eb0"}
Sep 10 21:15:32.066049 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:32.066028    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s" event={"ID":"f08c940f-490f-41e2-be8b-873e4d082f78","Type":"ContainerDied","Data":"25987bfdc9e911b218fc39f0d8fef9dd7567f0c098a3f8a48be01b1e18fde7c9"}
Sep 10 21:15:32.066049 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:32.066046    2564 scope.go:117] "RemoveContainer" containerID="eaa8118f584e0a279e816d63042b73a57627aa2d834ac72e340bea05f1f80eb0"
Sep 10 21:15:32.074601 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:32.074583    2564 scope.go:117] "RemoveContainer" containerID="eaa8118f584e0a279e816d63042b73a57627aa2d834ac72e340bea05f1f80eb0"
Sep 10 21:15:32.074842 ip-10-0-141-185 kubenswrapper[2564]: E0910 21:15:32.074822    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaa8118f584e0a279e816d63042b73a57627aa2d834ac72e340bea05f1f80eb0\": container with ID starting with eaa8118f584e0a279e816d63042b73a57627aa2d834ac72e340bea05f1f80eb0 not found: ID does not exist" containerID="eaa8118f584e0a279e816d63042b73a57627aa2d834ac72e340bea05f1f80eb0"
Sep 10 21:15:32.074901 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:32.074849    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaa8118f584e0a279e816d63042b73a57627aa2d834ac72e340bea05f1f80eb0"} err="failed to get container status \"eaa8118f584e0a279e816d63042b73a57627aa2d834ac72e340bea05f1f80eb0\": rpc error: code = NotFound desc = could not find container \"eaa8118f584e0a279e816d63042b73a57627aa2d834ac72e340bea05f1f80eb0\": container with ID starting with eaa8118f584e0a279e816d63042b73a57627aa2d834ac72e340bea05f1f80eb0 not found: ID does not exist"
Sep 10 21:15:32.087081 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:32.087061    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s"]
Sep 10 21:15:32.091140 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:32.091118    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-4c60742f-kserve-c86677fc9-b7z9s"]
Sep 10 21:15:32.541357 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:32.541323    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f08c940f-490f-41e2-be8b-873e4d082f78" path="/var/lib/kubelet/pods/f08c940f-490f-41e2-be8b-873e4d082f78/volumes"
Sep 10 21:15:33.609758 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:33.609713    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk" podUID="701683f0-955f-45f1-9f7e-feab82d17833" containerName="main" containerID="cri-o://a1d3e7e249a436f6328340482a0921c0ffb51a71047b9d67fdda6085208bcf99" gracePeriod=45
Sep 10 21:15:33.850630 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:33.850608    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk"
Sep 10 21:15:34.006193 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:34.006106    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/701683f0-955f-45f1-9f7e-feab82d17833-dshm\") pod \"701683f0-955f-45f1-9f7e-feab82d17833\" (UID: \"701683f0-955f-45f1-9f7e-feab82d17833\") "
Sep 10 21:15:34.006193 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:34.006150    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94cs7\" (UniqueName: \"kubernetes.io/projected/701683f0-955f-45f1-9f7e-feab82d17833-kube-api-access-94cs7\") pod \"701683f0-955f-45f1-9f7e-feab82d17833\" (UID: \"701683f0-955f-45f1-9f7e-feab82d17833\") "
Sep 10 21:15:34.006384 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:34.006219    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/701683f0-955f-45f1-9f7e-feab82d17833-tls-certs\") pod \"701683f0-955f-45f1-9f7e-feab82d17833\" (UID: \"701683f0-955f-45f1-9f7e-feab82d17833\") "
Sep 10 21:15:34.006384 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:34.006261    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/701683f0-955f-45f1-9f7e-feab82d17833-home\") pod \"701683f0-955f-45f1-9f7e-feab82d17833\" (UID: \"701683f0-955f-45f1-9f7e-feab82d17833\") "
Sep 10 21:15:34.006384 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:34.006313    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/701683f0-955f-45f1-9f7e-feab82d17833-model-cache\") pod \"701683f0-955f-45f1-9f7e-feab82d17833\" (UID: \"701683f0-955f-45f1-9f7e-feab82d17833\") "
Sep 10 21:15:34.006384 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:34.006359    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/701683f0-955f-45f1-9f7e-feab82d17833-tmp-dir\") pod \"701683f0-955f-45f1-9f7e-feab82d17833\" (UID: \"701683f0-955f-45f1-9f7e-feab82d17833\") "
Sep 10 21:15:34.006591 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:34.006547    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701683f0-955f-45f1-9f7e-feab82d17833-model-cache" (OuterVolumeSpecName: "model-cache") pod "701683f0-955f-45f1-9f7e-feab82d17833" (UID: "701683f0-955f-45f1-9f7e-feab82d17833"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:34.006648 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:34.006595    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701683f0-955f-45f1-9f7e-feab82d17833-home" (OuterVolumeSpecName: "home") pod "701683f0-955f-45f1-9f7e-feab82d17833" (UID: "701683f0-955f-45f1-9f7e-feab82d17833"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:34.006709 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:34.006647    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/701683f0-955f-45f1-9f7e-feab82d17833-model-cache\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:15:34.006709 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:34.006674    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701683f0-955f-45f1-9f7e-feab82d17833-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "701683f0-955f-45f1-9f7e-feab82d17833" (UID: "701683f0-955f-45f1-9f7e-feab82d17833"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:34.008099 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:34.008076    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701683f0-955f-45f1-9f7e-feab82d17833-dshm" (OuterVolumeSpecName: "dshm") pod "701683f0-955f-45f1-9f7e-feab82d17833" (UID: "701683f0-955f-45f1-9f7e-feab82d17833"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:34.008713 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:34.008694    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/701683f0-955f-45f1-9f7e-feab82d17833-kube-api-access-94cs7" (OuterVolumeSpecName: "kube-api-access-94cs7") pod "701683f0-955f-45f1-9f7e-feab82d17833" (UID: "701683f0-955f-45f1-9f7e-feab82d17833"). InnerVolumeSpecName "kube-api-access-94cs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:15:34.008713 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:34.008701    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/701683f0-955f-45f1-9f7e-feab82d17833-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "701683f0-955f-45f1-9f7e-feab82d17833" (UID: "701683f0-955f-45f1-9f7e-feab82d17833"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:15:34.078962 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:34.078937    2564 generic.go:356] "Generic (PLEG): container finished" podID="701683f0-955f-45f1-9f7e-feab82d17833" containerID="a1d3e7e249a436f6328340482a0921c0ffb51a71047b9d67fdda6085208bcf99" exitCode=0
Sep 10 21:15:34.079084 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:34.078974    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk"
Sep 10 21:15:34.079084 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:34.079021    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk" event={"ID":"701683f0-955f-45f1-9f7e-feab82d17833","Type":"ContainerDied","Data":"a1d3e7e249a436f6328340482a0921c0ffb51a71047b9d67fdda6085208bcf99"}
Sep 10 21:15:34.079084 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:34.079058    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk" event={"ID":"701683f0-955f-45f1-9f7e-feab82d17833","Type":"ContainerDied","Data":"04f1d9b0092807e63a1ce45399e4eca8121db05a6ffc0e64e630a8dd85dd820f"}
Sep 10 21:15:34.079084 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:34.079075    2564 scope.go:117] "RemoveContainer" containerID="a1d3e7e249a436f6328340482a0921c0ffb51a71047b9d67fdda6085208bcf99"
Sep 10 21:15:34.088337 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:34.088283    2564 scope.go:117] "RemoveContainer" containerID="a1d3e7e249a436f6328340482a0921c0ffb51a71047b9d67fdda6085208bcf99"
Sep 10 21:15:34.088591 ip-10-0-141-185 kubenswrapper[2564]: E0910 21:15:34.088573    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1d3e7e249a436f6328340482a0921c0ffb51a71047b9d67fdda6085208bcf99\": container with ID starting with a1d3e7e249a436f6328340482a0921c0ffb51a71047b9d67fdda6085208bcf99 not found: ID does not exist" containerID="a1d3e7e249a436f6328340482a0921c0ffb51a71047b9d67fdda6085208bcf99"
Sep 10 21:15:34.088643 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:34.088601    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1d3e7e249a436f6328340482a0921c0ffb51a71047b9d67fdda6085208bcf99"} err="failed to get container status \"a1d3e7e249a436f6328340482a0921c0ffb51a71047b9d67fdda6085208bcf99\": rpc error: code = NotFound desc = could not find container \"a1d3e7e249a436f6328340482a0921c0ffb51a71047b9d67fdda6085208bcf99\": container with ID starting with a1d3e7e249a436f6328340482a0921c0ffb51a71047b9d67fdda6085208bcf99 not found: ID does not exist"
Sep 10 21:15:34.100590 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:34.100570    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk"]
Sep 10 21:15:34.105863 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:34.105843    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-4c60742f-kserve-59965bff47-hj8bk"]
Sep 10 21:15:34.108031 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:34.108014    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94cs7\" (UniqueName: \"kubernetes.io/projected/701683f0-955f-45f1-9f7e-feab82d17833-kube-api-access-94cs7\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:15:34.108080 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:34.108035    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/701683f0-955f-45f1-9f7e-feab82d17833-tls-certs\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:15:34.108080 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:34.108052    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/701683f0-955f-45f1-9f7e-feab82d17833-home\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:15:34.108080 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:34.108060    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/701683f0-955f-45f1-9f7e-feab82d17833-tmp-dir\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:15:34.108080 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:34.108067    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/701683f0-955f-45f1-9f7e-feab82d17833-dshm\") on node \"ip-10-0-141-185.ec2.internal\" DevicePath \"\""
Sep 10 21:15:34.541989 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:15:34.541959    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="701683f0-955f-45f1-9f7e-feab82d17833" path="/var/lib/kubelet/pods/701683f0-955f-45f1-9f7e-feab82d17833/volumes"
Sep 10 21:16:01.013294 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:01.013260    2564 scope.go:117] "RemoveContainer" containerID="a02f99d01d3617236a8786d791eedb5e089a363dce6d9ec38516e62909afac99"
Sep 10 21:16:01.021640 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:01.021618    2564 scope.go:117] "RemoveContainer" containerID="56ee05822db047af167064f4b9194b515f9d7694816260f5e95806e0fadd943b"
Sep 10 21:16:01.029022 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:01.029002    2564 scope.go:117] "RemoveContainer" containerID="616c87c511604f3cdb8cbbe70ec8015795306452ab57a387860a52b54dde07fc"
Sep 10 21:16:11.788382 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:11.788348    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cbb6b7559-r9k87_eee5736a-14cc-46ad-9842-193b42c8190a/authorino/0.log"
Sep 10 21:16:17.620423 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:17.620386    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-f59c6_73bee6fb-d337-4d9f-96b5-8068b9468996/server/0.log"
Sep 10 21:16:17.750213 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:17.750181    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-8hktp_81b9c2e1-8b00-4fd2-91b4-a0d233ae7e05/manager/0.log"
Sep 10 21:16:17.877542 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:17.877466    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-59df7859b6-8lrgz_25cfc772-5c2c-4146-b2c4-952ae3f7c996/manager/0.log"
Sep 10 21:16:17.886442 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:17.886422    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-59df7859b6-8lrgz_25cfc772-5c2c-4146-b2c4-952ae3f7c996/copy-manifests/0.log"
Sep 10 21:16:18.680411 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:18.680378    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817900-cg7gk_49a8d782-4828-4beb-8614-93cb5b754b4d/cleanup/0.log"
Sep 10 21:16:19.700627 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:19.700591    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cbb6b7559-r9k87_eee5736a-14cc-46ad-9842-193b42c8190a/authorino/0.log"
Sep 10 21:16:20.735209 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:20.735155    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9_f502ab18-8459-4acf-a582-57b7ce34b53b/istio-proxy/0.log"
Sep 10 21:16:20.963041 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:20.963001    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-556d9c8688-7qzx6_a0be5df7-c7e6-4558-9ae1-79e4980571cb/kube-auth-proxy/0.log"
Sep 10 21:16:21.321497 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:21.321468    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-mcl7w_c4b72952-37d2-4c1e-8e16-36b020cf4553/payload-pre-processing/0.log"
Sep 10 21:16:21.903471 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:21.903423    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-jdwzn_7da88b3d-5776-4896-aa4f-e025abc7940e/main/0.log"
Sep 10 21:16:22.489025 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:22.488995    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-dgl5d_1da53b5b-3b85-460e-8846-1e90d264e21b/main/0.log"
Sep 10 21:16:22.605521 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:22.605486    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-dzczl_7820f223-3f55-48b0-8750-6e9a8440f32b/main/0.log"
Sep 10 21:16:30.722213 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:30.722115    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-v7c4f_e14c5919-af21-4e8f-9783-a2ee2b4885c0/global-pull-secret-syncer/0.log"
Sep 10 21:16:30.886824 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:30.886796    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vv9cn_821d4eb6-5a24-40b4-bee3-472994b9be36/konnectivity-agent/0.log"
Sep 10 21:16:30.932880 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:30.932853    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-185.ec2.internal_b75d22eafcad6f42168b54bcbeb68b91/haproxy/0.log"
Sep 10 21:16:35.730700 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:35.730666    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cbb6b7559-r9k87_eee5736a-14cc-46ad-9842-193b42c8190a/authorino/0.log"
Sep 10 21:16:37.942635 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:37.942594    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jltvz_bfaa24d2-46c0-4a59-972b-97f3b965a05d/node-exporter/0.log"
Sep 10 21:16:37.973225 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:37.973194    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jltvz_bfaa24d2-46c0-4a59-972b-97f3b965a05d/kube-rbac-proxy/0.log"
Sep 10 21:16:37.997972 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:37.997948    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jltvz_bfaa24d2-46c0-4a59-972b-97f3b965a05d/init-textfile/0.log"
Sep 10 21:16:40.640694 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:40.640663    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zw7zh/perf-node-gather-daemonset-67b76"]
Sep 10 21:16:40.643077 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:40.640951    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="701683f0-955f-45f1-9f7e-feab82d17833" containerName="main"
Sep 10 21:16:40.643077 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:40.640962    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="701683f0-955f-45f1-9f7e-feab82d17833" containerName="main"
Sep 10 21:16:40.643077 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:40.640978    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f08c940f-490f-41e2-be8b-873e4d082f78" containerName="main"
Sep 10 21:16:40.643077 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:40.640985    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f08c940f-490f-41e2-be8b-873e4d082f78" containerName="main"
Sep 10 21:16:40.643077 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:40.641044    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f08c940f-490f-41e2-be8b-873e4d082f78" containerName="main"
Sep 10 21:16:40.643077 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:40.641050    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="701683f0-955f-45f1-9f7e-feab82d17833" containerName="main"
Sep 10 21:16:40.643940 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:40.643924    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-67b76"
Sep 10 21:16:40.646211 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:40.646188    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zw7zh\"/\"default-dockercfg-sck8c\""
Sep 10 21:16:40.646337 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:40.646211    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zw7zh\"/\"kube-root-ca.crt\""
Sep 10 21:16:40.646873 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:40.646857    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zw7zh\"/\"openshift-service-ca.crt\""
Sep 10 21:16:40.651989 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:40.651961    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zw7zh/perf-node-gather-daemonset-67b76"]
Sep 10 21:16:40.783643 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:40.783618    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a648f4ee-eb21-4cbd-a9bf-0e2fa7717fda-podres\") pod \"perf-node-gather-daemonset-67b76\" (UID: \"a648f4ee-eb21-4cbd-a9bf-0e2fa7717fda\") " pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-67b76"
Sep 10 21:16:40.783778 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:40.783655    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a648f4ee-eb21-4cbd-a9bf-0e2fa7717fda-lib-modules\") pod \"perf-node-gather-daemonset-67b76\" (UID: \"a648f4ee-eb21-4cbd-a9bf-0e2fa7717fda\") " pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-67b76"
Sep 10 21:16:40.783778 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:40.783675    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a648f4ee-eb21-4cbd-a9bf-0e2fa7717fda-sys\") pod \"perf-node-gather-daemonset-67b76\" (UID: \"a648f4ee-eb21-4cbd-a9bf-0e2fa7717fda\") " pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-67b76"
Sep 10 21:16:40.783778 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:40.783724    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a648f4ee-eb21-4cbd-a9bf-0e2fa7717fda-proc\") pod \"perf-node-gather-daemonset-67b76\" (UID: \"a648f4ee-eb21-4cbd-a9bf-0e2fa7717fda\") " pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-67b76"
Sep 10 21:16:40.783778 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:40.783767    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfp7p\" (UniqueName: \"kubernetes.io/projected/a648f4ee-eb21-4cbd-a9bf-0e2fa7717fda-kube-api-access-vfp7p\") pod \"perf-node-gather-daemonset-67b76\" (UID: \"a648f4ee-eb21-4cbd-a9bf-0e2fa7717fda\") " pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-67b76"
Sep 10 21:16:40.884861 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:40.884838    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfp7p\" (UniqueName: \"kubernetes.io/projected/a648f4ee-eb21-4cbd-a9bf-0e2fa7717fda-kube-api-access-vfp7p\") pod \"perf-node-gather-daemonset-67b76\" (UID: \"a648f4ee-eb21-4cbd-a9bf-0e2fa7717fda\") " pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-67b76"
Sep 10 21:16:40.884988 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:40.884889    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a648f4ee-eb21-4cbd-a9bf-0e2fa7717fda-podres\") pod \"perf-node-gather-daemonset-67b76\" (UID: \"a648f4ee-eb21-4cbd-a9bf-0e2fa7717fda\") " pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-67b76"
Sep 10 21:16:40.885062 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:40.884993    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a648f4ee-eb21-4cbd-a9bf-0e2fa7717fda-podres\") pod \"perf-node-gather-daemonset-67b76\" (UID: \"a648f4ee-eb21-4cbd-a9bf-0e2fa7717fda\") " pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-67b76"
Sep 10 21:16:40.885062 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:40.885002    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a648f4ee-eb21-4cbd-a9bf-0e2fa7717fda-lib-modules\") pod \"perf-node-gather-daemonset-67b76\" (UID: \"a648f4ee-eb21-4cbd-a9bf-0e2fa7717fda\") " pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-67b76"
Sep 10 21:16:40.885062 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:40.885032    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a648f4ee-eb21-4cbd-a9bf-0e2fa7717fda-sys\") pod \"perf-node-gather-daemonset-67b76\" (UID: \"a648f4ee-eb21-4cbd-a9bf-0e2fa7717fda\") " pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-67b76"
Sep 10 21:16:40.885062 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:40.885051    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a648f4ee-eb21-4cbd-a9bf-0e2fa7717fda-proc\") pod \"perf-node-gather-daemonset-67b76\" (UID: \"a648f4ee-eb21-4cbd-a9bf-0e2fa7717fda\") " pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-67b76"
Sep 10 21:16:40.885279 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:40.885067    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a648f4ee-eb21-4cbd-a9bf-0e2fa7717fda-lib-modules\") pod \"perf-node-gather-daemonset-67b76\" (UID: \"a648f4ee-eb21-4cbd-a9bf-0e2fa7717fda\") " pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-67b76"
Sep 10 21:16:40.885279 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:40.885102    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a648f4ee-eb21-4cbd-a9bf-0e2fa7717fda-sys\") pod \"perf-node-gather-daemonset-67b76\" (UID: \"a648f4ee-eb21-4cbd-a9bf-0e2fa7717fda\") " pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-67b76"
Sep 10 21:16:40.885279 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:40.885117    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a648f4ee-eb21-4cbd-a9bf-0e2fa7717fda-proc\") pod \"perf-node-gather-daemonset-67b76\" (UID: \"a648f4ee-eb21-4cbd-a9bf-0e2fa7717fda\") " pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-67b76"
Sep 10 21:16:40.893232 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:40.893163    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfp7p\" (UniqueName: \"kubernetes.io/projected/a648f4ee-eb21-4cbd-a9bf-0e2fa7717fda-kube-api-access-vfp7p\") pod \"perf-node-gather-daemonset-67b76\" (UID: \"a648f4ee-eb21-4cbd-a9bf-0e2fa7717fda\") " pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-67b76"
Sep 10 21:16:40.954811 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:40.954765    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-67b76"
Sep 10 21:16:41.226554 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:41.226528    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zw7zh/perf-node-gather-daemonset-67b76"]
Sep 10 21:16:41.227936 ip-10-0-141-185 kubenswrapper[2564]: W0910 21:16:41.227905    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda648f4ee_eb21_4cbd_a9bf_0e2fa7717fda.slice/crio-c2d1815dc54d89e9312511271ff034ead4625476bda2bda57c16520cc945ff5a WatchSource:0}: Error finding container c2d1815dc54d89e9312511271ff034ead4625476bda2bda57c16520cc945ff5a: Status 404 returned error can't find the container with id c2d1815dc54d89e9312511271ff034ead4625476bda2bda57c16520cc945ff5a
Sep 10 21:16:41.440405 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:41.440378    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-67b76" event={"ID":"a648f4ee-eb21-4cbd-a9bf-0e2fa7717fda","Type":"ContainerStarted","Data":"9cd0268f34e90e5d489fcf2ebfedc28e1c3fdf637208d17cf27f5604714a188a"}
Sep 10 21:16:41.440525 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:41.440424    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-67b76" event={"ID":"a648f4ee-eb21-4cbd-a9bf-0e2fa7717fda","Type":"ContainerStarted","Data":"c2d1815dc54d89e9312511271ff034ead4625476bda2bda57c16520cc945ff5a"}
Sep 10 21:16:41.440525 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:41.440456    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-67b76"
Sep 10 21:16:41.460249 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:41.460204    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-67b76" podStartSLOduration=1.460190369 podStartE2EDuration="1.460190369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 21:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:16:41.45766437 +0000 UTC m=+2441.422732595" watchObservedRunningTime="2026-09-10 21:16:41.460190369 +0000 UTC m=+2441.425258560"
Sep 10 21:16:43.063745 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:43.063706    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-f6hc2_7d345a7a-fae7-4330-baf7-8be851843ba9/dns/0.log"
Sep 10 21:16:43.086534 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:43.086505    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-f6hc2_7d345a7a-fae7-4330-baf7-8be851843ba9/kube-rbac-proxy/0.log"
Sep 10 21:16:43.155665 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:43.155639    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kqb7x_af88567d-1c33-4670-acc8-484e2d8b11d4/dns-node-resolver/0.log"
Sep 10 21:16:43.745364 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:43.745333    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fxt2l_ad5d5af7-f551-415c-9a17-fb89dd6bbb0a/node-ca/0.log"
Sep 10 21:16:44.758117 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:44.758073    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9_f502ab18-8459-4acf-a582-57b7ce34b53b/istio-proxy/0.log"
Sep 10 21:16:45.007575 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:45.007542    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-556d9c8688-7qzx6_a0be5df7-c7e6-4558-9ae1-79e4980571cb/kube-auth-proxy/0.log"
Sep 10 21:16:45.129937 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:45.129908    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-mcl7w_c4b72952-37d2-4c1e-8e16-36b020cf4553/payload-pre-processing/0.log"
Sep 10 21:16:46.143649 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:46.143611    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x9fdk_88836648-21c4-45c8-9cd9-1e30c6fb4529/serve-healthcheck-canary/0.log"
Sep 10 21:16:46.819285 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:46.819255    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zvhvz_d024393a-e68c-4001-a13f-9a046bc2a924/kube-rbac-proxy/0.log"
Sep 10 21:16:46.840097 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:46.840050    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zvhvz_d024393a-e68c-4001-a13f-9a046bc2a924/exporter/0.log"
Sep 10 21:16:46.862475 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:46.862448    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zvhvz_d024393a-e68c-4001-a13f-9a046bc2a924/extractor/0.log"
Sep 10 21:16:47.457044 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:47.457017    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zw7zh/perf-node-gather-daemonset-67b76"
Sep 10 21:16:49.364069 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:49.364041    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-f59c6_73bee6fb-d337-4d9f-96b5-8068b9468996/server/0.log"
Sep 10 21:16:49.488833 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:49.488809    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-8hktp_81b9c2e1-8b00-4fd2-91b4-a0d233ae7e05/manager/0.log"
Sep 10 21:16:49.621833 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:49.621799    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-59df7859b6-8lrgz_25cfc772-5c2c-4146-b2c4-952ae3f7c996/manager/0.log"
Sep 10 21:16:49.638193 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:49.638154    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-59df7859b6-8lrgz_25cfc772-5c2c-4146-b2c4-952ae3f7c996/copy-manifests/0.log"
Sep 10 21:16:51.069478 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:51.069451    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9_f502ab18-8459-4acf-a582-57b7ce34b53b/istio-proxy/0.log"
Sep 10 21:16:51.112079 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:51.112057    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-8d95fb99-5m4qs_5ab23fce-8336-47a9-b0d4-a651446ffea7/manager/0.log"
Sep 10 21:16:51.300774 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:51.300734    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-556d9c8688-7qzx6_a0be5df7-c7e6-4558-9ae1-79e4980571cb/kube-auth-proxy/0.log"
Sep 10 21:16:51.385445 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:51.385387    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-mcl7w_c4b72952-37d2-4c1e-8e16-36b020cf4553/payload-pre-processing/0.log"
Sep 10 21:16:52.897307 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:52.897259    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-qdnc9_f502ab18-8459-4acf-a582-57b7ce34b53b/istio-proxy/0.log"
Sep 10 21:16:53.124026 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:53.124002    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-556d9c8688-7qzx6_a0be5df7-c7e6-4558-9ae1-79e4980571cb/kube-auth-proxy/0.log"
Sep 10 21:16:53.208062 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:53.207996    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-mcl7w_c4b72952-37d2-4c1e-8e16-36b020cf4553/payload-pre-processing/0.log"
Sep 10 21:16:58.010786 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:58.010755    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hqsvv_b1b63c96-edab-42b0-8af3-f9db7d20867a/kube-multus-additional-cni-plugins/0.log"
Sep 10 21:16:58.042053 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:58.042022    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hqsvv_b1b63c96-edab-42b0-8af3-f9db7d20867a/egress-router-binary-copy/0.log"
Sep 10 21:16:58.065657 ip-10-0-141-185 kubenswrapper[2564]: I0910 21:16:58.065627    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hqsvv_b1b63c96-edab-42b0-8af3-f9db7d20867a/cni-plugins/0.log"