Sep 10 02:46:47.911289 ip-10-0-128-101 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 02:46:48.270169 ip-10-0-128-101 kubenswrapper[2558]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 02:46:48.270169 ip-10-0-128-101 kubenswrapper[2558]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 02:46:48.270169 ip-10-0-128-101 kubenswrapper[2558]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 02:46:48.270169 ip-10-0-128-101 kubenswrapper[2558]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 02:46:48.270169 ip-10-0-128-101 kubenswrapper[2558]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 02:46:48.272300 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.272233    2558 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 02:46:48.275952 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.275937    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 02:46:48.275952 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.275953    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 02:46:48.276021 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.275958    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 02:46:48.276021 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.275962    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 02:46:48.276021 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.275965    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 02:46:48.276021 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.275968    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 02:46:48.276021 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.275972    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 02:46:48.276021 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.275976    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 02:46:48.276021 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.275979    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 02:46:48.276021 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.275982    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 02:46:48.276021 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.275984    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 02:46:48.276021 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.275987    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 02:46:48.276021 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.275990    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 02:46:48.276021 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.275993    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 02:46:48.276021 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.275996    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 02:46:48.276021 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.275999    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 02:46:48.276021 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276001    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 02:46:48.276021 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276004    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 02:46:48.276021 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276006    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 02:46:48.276021 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276009    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 02:46:48.276021 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276011    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 02:46:48.276021 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276014    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 02:46:48.276520 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276017    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 02:46:48.276520 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276019    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 02:46:48.276520 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276022    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 02:46:48.276520 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276025    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 02:46:48.276520 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276039    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 02:46:48.276520 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276042    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 02:46:48.276520 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276045    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 02:46:48.276520 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276048    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 02:46:48.276520 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276051    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 02:46:48.276520 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276053    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 02:46:48.276520 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276056    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 02:46:48.276520 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276059    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 02:46:48.276520 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276062    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 02:46:48.276520 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276064    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 02:46:48.276520 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276067    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 02:46:48.276520 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276069    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 02:46:48.276520 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276072    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 02:46:48.276520 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276074    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 02:46:48.276520 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276077    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 02:46:48.276520 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276079    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 02:46:48.276993 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276082    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 02:46:48.276993 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276084    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 02:46:48.276993 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276087    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 02:46:48.276993 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276090    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 02:46:48.276993 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276092    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 02:46:48.276993 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276096    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 02:46:48.276993 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276098    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 02:46:48.276993 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276102    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 02:46:48.276993 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276105    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 02:46:48.276993 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276107    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 02:46:48.276993 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276110    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 02:46:48.276993 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276112    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 02:46:48.276993 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276115    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 02:46:48.276993 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276118    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 02:46:48.276993 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276121    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 02:46:48.276993 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276125    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 02:46:48.276993 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276128    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 02:46:48.276993 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276130    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 02:46:48.276993 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276133    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 02:46:48.277464 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276136    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 02:46:48.277464 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276138    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 02:46:48.277464 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276141    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 02:46:48.277464 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276143    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 02:46:48.277464 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276146    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 02:46:48.277464 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276149    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 02:46:48.277464 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276151    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 02:46:48.277464 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276153    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 02:46:48.277464 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276156    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 02:46:48.277464 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276158    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 02:46:48.277464 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276161    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 02:46:48.277464 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276165    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 02:46:48.277464 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276168    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 02:46:48.277464 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276171    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 02:46:48.277464 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276175    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 02:46:48.277464 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276177    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 02:46:48.277464 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276181    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 02:46:48.277464 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276185    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 02:46:48.277464 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276187    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 02:46:48.277951 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276190    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 02:46:48.277951 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276193    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 02:46:48.277951 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276197    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 02:46:48.277951 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276199    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 02:46:48.277951 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276202    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 02:46:48.277951 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276205    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 02:46:48.277951 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276207    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 02:46:48.277951 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276211    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 02:46:48.277951 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276213    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 02:46:48.277951 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276216    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 02:46:48.277951 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276218    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 02:46:48.277951 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276221    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 02:46:48.277951 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276224    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 02:46:48.277951 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276226    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 02:46:48.277951 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276229    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 02:46:48.277951 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276231    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 02:46:48.277951 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276234    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 02:46:48.277951 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276237    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 02:46:48.277951 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276239    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 02:46:48.277951 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276912    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 02:46:48.278463 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276918    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 02:46:48.278463 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276921    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 02:46:48.278463 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276925    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 02:46:48.278463 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276927    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 02:46:48.278463 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276930    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 02:46:48.278463 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276933    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 02:46:48.278463 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276936    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 02:46:48.278463 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276939    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 02:46:48.278463 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276942    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 02:46:48.278463 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276945    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 02:46:48.278463 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276949    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 02:46:48.278463 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276953    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 02:46:48.278463 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276957    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 02:46:48.278463 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276959    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 02:46:48.278463 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276962    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 02:46:48.278463 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276965    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 02:46:48.278463 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276967    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 02:46:48.278463 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276970    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 02:46:48.278463 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276973    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 02:46:48.278463 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276975    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 02:46:48.278958 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276978    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 02:46:48.278958 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276981    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 02:46:48.278958 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276983    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 02:46:48.278958 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276986    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 02:46:48.278958 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276988    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 02:46:48.278958 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276991    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 02:46:48.278958 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276993    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 02:46:48.278958 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276996    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 02:46:48.278958 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.276999    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 02:46:48.278958 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277001    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 02:46:48.278958 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277004    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 02:46:48.278958 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277007    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 02:46:48.278958 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277009    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 02:46:48.278958 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277011    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 02:46:48.278958 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277014    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 02:46:48.278958 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277017    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 02:46:48.278958 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277019    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 02:46:48.278958 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277022    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 02:46:48.278958 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277024    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 02:46:48.278958 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277043    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 02:46:48.279468 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277046    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 02:46:48.279468 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277049    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 02:46:48.279468 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277052    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 02:46:48.279468 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277055    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 02:46:48.279468 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277058    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 02:46:48.279468 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277060    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 02:46:48.279468 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277063    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 02:46:48.279468 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277066    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 02:46:48.279468 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277068    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 02:46:48.279468 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277071    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 02:46:48.279468 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277073    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 02:46:48.279468 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277076    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 02:46:48.279468 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277079    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 02:46:48.279468 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277081    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 02:46:48.279468 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277083    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 02:46:48.279468 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277086    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 02:46:48.279468 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277088    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 02:46:48.279468 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277091    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 02:46:48.279468 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277094    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 02:46:48.279468 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277096    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 02:46:48.280068 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277099    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 02:46:48.280068 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277101    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 02:46:48.280068 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277103    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 02:46:48.280068 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277106    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 02:46:48.280068 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277109    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 02:46:48.280068 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277111    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 02:46:48.280068 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277113    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 02:46:48.280068 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277116    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 02:46:48.280068 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277119    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 02:46:48.280068 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277121    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 02:46:48.280068 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277125    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 02:46:48.280068 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277127    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 02:46:48.280068 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277130    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 02:46:48.280068 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277133    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 02:46:48.280068 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277137    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 02:46:48.280068 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277139    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 02:46:48.280068 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277141    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 02:46:48.280068 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277144    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 02:46:48.280068 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277146    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 02:46:48.280068 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277149    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 02:46:48.280625 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277152    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 02:46:48.280625 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277154    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 02:46:48.280625 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277157    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 02:46:48.280625 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277159    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 02:46:48.280625 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277162    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 02:46:48.280625 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277166    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 02:46:48.280625 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277169    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 02:46:48.280625 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277172    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 02:46:48.280625 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277175    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 02:46:48.280625 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277179    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 02:46:48.280625 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277182    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 02:46:48.280625 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277185    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 02:46:48.280625 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277188    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 02:46:48.280625 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277191    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 02:46:48.280625 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277195    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 02:46:48.280625 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277199    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 02:46:48.280625 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277202    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 02:46:48.280625 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277204    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 02:46:48.280625 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277574    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 02:46:48.281102 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277579    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 02:46:48.281102 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277582    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 02:46:48.281102 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277585    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 02:46:48.281102 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277590    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 02:46:48.281102 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277593    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 02:46:48.281102 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277597    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 02:46:48.281102 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277600    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 02:46:48.281102 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277604    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 02:46:48.281102 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277606    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 02:46:48.281102 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277609    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 02:46:48.281102 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277612    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 02:46:48.281102 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277615    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 02:46:48.281102 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277617    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 02:46:48.281102 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277620    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 02:46:48.281102 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277622    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 02:46:48.281102 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277625    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 02:46:48.281102 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277629    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 02:46:48.281102 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277632    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 02:46:48.281102 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277635    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 02:46:48.281102 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277638    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 02:46:48.281582 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277642    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 02:46:48.281582 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277645    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 02:46:48.281582 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277648    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 02:46:48.281582 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277650    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 02:46:48.281582 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277653    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 02:46:48.281582 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277656    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 02:46:48.281582 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277658    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 02:46:48.281582 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277660    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 02:46:48.281582 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277663    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 02:46:48.281582 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277665    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 02:46:48.281582 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277668    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 02:46:48.281582 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277670    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 02:46:48.281582 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277673    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 02:46:48.281582 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277676    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 02:46:48.281582 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277678    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 02:46:48.281582 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277682    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 02:46:48.281582 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277685    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 02:46:48.281582 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277689    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 02:46:48.281582 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277692    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 02:46:48.281582 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277696    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 02:46:48.282147 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277699    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 02:46:48.282147 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277702    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 02:46:48.282147 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277704    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 02:46:48.282147 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277707    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 02:46:48.282147 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277709    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 02:46:48.282147 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277712    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 02:46:48.282147 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277714    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 02:46:48.282147 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277717    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 02:46:48.282147 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277719    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 02:46:48.282147 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277722    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 02:46:48.282147 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277724    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 02:46:48.282147 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277727    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 02:46:48.282147 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277729    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 02:46:48.282147 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277732    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 02:46:48.282147 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277734    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 02:46:48.282147 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277737    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 02:46:48.282147 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277739    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 02:46:48.282147 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277742    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 02:46:48.282147 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277744    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 02:46:48.282147 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277747    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 02:46:48.282622 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277750    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 02:46:48.282622 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277753    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 02:46:48.282622 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277756    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 02:46:48.282622 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277759    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 02:46:48.282622 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277762    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 02:46:48.282622 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277764    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 02:46:48.282622 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277767    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 02:46:48.282622 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277771    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 02:46:48.282622 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277773    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 02:46:48.282622 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277777    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 02:46:48.282622 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277780    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 02:46:48.282622 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277783    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 02:46:48.282622 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277786    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 02:46:48.282622 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277789    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 02:46:48.282622 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277792    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 02:46:48.282622 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277794    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 02:46:48.282622 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277797    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 02:46:48.282622 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277799    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 02:46:48.282622 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277802    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 02:46:48.283144 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277804    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 02:46:48.283144 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277807    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 02:46:48.283144 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277810    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 02:46:48.283144 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277812    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 02:46:48.283144 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277814    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 02:46:48.283144 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277817    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 02:46:48.283144 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277819    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 02:46:48.283144 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277822    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 02:46:48.283144 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277824    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 02:46:48.283144 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277827    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 02:46:48.283144 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277829    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 02:46:48.283144 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277832    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 02:46:48.283144 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277834    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 02:46:48.283144 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277837    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 02:46:48.283144 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277839    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 02:46:48.283144 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277842    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 02:46:48.283144 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277845    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 02:46:48.283144 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277847    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 02:46:48.283144 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.277849    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 02:46:48.283144 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.277931    2558 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 02:46:48.283632 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.277984    2558 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 02:46:48.283632 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.277992    2558 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 02:46:48.283632 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.277997    2558 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 02:46:48.283632 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278001    2558 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 02:46:48.283632 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278005    2558 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 02:46:48.283632 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278009    2558 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 02:46:48.283632 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278013    2558 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 02:46:48.283632 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278016    2558 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 02:46:48.283632 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278019    2558 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 02:46:48.283632 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278023    2558 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 02:46:48.283632 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278037    2558 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 02:46:48.283632 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278041    2558 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 02:46:48.283632 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278044    2558 flags.go:64] FLAG: --cgroup-root=""
Sep 10 02:46:48.283632 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278047    2558 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 02:46:48.283632 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278050    2558 flags.go:64] FLAG: --client-ca-file=""
Sep 10 02:46:48.283632 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278053    2558 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 02:46:48.283632 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278056    2558 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 02:46:48.283632 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278060    2558 flags.go:64] FLAG: --cluster-domain=""
Sep 10 02:46:48.283632 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278062    2558 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 02:46:48.283632 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278065    2558 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 02:46:48.283632 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278068    2558 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 02:46:48.283632 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278072    2558 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 02:46:48.283632 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278075    2558 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 02:46:48.283632 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278078    2558 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 02:46:48.284236 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278081    2558 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 02:46:48.284236 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278085    2558 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 02:46:48.284236 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278088    2558 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 02:46:48.284236 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278094    2558 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 02:46:48.284236 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278097    2558 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 02:46:48.284236 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278100    2558 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 02:46:48.284236 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278103    2558 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 02:46:48.284236 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278107    2558 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 02:46:48.284236 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278111    2558 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 02:46:48.284236 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278114    2558 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 02:46:48.284236 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278119    2558 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 02:46:48.284236 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278122    2558 flags.go:64] FLAG: --enable-server="true"
Sep 10 02:46:48.284236 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278125    2558 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 02:46:48.284236 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278130    2558 flags.go:64] FLAG: --event-burst="100"
Sep 10 02:46:48.284236 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278133    2558 flags.go:64] FLAG: --event-qps="50"
Sep 10 02:46:48.284236 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278136    2558 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 02:46:48.284236 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278139    2558 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 02:46:48.284236 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278143    2558 flags.go:64] FLAG: --eviction-hard=""
Sep 10 02:46:48.284236 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278146    2558 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 02:46:48.284236 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278149    2558 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 02:46:48.284236 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278152    2558 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 02:46:48.284236 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278155    2558 flags.go:64] FLAG: --eviction-soft=""
Sep 10 02:46:48.284236 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278157    2558 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 02:46:48.284236 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278160    2558 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 02:46:48.284236 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278163    2558 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 02:46:48.284847 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278166    2558 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 02:46:48.284847 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278169    2558 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 02:46:48.284847 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278171    2558 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 02:46:48.284847 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278174    2558 flags.go:64] FLAG: --feature-gates=""
Sep 10 02:46:48.284847 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278178    2558 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 02:46:48.284847 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278181    2558 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 02:46:48.284847 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278184    2558 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 02:46:48.284847 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278187    2558 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 02:46:48.284847 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278190    2558 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 02:46:48.284847 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278193    2558 flags.go:64] FLAG: --help="false"
Sep 10 02:46:48.284847 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278196    2558 flags.go:64] FLAG: --hostname-override="ip-10-0-128-101.ec2.internal"
Sep 10 02:46:48.284847 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278199    2558 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 02:46:48.284847 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278202    2558 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 02:46:48.284847 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278205    2558 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 02:46:48.284847 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278208    2558 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 02:46:48.284847 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278212    2558 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 02:46:48.284847 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278215    2558 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 02:46:48.284847 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278219    2558 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 02:46:48.284847 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278222    2558 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 02:46:48.284847 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278224    2558 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 02:46:48.284847 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278227    2558 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 02:46:48.284847 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278230    2558 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 02:46:48.284847 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278233    2558 flags.go:64] FLAG: --kube-reserved=""
Sep 10 02:46:48.284847 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278236    2558 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 02:46:48.285455 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278239    2558 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 02:46:48.285455 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278242    2558 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 02:46:48.285455 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278245    2558 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 02:46:48.285455 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278248    2558 flags.go:64] FLAG: --lock-file=""
Sep 10 02:46:48.285455 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278250    2558 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 02:46:48.285455 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278253    2558 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 02:46:48.285455 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278256    2558 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 02:46:48.285455 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278262    2558 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 02:46:48.285455 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278265    2558 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 02:46:48.285455 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278267    2558 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 02:46:48.285455 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278270    2558 flags.go:64] FLAG: --logging-format="text"
Sep 10 02:46:48.285455 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278273    2558 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 02:46:48.285455 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278276    2558 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 02:46:48.285455 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278279    2558 flags.go:64] FLAG: --manifest-url=""
Sep 10 02:46:48.285455 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278282    2558 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 02:46:48.285455 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278286    2558 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 02:46:48.285455 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278289    2558 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 02:46:48.285455 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278292    2558 flags.go:64] FLAG: --max-pods="110"
Sep 10 02:46:48.285455 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278295    2558 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 02:46:48.285455 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278298    2558 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 02:46:48.285455 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278301    2558 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 02:46:48.285455 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278304    2558 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 02:46:48.285455 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278307    2558 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 02:46:48.285455 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278311    2558 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 02:46:48.285455 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278314    2558 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 02:46:48.286115 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278322    2558 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 02:46:48.286115 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278325    2558 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 02:46:48.286115 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278328    2558 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 02:46:48.286115 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278331    2558 flags.go:64] FLAG: --pod-cidr=""
Sep 10 02:46:48.286115 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278334    2558 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 10 02:46:48.286115 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278339    2558 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 02:46:48.286115 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278342    2558 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 02:46:48.286115 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278345    2558 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 02:46:48.286115 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278349    2558 flags.go:64] FLAG: --port="10250"
Sep 10 02:46:48.286115 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278352    2558 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 02:46:48.286115 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278355    2558 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d0a4df21de83d12f"
Sep 10 02:46:48.286115 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278358    2558 flags.go:64] FLAG: --qos-reserved=""
Sep 10 02:46:48.286115 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278361    2558 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 02:46:48.286115 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278364    2558 flags.go:64] FLAG: --register-node="true"
Sep 10 02:46:48.286115 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278367    2558 flags.go:64] FLAG: --register-with-taints=""
Sep 10 02:46:48.286115 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278370    2558 flags.go:64] FLAG: --registry-burst="10"
Sep 10 02:46:48.286115 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278373    2558 flags.go:64] FLAG: --registry-qps="5"
Sep 10 02:46:48.286115 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278376    2558 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 02:46:48.286115 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278379    2558 flags.go:64] FLAG: --reserved-memory=""
Sep 10 02:46:48.286115 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278382    2558 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 02:46:48.286115 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278385    2558 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 02:46:48.286115 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278388    2558 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 02:46:48.286115 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278391    2558 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 02:46:48.286115 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278394    2558 flags.go:64] FLAG: --runonce="false"
Sep 10 02:46:48.286115 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278396    2558 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 02:46:48.286731 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278400    2558 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 02:46:48.286731 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278402    2558 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 02:46:48.286731 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278405    2558 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 02:46:48.286731 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278408    2558 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 02:46:48.286731 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278411    2558 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 02:46:48.286731 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278415    2558 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 02:46:48.286731 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278418    2558 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 02:46:48.286731 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278421    2558 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 02:46:48.286731 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278425    2558 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 02:46:48.286731 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278428    2558 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 02:46:48.286731 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278430    2558 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 02:46:48.286731 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278433    2558 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 02:46:48.286731 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278436    2558 flags.go:64] FLAG: --system-cgroups=""
Sep 10 02:46:48.286731 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278439    2558 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 02:46:48.286731 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278444    2558 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 02:46:48.286731 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278447    2558 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 02:46:48.286731 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278451    2558 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 02:46:48.286731 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278455    2558 flags.go:64] FLAG: --tls-min-version=""
Sep 10 02:46:48.286731 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278457    2558 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 02:46:48.286731 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278460    2558 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 02:46:48.286731 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278463    2558 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 02:46:48.286731 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278466    2558 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 02:46:48.286731 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278469    2558 flags.go:64] FLAG: --v="2"
Sep 10 02:46:48.286731 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278473    2558 flags.go:64] FLAG: --version="false"
Sep 10 02:46:48.286731 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278477    2558 flags.go:64] FLAG: --vmodule=""
Sep 10 02:46:48.287350 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278481    2558 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 02:46:48.287350 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.278484    2558 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 02:46:48.287350 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278620    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 02:46:48.287350 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278624    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 02:46:48.287350 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278628    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 02:46:48.287350 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278630    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 02:46:48.287350 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278634    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 02:46:48.287350 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278636    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 02:46:48.287350 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278640    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 02:46:48.287350 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278643    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 02:46:48.287350 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278645    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 02:46:48.287350 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278649    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 02:46:48.287350 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278652    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 02:46:48.287350 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278655    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 02:46:48.287350 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278658    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 02:46:48.287350 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278661    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 02:46:48.287350 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278664    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 02:46:48.287350 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278667    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 02:46:48.287350 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278669    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 02:46:48.287350 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278672    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 02:46:48.287934 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278675    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 02:46:48.287934 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278677    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 02:46:48.287934 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278680    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 02:46:48.287934 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278682    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 02:46:48.287934 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278686    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 02:46:48.287934 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278688    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 02:46:48.287934 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278691    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 02:46:48.287934 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278693    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 02:46:48.287934 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278696    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 02:46:48.287934 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278698    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 02:46:48.287934 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278701    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 02:46:48.287934 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278703    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 02:46:48.287934 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278707    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 02:46:48.287934 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278711    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 02:46:48.287934 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278713    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 02:46:48.287934 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278716    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 02:46:48.287934 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278719    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 02:46:48.287934 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278722    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 02:46:48.287934 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278724    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 02:46:48.288423 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278727    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 02:46:48.288423 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278730    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 02:46:48.288423 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278732    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 02:46:48.288423 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278735    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 02:46:48.288423 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278737    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 02:46:48.288423 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278741    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 02:46:48.288423 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278744    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 02:46:48.288423 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278747    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 02:46:48.288423 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278749    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 02:46:48.288423 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278752    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 02:46:48.288423 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278755    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 02:46:48.288423 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278757    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 02:46:48.288423 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278760    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 02:46:48.288423 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278762    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 02:46:48.288423 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278765    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 02:46:48.288423 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278767    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 02:46:48.288423 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278770    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 02:46:48.288423 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278772    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 02:46:48.288423 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278775    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 02:46:48.288423 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278779    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 02:46:48.288909 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278781    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 02:46:48.288909 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278783    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 02:46:48.288909 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278786    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 02:46:48.288909 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278788    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 02:46:48.288909 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278791    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 02:46:48.288909 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278795    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 02:46:48.288909 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278798    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 02:46:48.288909 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278801    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 02:46:48.288909 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278804    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 02:46:48.288909 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278807    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 02:46:48.288909 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278809    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 02:46:48.288909 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278812    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 02:46:48.288909 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278815    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 02:46:48.288909 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278817    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 02:46:48.288909 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278820    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 02:46:48.288909 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278824    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 02:46:48.288909 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278828    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 02:46:48.288909 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278833    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 02:46:48.288909 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278836    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 02:46:48.289475 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278839    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 02:46:48.289475 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278842    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 02:46:48.289475 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278844    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 02:46:48.289475 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278847    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 02:46:48.289475 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278850    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 02:46:48.289475 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278852    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 02:46:48.289475 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278855    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 02:46:48.289475 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278857    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 02:46:48.289475 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278860    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 02:46:48.289475 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278862    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 02:46:48.289475 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278865    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 02:46:48.289475 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278867    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 02:46:48.289475 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278871    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 02:46:48.289475 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278873    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 02:46:48.289475 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278876    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 02:46:48.289475 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278878    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 02:46:48.289475 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278881    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 02:46:48.289475 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278883    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 02:46:48.289475 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278886    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 02:46:48.289475 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278888    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 02:46:48.289977 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278891    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 02:46:48.289977 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278893    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 02:46:48.289977 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.278896    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 02:46:48.289977 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.279476    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 02:46:48.289977 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.285871    2558 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 02:46:48.289977 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.285885    2558 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 02:46:48.289977 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.286646    2558 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 02:46:48.289977 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.286654    2558 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 02:46:48.289977 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286714    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 02:46:48.289977 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286719    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 02:46:48.289977 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286722    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 02:46:48.289977 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286725    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 02:46:48.289977 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286728    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 02:46:48.289977 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286731    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 02:46:48.289977 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286734    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 02:46:48.290392 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286737    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 02:46:48.290392 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286739    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 02:46:48.290392 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286742    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 02:46:48.290392 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286745    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 02:46:48.290392 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286748    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 02:46:48.290392 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286751    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 02:46:48.290392 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286754    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 02:46:48.290392 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286756    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 02:46:48.290392 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286759    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 02:46:48.290392 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286762    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 02:46:48.290392 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286765    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 02:46:48.290392 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286768    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 02:46:48.290392 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286770    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 02:46:48.290392 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286773    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 02:46:48.290392 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286776    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 02:46:48.290392 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286778    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 02:46:48.290392 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286781    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 02:46:48.290392 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286784    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 02:46:48.290392 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286786    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 02:46:48.290392 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286789    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 02:46:48.290866 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286791    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 02:46:48.290866 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286794    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 02:46:48.290866 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286798    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 02:46:48.290866 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286800    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 02:46:48.290866 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286803    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 02:46:48.290866 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286807    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 02:46:48.290866 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286811    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 02:46:48.290866 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286814    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 02:46:48.290866 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286817    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 02:46:48.290866 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286820    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 02:46:48.290866 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286822    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 02:46:48.290866 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286825    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 02:46:48.290866 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286827    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 02:46:48.290866 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286830    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 02:46:48.290866 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286832    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 02:46:48.290866 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286835    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 02:46:48.290866 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286837    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 02:46:48.290866 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286841    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 02:46:48.290866 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286845    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 02:46:48.291332 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286848    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 02:46:48.291332 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286851    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 02:46:48.291332 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286854    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 02:46:48.291332 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286858    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 02:46:48.291332 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286861    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 02:46:48.291332 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286864    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 02:46:48.291332 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286866    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 02:46:48.291332 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286869    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 02:46:48.291332 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286871    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 02:46:48.291332 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286874    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 02:46:48.291332 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286876    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 02:46:48.291332 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286879    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 02:46:48.291332 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286882    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 02:46:48.291332 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286884    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 02:46:48.291332 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286887    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 02:46:48.291332 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286890    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 02:46:48.291332 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286893    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 02:46:48.291332 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286895    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 02:46:48.291332 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286897    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 02:46:48.291332 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286900    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 02:46:48.291809 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286904    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 02:46:48.291809 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286907    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 02:46:48.291809 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286910    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 02:46:48.291809 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286913    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 02:46:48.291809 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286916    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 02:46:48.291809 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286918    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 02:46:48.291809 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286921    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 02:46:48.291809 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286923    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 02:46:48.291809 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286926    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 02:46:48.291809 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286928    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 02:46:48.291809 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286931    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 02:46:48.291809 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286933    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 02:46:48.291809 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286936    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 02:46:48.291809 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286938    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 02:46:48.291809 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286941    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 02:46:48.291809 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286943    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 02:46:48.291809 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286946    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 02:46:48.291809 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286949    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 02:46:48.291809 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286951    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 02:46:48.292287 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286954    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 02:46:48.292287 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286956    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 02:46:48.292287 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286959    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 02:46:48.292287 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286961    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 02:46:48.292287 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286964    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 02:46:48.292287 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286966    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 02:46:48.292287 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286969    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 02:46:48.292287 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286972    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 02:46:48.292287 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286975    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 02:46:48.292287 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286977    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 02:46:48.292287 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286980    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 02:46:48.292287 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286982    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 02:46:48.292287 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286985    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 02:46:48.292287 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.286987    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 02:46:48.292287 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.286992    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 02:46:48.292657 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287112    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 02:46:48.292657 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287117    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 02:46:48.292657 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287121    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 02:46:48.292657 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287124    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 02:46:48.292657 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287127    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 02:46:48.292657 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287130    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 02:46:48.292657 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287133    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 02:46:48.292657 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287136    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 02:46:48.292657 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287138    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 02:46:48.292657 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287141    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 02:46:48.292657 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287143    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 02:46:48.292657 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287146    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 02:46:48.292657 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287148    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 02:46:48.292657 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287151    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 02:46:48.292657 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287153    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 02:46:48.292657 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287156    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 02:46:48.292657 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287159    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 02:46:48.292657 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287161    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 02:46:48.292657 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287164    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 02:46:48.292657 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287167    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 02:46:48.293372 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287169    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 02:46:48.293372 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287172    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 02:46:48.293372 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287174    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 02:46:48.293372 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287177    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 02:46:48.293372 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287179    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 02:46:48.293372 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287182    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 02:46:48.293372 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287184    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 02:46:48.293372 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287187    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 02:46:48.293372 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287189    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 02:46:48.293372 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287192    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 02:46:48.293372 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287194    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 02:46:48.293372 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287197    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 02:46:48.293372 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287199    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 02:46:48.293372 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287201    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 02:46:48.293372 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287204    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 02:46:48.293372 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287206    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 02:46:48.293372 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287209    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 02:46:48.293372 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287211    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 02:46:48.293372 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287214    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 02:46:48.293372 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287216    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 02:46:48.293842 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287219    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 02:46:48.293842 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287221    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 02:46:48.293842 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287223    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 02:46:48.293842 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287226    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 02:46:48.293842 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287228    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 02:46:48.293842 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287231    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 02:46:48.293842 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287233    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 02:46:48.293842 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287236    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 02:46:48.293842 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287239    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 02:46:48.293842 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287242    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 02:46:48.293842 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287244    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 02:46:48.293842 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287247    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 02:46:48.293842 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287249    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 02:46:48.293842 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287252    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 02:46:48.293842 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287256    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 02:46:48.293842 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287259    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 02:46:48.293842 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287262    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 02:46:48.293842 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287264    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 02:46:48.293842 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287266    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 02:46:48.293842 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287269    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 02:46:48.294352 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287272    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 02:46:48.294352 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287274    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 02:46:48.294352 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287277    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 02:46:48.294352 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287279    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 02:46:48.294352 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287281    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 02:46:48.294352 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287284    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 02:46:48.294352 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287286    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 02:46:48.294352 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287289    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 02:46:48.294352 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287291    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 02:46:48.294352 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287294    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 02:46:48.294352 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287296    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 02:46:48.294352 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287300    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 02:46:48.294352 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287303    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 02:46:48.294352 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287306    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 02:46:48.294352 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287309    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 02:46:48.294352 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287312    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 02:46:48.294352 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287314    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 02:46:48.294352 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287317    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 02:46:48.294352 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287320    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 02:46:48.294818 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287323    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 02:46:48.294818 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287326    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 02:46:48.294818 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287329    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 02:46:48.294818 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287331    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 02:46:48.294818 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287334    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 02:46:48.294818 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287337    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 02:46:48.294818 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287339    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 02:46:48.294818 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287342    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 02:46:48.294818 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287345    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 02:46:48.294818 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287348    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 02:46:48.294818 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287351    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 02:46:48.294818 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287355    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 02:46:48.294818 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287357    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 02:46:48.294818 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287361    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 02:46:48.294818 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287364    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 02:46:48.294818 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287366    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 02:46:48.294818 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287369    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 02:46:48.294818 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287372    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 02:46:48.294818 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287374    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 02:46:48.295285 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:48.287376    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 02:46:48.295285 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.287381    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 02:46:48.295285 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.287476    2558 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 02:46:48.295285 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.289634    2558 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 02:46:48.295285 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.290444    2558 server.go:1019] "Starting client certificate rotation"
Sep 10 02:46:48.295285 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.290540    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 02:46:48.295285 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.290572    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 02:46:48.295484 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.295470    2558 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 02:46:48.299319 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.299303    2558 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 02:46:48.311418 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.311398    2558 log.go:25] "Validated CRI v1 runtime API"
Sep 10 02:46:48.315419 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.315398    2558 log.go:25] "Validated CRI v1 image API"
Sep 10 02:46:48.316300 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.316278    2558 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 02:46:48.318912 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.318890    2558 fs.go:135] Filesystem UUIDs: map[52f0c72a-88aa-4fba-be97-70c94b5bcb75:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c8d9ebd0-603f-4f49-9dd8-770946edf309:/dev/nvme0n1p3]
Sep 10 02:46:48.318966 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.318912    2558 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 02:46:48.324756 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.324619    2558 manager.go:216] Machine: {Timestamp:2026-09-10 02:46:48.322985195 +0000 UTC m=+0.307143923 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099880 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fbff8e45106eb0437683e8e0eb1c1 SystemUUID:ec2fbff8-e451-06eb-0437-683e8e0eb1c1 BootID:280fbde1-b3a8-42f8-9226-4a1276c53674 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ca:ed:d0:eb:c3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ca:ed:d0:eb:c3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:1c:29:6d:d1:ec Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 02:46:48.324756 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.324755    2558 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 02:46:48.324862 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.324851    2558 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 02:46:48.327650 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.327625    2558 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 02:46:48.327774 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.327653    2558 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-101.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 02:46:48.328710 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.328700    2558 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 02:46:48.328742 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.328713    2558 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 02:46:48.328742 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.328721    2558 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 02:46:48.329322 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.329303    2558 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 02:46:48.330969 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.330957    2558 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 02:46:48.334764 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.334748    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 02:46:48.334843 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.334830    2558 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 02:46:48.334991 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.334981    2558 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 02:46:48.335022 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.335012    2558 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 02:46:48.335062 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.335025    2558 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 02:46:48.335062 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.335054    2558 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 02:46:48.335122 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.335070    2558 kubelet.go:398] "Adding apiserver pod source"
Sep 10 02:46:48.335122 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.335092    2558 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 02:46:48.335610 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.335596    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 02:46:48.335653 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.335622    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 02:46:48.337680 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.337656    2558 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 02:46:48.340196 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.340182    2558 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 02:46:48.340246 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.340213    2558 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 02:46:48.340385 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.340376    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 02:46:48.340415 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.340392    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 02:46:48.340415 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.340398    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 02:46:48.340415 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.340404    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 02:46:48.340415 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.340413    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 02:46:48.340528 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.340421    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 02:46:48.341063 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.341054    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 02:46:48.341063 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.341064    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 02:46:48.341125 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.341070    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 02:46:48.341125 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.341082    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 02:46:48.341125 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.341098    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 02:46:48.341125 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.341105    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 02:46:48.341856 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.341847    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 02:46:48.341890 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.341858    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 02:46:48.345876 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.345861    2558 server.go:1268] "Started kubelet"
Sep 10 02:46:48.345983 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.345939    2558 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 02:46:48.346064 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.345971    2558 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 02:46:48.346064 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.346055    2558 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 02:46:48.346719 ip-10-0-128-101 systemd[1]: Started Kubernetes Kubelet.
Sep 10 02:46:48.347648 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.347628    2558 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 02:46:48.347923 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.347910    2558 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 02:46:48.349981 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:48.349962    2558 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 02:46:48.352588 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:48.352563    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-101.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 02:46:48.352588 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.352567    2558 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-101.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 02:46:48.352712 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:48.352599    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 02:46:48.352895 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.352884    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 02:46:48.353358 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:48.352564    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-101.ec2.internal.18d3d5ab37a3ed53  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-101.ec2.internal,UID:ip-10-0-128-101.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-101.ec2.internal,},FirstTimestamp:2026-09-10 02:46:48.345832787 +0000 UTC m=+0.329991515,LastTimestamp:2026-09-10 02:46:48.345832787 +0000 UTC m=+0.329991515,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-101.ec2.internal,}"
Sep 10 02:46:48.353436 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.353381    2558 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 02:46:48.353585 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:48.353560    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-101.ec2.internal\" not found"
Sep 10 02:46:48.354040 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.354000    2558 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 02:46:48.354040 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.354003    2558 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 02:46:48.354170 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.354053    2558 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 02:46:48.354170 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.354109    2558 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 02:46:48.354170 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.354119    2558 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 02:46:48.354322 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.354245    2558 factory.go:55] Registering systemd factory
Sep 10 02:46:48.354322 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.354261    2558 factory.go:223] Registration of the systemd container factory successfully
Sep 10 02:46:48.354436 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.354420    2558 factory.go:153] Registering CRI-O factory
Sep 10 02:46:48.354436 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.354437    2558 factory.go:223] Registration of the crio container factory successfully
Sep 10 02:46:48.354580 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.354503    2558 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 02:46:48.354580 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.354533    2558 factory.go:103] Registering Raw factory
Sep 10 02:46:48.354580 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.354547    2558 manager.go:1196] Started watching for new ooms in manager
Sep 10 02:46:48.354985 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.354972    2558 manager.go:349] Starting recovery of all containers
Sep 10 02:46:48.358074 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:48.358048    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 02:46:48.358173 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:48.358135    2558 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-101.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 02:46:48.367288 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.367274    2558 manager.go:354] Recovery completed
Sep 10 02:46:48.374621 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.374605    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 02:46:48.376283 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.376263    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 02:46:48.376349 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.376295    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 02:46:48.376349 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.376305    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasSufficientPID"
Sep 10 02:46:48.376925 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.376831    2558 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 02:46:48.376976 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.376929    2558 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 02:46:48.376976 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.376944    2558 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 02:46:48.377056 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.376853    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 02:46:48.377712 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.377696    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 02:46:48.377764 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.377726    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 02:46:48.377764 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.377739    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasSufficientPID"
Sep 10 02:46:48.378710 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:48.378652    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-101.ec2.internal.18d3d5ab39748eaa  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-101.ec2.internal,UID:ip-10-0-128-101.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-101.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-101.ec2.internal,},FirstTimestamp:2026-09-10 02:46:48.376282794 +0000 UTC m=+0.360441524,LastTimestamp:2026-09-10 02:46:48.376282794 +0000 UTC m=+0.360441524,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-101.ec2.internal,}"
Sep 10 02:46:48.379137 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.379126    2558 policy_none.go:49] "None policy: Start"
Sep 10 02:46:48.379173 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.379141    2558 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 02:46:48.379173 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.379150    2558 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 02:46:48.380858 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.380848    2558 policy_none.go:47] "Start"
Sep 10 02:46:48.393187 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:48.393103    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-101.ec2.internal.18d3d5ab3974d1cc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-101.ec2.internal,UID:ip-10-0-128-101.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-101.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-101.ec2.internal,},FirstTimestamp:2026-09-10 02:46:48.37629998 +0000 UTC m=+0.360458710,LastTimestamp:2026-09-10 02:46:48.37629998 +0000 UTC m=+0.360458710,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-101.ec2.internal,}"
Sep 10 02:46:48.403446 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:48.403378    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-101.ec2.internal.18d3d5ab3974f601  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-101.ec2.internal,UID:ip-10-0-128-101.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-101.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-101.ec2.internal,},FirstTimestamp:2026-09-10 02:46:48.376309249 +0000 UTC m=+0.360467979,LastTimestamp:2026-09-10 02:46:48.376309249 +0000 UTC m=+0.360467979,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-101.ec2.internal,}"
Sep 10 02:46:48.431188 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.405163    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wqpmx"
Sep 10 02:46:48.431188 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.413184    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wqpmx"
Sep 10 02:46:48.431188 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:48.415761    2558 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-101.ec2.internal.18d3d5ab39748eaa\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-101.ec2.internal.18d3d5ab39748eaa  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-101.ec2.internal,UID:ip-10-0-128-101.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-101.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-101.ec2.internal,},FirstTimestamp:2026-09-10 02:46:48.376282794 +0000 UTC m=+0.360441524,LastTimestamp:2026-09-10 02:46:48.377711599 +0000 UTC m=+0.361870330,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-101.ec2.internal,}"
Sep 10 02:46:48.431188 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.417652    2558 manager.go:337] "Starting Device Plugin manager"
Sep 10 02:46:48.431188 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:48.417681    2558 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 02:46:48.431188 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.417690    2558 server.go:87] "Starting device plugin registration server"
Sep 10 02:46:48.431188 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.417965    2558 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 02:46:48.431188 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.417978    2558 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 02:46:48.431188 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.418079    2558 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 02:46:48.431188 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.418188    2558 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 02:46:48.431188 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.418200    2558 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 02:46:48.431188 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:48.418764    2558 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 02:46:48.431188 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:48.418803    2558 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-101.ec2.internal\" not found"
Sep 10 02:46:48.451718 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.451696    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 02:46:48.452869 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.452852    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 02:46:48.452944 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.452891    2558 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 02:46:48.452944 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.452907    2558 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 02:46:48.453056 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:48.452977    2558 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 02:46:48.454717 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.454697    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 02:46:48.518690 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.518675    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 02:46:48.519457 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.519442    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 02:46:48.519588 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.519476    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 02:46:48.519588 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.519490    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasSufficientPID"
Sep 10 02:46:48.519588 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.519516    2558 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-101.ec2.internal"
Sep 10 02:46:48.538957 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.538910    2558 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-101.ec2.internal"
Sep 10 02:46:48.538957 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:48.538928    2558 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-101.ec2.internal\": node \"ip-10-0-128-101.ec2.internal\" not found"
Sep 10 02:46:48.553389 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.553372    2558 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-101.ec2.internal"]
Sep 10 02:46:48.553471 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.553443    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 02:46:48.554212 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.554190    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 02:46:48.554309 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.554216    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 02:46:48.554309 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.554215    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d3279872c23798b7320a2d8a5e9247b5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal\" (UID: \"d3279872c23798b7320a2d8a5e9247b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal"
Sep 10 02:46:48.554309 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.554226    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasSufficientPID"
Sep 10 02:46:48.554309 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.554266    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3279872c23798b7320a2d8a5e9247b5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal\" (UID: \"d3279872c23798b7320a2d8a5e9247b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal"
Sep 10 02:46:48.556275 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.556260    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 02:46:48.556380 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.556368    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal"
Sep 10 02:46:48.556425 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.556404    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 02:46:48.556958 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.556942    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 02:46:48.556958 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.556953    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 02:46:48.557083 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.556977    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 02:46:48.557083 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.556989    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 02:46:48.557083 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.557001    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasSufficientPID"
Sep 10 02:46:48.557209 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.556991    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasSufficientPID"
Sep 10 02:46:48.558999 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.558986    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-101.ec2.internal"
Sep 10 02:46:48.559073 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.559009    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 02:46:48.559710 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.559693    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 02:46:48.559779 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.559720    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 02:46:48.559779 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.559730    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeHasSufficientPID"
Sep 10 02:46:48.568161 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:48.568145    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-101.ec2.internal\" not found"
Sep 10 02:46:48.585615 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:48.585591    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-101.ec2.internal\" not found" node="ip-10-0-128-101.ec2.internal"
Sep 10 02:46:48.589886 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:48.589869    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-101.ec2.internal\" not found" node="ip-10-0-128-101.ec2.internal"
Sep 10 02:46:48.654337 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.654318    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d3279872c23798b7320a2d8a5e9247b5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal\" (UID: \"d3279872c23798b7320a2d8a5e9247b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal"
Sep 10 02:46:48.654399 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.654341    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3279872c23798b7320a2d8a5e9247b5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal\" (UID: \"d3279872c23798b7320a2d8a5e9247b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal"
Sep 10 02:46:48.654399 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.654386    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/01f46892139817c71935091c0eabf5f5-config\") pod \"kube-apiserver-proxy-ip-10-0-128-101.ec2.internal\" (UID: \"01f46892139817c71935091c0eabf5f5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-101.ec2.internal"
Sep 10 02:46:48.654475 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.654450    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3279872c23798b7320a2d8a5e9247b5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal\" (UID: \"d3279872c23798b7320a2d8a5e9247b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal"
Sep 10 02:46:48.654508 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.654488    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d3279872c23798b7320a2d8a5e9247b5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal\" (UID: \"d3279872c23798b7320a2d8a5e9247b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal"
Sep 10 02:46:48.669019 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:48.669003    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-101.ec2.internal\" not found"
Sep 10 02:46:48.755181 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.755159    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/01f46892139817c71935091c0eabf5f5-config\") pod \"kube-apiserver-proxy-ip-10-0-128-101.ec2.internal\" (UID: \"01f46892139817c71935091c0eabf5f5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-101.ec2.internal"
Sep 10 02:46:48.755245 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.755203    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/01f46892139817c71935091c0eabf5f5-config\") pod \"kube-apiserver-proxy-ip-10-0-128-101.ec2.internal\" (UID: \"01f46892139817c71935091c0eabf5f5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-101.ec2.internal"
Sep 10 02:46:48.769301 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:48.769279    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-101.ec2.internal\" not found"
Sep 10 02:46:48.870096 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:48.870046    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-101.ec2.internal\" not found"
Sep 10 02:46:48.889213 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.889192    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal"
Sep 10 02:46:48.893056 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:48.893026    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-101.ec2.internal"
Sep 10 02:46:48.970323 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:48.970284    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-101.ec2.internal\" not found"
Sep 10 02:46:49.070780 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:49.070750    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-101.ec2.internal\" not found"
Sep 10 02:46:49.171390 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:49.171345    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-101.ec2.internal\" not found"
Sep 10 02:46:49.180591 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:49.180572    2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 02:46:49.254431 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:49.254406    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal"
Sep 10 02:46:49.265376 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:49.265357    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 02:46:49.267400 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:49.267384    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-101.ec2.internal"
Sep 10 02:46:49.276495 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:49.276474    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 02:46:49.290655 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:49.290634    2558 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 02:46:49.290807 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:49.290784    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 02:46:49.290866 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:49.290825    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 02:46:50.944996 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.944978    2558 apiserver.go:52] "Watching apiserver"
Sep 10 02:46:50.960046 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.960023    2558 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 02:46:50.960430 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.960408    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-5qtg6","kube-system/kube-apiserver-proxy-ip-10-0-128-101.ec2.internal","openshift-image-registry/node-ca-4xpxz","openshift-multus/multus-pptgm","openshift-network-operator/iptables-alerter-9j9fc","openshift-ovn-kubernetes/ovnkube-node-bltxf","kube-system/konnectivity-agent-296rx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfz5c","openshift-cluster-node-tuning-operator/tuned-qhf58","openshift-dns/node-resolver-rl4nr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal","openshift-multus/multus-additional-cni-plugins-btm4f","openshift-multus/network-metrics-daemon-m5ph4","openshift-network-diagnostics/network-check-target-7xn49"]
Sep 10 02:46:50.962731 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.962710    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 02:46:50.962857 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.962836    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:46:50.962943 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:50.962917    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5qtg6" podUID="de3a2ed4-28df-4dea-8e14-c5dfe2981ba5"
Sep 10 02:46:50.965068 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.965048    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4xpxz"
Sep 10 02:46:50.968225 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.968186    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 02:46:50.968833 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.968806    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/de3a2ed4-28df-4dea-8e14-c5dfe2981ba5-kubelet-config\") pod \"global-pull-secret-syncer-5qtg6\" (UID: \"de3a2ed4-28df-4dea-8e14-c5dfe2981ba5\") " pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:46:50.968930 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.968849    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/de3a2ed4-28df-4dea-8e14-c5dfe2981ba5-dbus\") pod \"global-pull-secret-syncer-5qtg6\" (UID: \"de3a2ed4-28df-4dea-8e14-c5dfe2981ba5\") " pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:46:50.968930 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.968900    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de3a2ed4-28df-4dea-8e14-c5dfe2981ba5-original-pull-secret\") pod \"global-pull-secret-syncer-5qtg6\" (UID: \"de3a2ed4-28df-4dea-8e14-c5dfe2981ba5\") " pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:46:50.969095 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.968970    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c68ab7ea-a147-44d5-81a1-91eca0da2366-serviceca\") pod \"node-ca-4xpxz\" (UID: \"c68ab7ea-a147-44d5-81a1-91eca0da2366\") " pod="openshift-image-registry/node-ca-4xpxz"
Sep 10 02:46:50.969396 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.969079    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 02:46:50.969570 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.969545    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c68ab7ea-a147-44d5-81a1-91eca0da2366-host\") pod \"node-ca-4xpxz\" (UID: \"c68ab7ea-a147-44d5-81a1-91eca0da2366\") " pod="openshift-image-registry/node-ca-4xpxz"
Sep 10 02:46:50.969854 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.969298    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 02:46:50.969953 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.969852    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pptgm"
Sep 10 02:46:50.969953 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.969863    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9b9k\" (UniqueName: \"kubernetes.io/projected/c68ab7ea-a147-44d5-81a1-91eca0da2366-kube-api-access-t9b9k\") pod \"node-ca-4xpxz\" (UID: \"c68ab7ea-a147-44d5-81a1-91eca0da2366\") " pod="openshift-image-registry/node-ca-4xpxz"
Sep 10 02:46:50.970388 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.970365    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-t6556\""
Sep 10 02:46:50.970547 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.970524    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9j9fc"
Sep 10 02:46:50.972386 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.972365    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xpgld\""
Sep 10 02:46:50.972386 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.972370    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 02:46:50.972710 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.972695    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 02:46:50.972858 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.972836    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:50.973005 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.972964    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 02:46:50.973153 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.973066    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 02:46:50.973153 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.973064    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 02:46:50.973153 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.973146    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c4x58\""
Sep 10 02:46:50.973564 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.973496    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 02:46:50.973564 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.973517    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 02:46:50.973564 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.973534    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 02:46:50.974950 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.974930    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-296rx"
Sep 10 02:46:50.975058 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.974984    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 02:46:50.975267 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.975215    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 02:46:50.975449 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.975430    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 02:46:50.975522 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.975433    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 02:46:50.975735 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.975705    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 02:46:50.976033 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.976014    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 02:46:50.976205 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.976190    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rb9q9\""
Sep 10 02:46:50.977238 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.977217    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfz5c"
Sep 10 02:46:50.977358 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.977338    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 02:46:50.977422 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.977357    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 02:46:50.977477 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.977423    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7px2q\""
Sep 10 02:46:50.979281 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.979262    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 02:46:50.979455 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.979434    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 02:46:50.979537 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.979459    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 02:46:50.979537 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.979528    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p4djz\""
Sep 10 02:46:50.980946 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.980928    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:50.982922 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.982835    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 02:46:50.982922 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.982854    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ltpr5\""
Sep 10 02:46:50.983079 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.982938    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 02:46:50.983501 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.983483    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rl4nr"
Sep 10 02:46:50.985459 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.985292    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 02:46:50.985459 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.985356    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-x4xgp\""
Sep 10 02:46:50.985459 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.985387    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 02:46:50.985861 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.985843    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-btm4f"
Sep 10 02:46:50.987475 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.987457    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 02:46:50.990529 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.990461    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 02:46:50.993267 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.993239    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5ph4"
Sep 10 02:46:50.993360 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.993282    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-26p67\""
Sep 10 02:46:50.993360 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:50.993333    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5ph4" podUID="aa599110-d0eb-4cc2-b4e9-e3984e9717fe"
Sep 10 02:46:50.995897 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:50.995825    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xn49"
Sep 10 02:46:50.996503 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:50.995899    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xn49" podUID="e7d69b54-6b09-4496-abe6-6da05c77e958"
Sep 10 02:46:51.003256 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.003238    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-x97jx"
Sep 10 02:46:51.011059 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.011037    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-x97jx"
Sep 10 02:46:51.026569 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.026531    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 02:41:50 +0000 UTC" deadline="2028-05-08 10:39:14.917844342 +0000 UTC"
Sep 10 02:46:51.026569 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.026568    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14551h52m23.891279293s"
Sep 10 02:46:51.063858 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.063842    2558 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 02:46:51.070171 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070146    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/62603892-88ce-474d-8d32-3b085dbbedd7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-btm4f\" (UID: \"62603892-88ce-474d-8d32-3b085dbbedd7\") " pod="openshift-multus/multus-additional-cni-plugins-btm4f"
Sep 10 02:46:51.070260 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070178    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-run-systemd\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.070260 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070200    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-host-run-netns\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.070260 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070221    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-etc-modprobe-d\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.070377 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070259    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-etc-sysctl-conf\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.070377 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070282    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-lib-modules\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.070377 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070314    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-host\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.070377 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070348    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/62603892-88ce-474d-8d32-3b085dbbedd7-cni-binary-copy\") pod \"multus-additional-cni-plugins-btm4f\" (UID: \"62603892-88ce-474d-8d32-3b085dbbedd7\") " pod="openshift-multus/multus-additional-cni-plugins-btm4f"
Sep 10 02:46:51.070509 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070375    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-node-log\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.070509 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070414    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/de3a2ed4-28df-4dea-8e14-c5dfe2981ba5-kubelet-config\") pod \"global-pull-secret-syncer-5qtg6\" (UID: \"de3a2ed4-28df-4dea-8e14-c5dfe2981ba5\") " pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:46:51.070509 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070479    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/de3a2ed4-28df-4dea-8e14-c5dfe2981ba5-dbus\") pod \"global-pull-secret-syncer-5qtg6\" (UID: \"de3a2ed4-28df-4dea-8e14-c5dfe2981ba5\") " pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:46:51.070620 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070506    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-system-cni-dir\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.070620 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070515    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/de3a2ed4-28df-4dea-8e14-c5dfe2981ba5-kubelet-config\") pod \"global-pull-secret-syncer-5qtg6\" (UID: \"de3a2ed4-28df-4dea-8e14-c5dfe2981ba5\") " pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:46:51.070620 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070530    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-etc-systemd\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.070620 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070553    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba9d4732-9d25-4549-98de-12a27aee412f-ovnkube-config\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.070620 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070575    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba9d4732-9d25-4549-98de-12a27aee412f-env-overrides\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.070620 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070576    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/de3a2ed4-28df-4dea-8e14-c5dfe2981ba5-dbus\") pod \"global-pull-secret-syncer-5qtg6\" (UID: \"de3a2ed4-28df-4dea-8e14-c5dfe2981ba5\") " pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:46:51.070620 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070590    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bf5695bb-3571-4150-b792-ab3b5d1283c6-multus-daemon-config\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.070620 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070606    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-etc-kubernetes\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.070928 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070625    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlslm\" (UniqueName: \"kubernetes.io/projected/62603892-88ce-474d-8d32-3b085dbbedd7-kube-api-access-jlslm\") pod \"multus-additional-cni-plugins-btm4f\" (UID: \"62603892-88ce-474d-8d32-3b085dbbedd7\") " pod="openshift-multus/multus-additional-cni-plugins-btm4f"
Sep 10 02:46:51.070928 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070650    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3ab870ac-9723-4beb-b2d1-c043dd831bae-iptables-alerter-script\") pod \"iptables-alerter-9j9fc\" (UID: \"3ab870ac-9723-4beb-b2d1-c043dd831bae\") " pod="openshift-network-operator/iptables-alerter-9j9fc"
Sep 10 02:46:51.070928 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070682    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-run-openvswitch\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.070928 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070718    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-host-var-lib-cni-bin\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.070928 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070743    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rstwl\" (UniqueName: \"kubernetes.io/projected/f90981c4-23c1-4950-8786-2f4bb0d6e3fb-kube-api-access-rstwl\") pod \"node-resolver-rl4nr\" (UID: \"f90981c4-23c1-4950-8786-2f4bb0d6e3fb\") " pod="openshift-dns/node-resolver-rl4nr"
Sep 10 02:46:51.070928 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070771    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62603892-88ce-474d-8d32-3b085dbbedd7-cnibin\") pod \"multus-additional-cni-plugins-btm4f\" (UID: \"62603892-88ce-474d-8d32-3b085dbbedd7\") " pod="openshift-multus/multus-additional-cni-plugins-btm4f"
Sep 10 02:46:51.070928 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070801    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3ab870ac-9723-4beb-b2d1-c043dd831bae-host-slash\") pod \"iptables-alerter-9j9fc\" (UID: \"3ab870ac-9723-4beb-b2d1-c043dd831bae\") " pod="openshift-network-operator/iptables-alerter-9j9fc"
Sep 10 02:46:51.070928 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070825    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-host-var-lib-kubelet\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.070928 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070839    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjf96\" (UniqueName: \"kubernetes.io/projected/bf5695bb-3571-4150-b792-ab3b5d1283c6-kube-api-access-tjf96\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.070928 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070854    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/494fc5ce-7947-4179-ae56-903c34b716b5-sys-fs\") pod \"aws-ebs-csi-driver-node-qfz5c\" (UID: \"494fc5ce-7947-4179-ae56-903c34b716b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfz5c"
Sep 10 02:46:51.070928 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070873    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-etc-kubernetes\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.070928 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070887    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-etc-tuned\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.070928 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070900    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-tmp\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.070928 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070926    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62603892-88ce-474d-8d32-3b085dbbedd7-system-cni-dir\") pod \"multus-additional-cni-plugins-btm4f\" (UID: \"62603892-88ce-474d-8d32-3b085dbbedd7\") " pod="openshift-multus/multus-additional-cni-plugins-btm4f"
Sep 10 02:46:51.071412 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070954    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c68ab7ea-a147-44d5-81a1-91eca0da2366-serviceca\") pod \"node-ca-4xpxz\" (UID: \"c68ab7ea-a147-44d5-81a1-91eca0da2366\") " pod="openshift-image-registry/node-ca-4xpxz"
Sep 10 02:46:51.071412 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.070979    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-multus-cni-dir\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.071412 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071020    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxjmr\" (UniqueName: \"kubernetes.io/projected/3ab870ac-9723-4beb-b2d1-c043dd831bae-kube-api-access-gxjmr\") pod \"iptables-alerter-9j9fc\" (UID: \"3ab870ac-9723-4beb-b2d1-c043dd831bae\") " pod="openshift-network-operator/iptables-alerter-9j9fc"
Sep 10 02:46:51.071412 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071040    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-etc-openvswitch\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.071412 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071058    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-run-ovn\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.071412 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071086    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-log-socket\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.071412 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071134    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96gxm\" (UniqueName: \"kubernetes.io/projected/e7d69b54-6b09-4496-abe6-6da05c77e958-kube-api-access-96gxm\") pod \"network-check-target-7xn49\" (UID: \"e7d69b54-6b09-4496-abe6-6da05c77e958\") " pod="openshift-network-diagnostics/network-check-target-7xn49"
Sep 10 02:46:51.071412 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071152    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c68ab7ea-a147-44d5-81a1-91eca0da2366-host\") pod \"node-ca-4xpxz\" (UID: \"c68ab7ea-a147-44d5-81a1-91eca0da2366\") " pod="openshift-image-registry/node-ca-4xpxz"
Sep 10 02:46:51.071412 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071166    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx5r4\" (UniqueName: \"kubernetes.io/projected/494fc5ce-7947-4179-ae56-903c34b716b5-kube-api-access-tx5r4\") pod \"aws-ebs-csi-driver-node-qfz5c\" (UID: \"494fc5ce-7947-4179-ae56-903c34b716b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfz5c"
Sep 10 02:46:51.071412 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071179    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-run\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.071412 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071193    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-var-lib-kubelet\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.071412 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071208    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c68ab7ea-a147-44d5-81a1-91eca0da2366-host\") pod \"node-ca-4xpxz\" (UID: \"c68ab7ea-a147-44d5-81a1-91eca0da2366\") " pod="openshift-image-registry/node-ca-4xpxz"
Sep 10 02:46:51.071412 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071207    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx99d\" (UniqueName: \"kubernetes.io/projected/aa599110-d0eb-4cc2-b4e9-e3984e9717fe-kube-api-access-sx99d\") pod \"network-metrics-daemon-m5ph4\" (UID: \"aa599110-d0eb-4cc2-b4e9-e3984e9717fe\") " pod="openshift-multus/network-metrics-daemon-m5ph4"
Sep 10 02:46:51.071412 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071252    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de3a2ed4-28df-4dea-8e14-c5dfe2981ba5-original-pull-secret\") pod \"global-pull-secret-syncer-5qtg6\" (UID: \"de3a2ed4-28df-4dea-8e14-c5dfe2981ba5\") " pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:46:51.071412 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071271    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-multus-conf-dir\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.071412 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071286    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/494fc5ce-7947-4179-ae56-903c34b716b5-device-dir\") pod \"aws-ebs-csi-driver-node-qfz5c\" (UID: \"494fc5ce-7947-4179-ae56-903c34b716b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfz5c"
Sep 10 02:46:51.071412 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071315    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-etc-sysconfig\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.072155 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:51.071345    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 02:46:51.072155 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071375    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-sys\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.072155 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:51.071413    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de3a2ed4-28df-4dea-8e14-c5dfe2981ba5-original-pull-secret podName:de3a2ed4-28df-4dea-8e14-c5dfe2981ba5 nodeName:}" failed. No retries permitted until 2026-09-10 02:46:51.571398338 +0000 UTC m=+1.946484237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de3a2ed4-28df-4dea-8e14-c5dfe2981ba5-original-pull-secret") pod "global-pull-secret-syncer-5qtg6" (UID: "de3a2ed4-28df-4dea-8e14-c5dfe2981ba5") : object "kube-system"/"original-pull-secret" not registered
Sep 10 02:46:51.072155 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071434    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c68ab7ea-a147-44d5-81a1-91eca0da2366-serviceca\") pod \"node-ca-4xpxz\" (UID: \"c68ab7ea-a147-44d5-81a1-91eca0da2366\") " pod="openshift-image-registry/node-ca-4xpxz"
Sep 10 02:46:51.072155 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071456    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-host-run-netns\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.072155 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071477    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-host-cni-netd\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.072155 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071491    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba9d4732-9d25-4549-98de-12a27aee412f-ovn-node-metrics-cert\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.072155 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071520    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-host-kubelet\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.072155 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071538    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/82b3bbcd-1ae2-4bbc-88a7-61aff07f409a-agent-certs\") pod \"konnectivity-agent-296rx\" (UID: \"82b3bbcd-1ae2-4bbc-88a7-61aff07f409a\") " pod="kube-system/konnectivity-agent-296rx"
Sep 10 02:46:51.072155 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071551    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/494fc5ce-7947-4179-ae56-903c34b716b5-etc-selinux\") pod \"aws-ebs-csi-driver-node-qfz5c\" (UID: \"494fc5ce-7947-4179-ae56-903c34b716b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfz5c"
Sep 10 02:46:51.072155 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071566    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62603892-88ce-474d-8d32-3b085dbbedd7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-btm4f\" (UID: \"62603892-88ce-474d-8d32-3b085dbbedd7\") " pod="openshift-multus/multus-additional-cni-plugins-btm4f"
Sep 10 02:46:51.072155 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071592    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9b9k\" (UniqueName: \"kubernetes.io/projected/c68ab7ea-a147-44d5-81a1-91eca0da2366-kube-api-access-t9b9k\") pod \"node-ca-4xpxz\" (UID: \"c68ab7ea-a147-44d5-81a1-91eca0da2366\") " pod="openshift-image-registry/node-ca-4xpxz"
Sep 10 02:46:51.072155 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071613    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/494fc5ce-7947-4179-ae56-903c34b716b5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qfz5c\" (UID: \"494fc5ce-7947-4179-ae56-903c34b716b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfz5c"
Sep 10 02:46:51.072155 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071628    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2ch5\" (UniqueName: \"kubernetes.io/projected/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-kube-api-access-b2ch5\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.072155 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071642    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqjjk\" (UniqueName: \"kubernetes.io/projected/ba9d4732-9d25-4549-98de-12a27aee412f-kube-api-access-cqjjk\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.072155 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071654    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-etc-sysctl-d\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.072704 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071689    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/82b3bbcd-1ae2-4bbc-88a7-61aff07f409a-konnectivity-ca\") pod \"konnectivity-agent-296rx\" (UID: \"82b3bbcd-1ae2-4bbc-88a7-61aff07f409a\") " pod="kube-system/konnectivity-agent-296rx"
Sep 10 02:46:51.072704 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071750    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-host-run-k8s-cni-cncf-io\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.072704 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071779    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-hostroot\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.072704 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071828    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-host-run-multus-certs\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.072704 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071858    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/494fc5ce-7947-4179-ae56-903c34b716b5-registration-dir\") pod \"aws-ebs-csi-driver-node-qfz5c\" (UID: \"494fc5ce-7947-4179-ae56-903c34b716b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfz5c"
Sep 10 02:46:51.072704 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071883    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-host-slash\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.072704 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071915    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf5695bb-3571-4150-b792-ab3b5d1283c6-cni-binary-copy\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.072704 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071945    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f90981c4-23c1-4950-8786-2f4bb0d6e3fb-tmp-dir\") pod \"node-resolver-rl4nr\" (UID: \"f90981c4-23c1-4950-8786-2f4bb0d6e3fb\") " pod="openshift-dns/node-resolver-rl4nr"
Sep 10 02:46:51.072704 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071969    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/62603892-88ce-474d-8d32-3b085dbbedd7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-btm4f\" (UID: \"62603892-88ce-474d-8d32-3b085dbbedd7\") " pod="openshift-multus/multus-additional-cni-plugins-btm4f"
Sep 10 02:46:51.072704 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.071996    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-systemd-units\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.072704 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.072031    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-host-run-ovn-kubernetes\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.072704 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.072056    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-host-cni-bin\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.072704 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.072083    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.072704 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.072123    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-cnibin\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.072704 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.072159    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/494fc5ce-7947-4179-ae56-903c34b716b5-socket-dir\") pod \"aws-ebs-csi-driver-node-qfz5c\" (UID: \"494fc5ce-7947-4179-ae56-903c34b716b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfz5c"
Sep 10 02:46:51.072704 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.072198    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f90981c4-23c1-4950-8786-2f4bb0d6e3fb-hosts-file\") pod \"node-resolver-rl4nr\" (UID: \"f90981c4-23c1-4950-8786-2f4bb0d6e3fb\") " pod="openshift-dns/node-resolver-rl4nr"
Sep 10 02:46:51.073224 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.072225    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-var-lib-openvswitch\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.073224 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.072258    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba9d4732-9d25-4549-98de-12a27aee412f-ovnkube-script-lib\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.073224 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.072279    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa599110-d0eb-4cc2-b4e9-e3984e9717fe-metrics-certs\") pod \"network-metrics-daemon-m5ph4\" (UID: \"aa599110-d0eb-4cc2-b4e9-e3984e9717fe\") " pod="openshift-multus/network-metrics-daemon-m5ph4"
Sep 10 02:46:51.073224 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.072314    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-multus-socket-dir-parent\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.073224 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.072342    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-host-var-lib-cni-multus\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.077439 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.077421    2558 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 02:46:51.080451 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.080436    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9b9k\" (UniqueName: \"kubernetes.io/projected/c68ab7ea-a147-44d5-81a1-91eca0da2366-kube-api-access-t9b9k\") pod \"node-ca-4xpxz\" (UID: \"c68ab7ea-a147-44d5-81a1-91eca0da2366\") " pod="openshift-image-registry/node-ca-4xpxz"
Sep 10 02:46:51.124476 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:51.124437    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01f46892139817c71935091c0eabf5f5.slice/crio-b4124b7b15847a4128eddcc546fb7984617e59ce4ce8708b90f0a23dc26f39b8 WatchSource:0}: Error finding container b4124b7b15847a4128eddcc546fb7984617e59ce4ce8708b90f0a23dc26f39b8: Status 404 returned error can't find the container with id b4124b7b15847a4128eddcc546fb7984617e59ce4ce8708b90f0a23dc26f39b8
Sep 10 02:46:51.124688 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:51.124665    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3279872c23798b7320a2d8a5e9247b5.slice/crio-8999ec8e822e76309bbc75a1459761b70bf48553933cc924945e7e8cc90ea4c4 WatchSource:0}: Error finding container 8999ec8e822e76309bbc75a1459761b70bf48553933cc924945e7e8cc90ea4c4: Status 404 returned error can't find the container with id 8999ec8e822e76309bbc75a1459761b70bf48553933cc924945e7e8cc90ea4c4
Sep 10 02:46:51.128515 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.128494    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 02:46:51.172710 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.172689    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-multus-socket-dir-parent\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.172808 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.172714    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-host-var-lib-cni-multus\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.172808 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.172729    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/62603892-88ce-474d-8d32-3b085dbbedd7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-btm4f\" (UID: \"62603892-88ce-474d-8d32-3b085dbbedd7\") " pod="openshift-multus/multus-additional-cni-plugins-btm4f"
Sep 10 02:46:51.172808 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.172743    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-run-systemd\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.172808 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.172758    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-host-run-netns\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.172808 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.172779    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-etc-modprobe-d\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.172808 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.172790    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-host-var-lib-cni-multus\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.172808 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.172801    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-etc-sysctl-conf\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.173139 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.172825    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-lib-modules\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.173139 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.172822    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-multus-socket-dir-parent\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.173139 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.172848    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-host\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.173139 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.172864    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/62603892-88ce-474d-8d32-3b085dbbedd7-cni-binary-copy\") pod \"multus-additional-cni-plugins-btm4f\" (UID: \"62603892-88ce-474d-8d32-3b085dbbedd7\") " pod="openshift-multus/multus-additional-cni-plugins-btm4f"
Sep 10 02:46:51.173139 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.172840    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-run-systemd\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.173139 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.172883    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-etc-modprobe-d\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.173139 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.172878    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-node-log\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.173139 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.172910    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-node-log\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.173139 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.172922    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-system-cni-dir\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.173139 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.172946    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-etc-systemd\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.173139 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.172941    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-host\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.173139 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.172961    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-system-cni-dir\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.173139 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.172844    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-host-run-netns\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.173139 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.172977    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-etc-systemd\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.173139 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.172979    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba9d4732-9d25-4549-98de-12a27aee412f-ovnkube-config\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.173139 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173008    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba9d4732-9d25-4549-98de-12a27aee412f-env-overrides\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.173139 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173012    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-etc-sysctl-conf\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.173139 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173033    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bf5695bb-3571-4150-b792-ab3b5d1283c6-multus-daemon-config\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.174011 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173057    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-etc-kubernetes\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.174011 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173083    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlslm\" (UniqueName: \"kubernetes.io/projected/62603892-88ce-474d-8d32-3b085dbbedd7-kube-api-access-jlslm\") pod \"multus-additional-cni-plugins-btm4f\" (UID: \"62603892-88ce-474d-8d32-3b085dbbedd7\") " pod="openshift-multus/multus-additional-cni-plugins-btm4f"
Sep 10 02:46:51.174011 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173125    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3ab870ac-9723-4beb-b2d1-c043dd831bae-iptables-alerter-script\") pod \"iptables-alerter-9j9fc\" (UID: \"3ab870ac-9723-4beb-b2d1-c043dd831bae\") " pod="openshift-network-operator/iptables-alerter-9j9fc"
Sep 10 02:46:51.174011 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173151    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-run-openvswitch\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.174011 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173174    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-host-var-lib-cni-bin\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.174011 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173198    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rstwl\" (UniqueName: \"kubernetes.io/projected/f90981c4-23c1-4950-8786-2f4bb0d6e3fb-kube-api-access-rstwl\") pod \"node-resolver-rl4nr\" (UID: \"f90981c4-23c1-4950-8786-2f4bb0d6e3fb\") " pod="openshift-dns/node-resolver-rl4nr"
Sep 10 02:46:51.174011 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173220    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62603892-88ce-474d-8d32-3b085dbbedd7-cnibin\") pod \"multus-additional-cni-plugins-btm4f\" (UID: \"62603892-88ce-474d-8d32-3b085dbbedd7\") " pod="openshift-multus/multus-additional-cni-plugins-btm4f"
Sep 10 02:46:51.174011 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173244    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3ab870ac-9723-4beb-b2d1-c043dd831bae-host-slash\") pod \"iptables-alerter-9j9fc\" (UID: \"3ab870ac-9723-4beb-b2d1-c043dd831bae\") " pod="openshift-network-operator/iptables-alerter-9j9fc"
Sep 10 02:46:51.174011 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173266    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-host-var-lib-kubelet\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.174011 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173289    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjf96\" (UniqueName: \"kubernetes.io/projected/bf5695bb-3571-4150-b792-ab3b5d1283c6-kube-api-access-tjf96\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.174011 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173317    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/494fc5ce-7947-4179-ae56-903c34b716b5-sys-fs\") pod \"aws-ebs-csi-driver-node-qfz5c\" (UID: \"494fc5ce-7947-4179-ae56-903c34b716b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfz5c"
Sep 10 02:46:51.174011 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173354    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-etc-kubernetes\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.174011 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173371    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-etc-kubernetes\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.174011 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173377    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-etc-tuned\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.174011 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173419    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-tmp\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.174011 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173425    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/62603892-88ce-474d-8d32-3b085dbbedd7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-btm4f\" (UID: \"62603892-88ce-474d-8d32-3b085dbbedd7\") " pod="openshift-multus/multus-additional-cni-plugins-btm4f"
Sep 10 02:46:51.174011 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173445    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62603892-88ce-474d-8d32-3b085dbbedd7-system-cni-dir\") pod \"multus-additional-cni-plugins-btm4f\" (UID: \"62603892-88ce-474d-8d32-3b085dbbedd7\") " pod="openshift-multus/multus-additional-cni-plugins-btm4f"
Sep 10 02:46:51.174819 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173448    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/62603892-88ce-474d-8d32-3b085dbbedd7-cni-binary-copy\") pod \"multus-additional-cni-plugins-btm4f\" (UID: \"62603892-88ce-474d-8d32-3b085dbbedd7\") " pod="openshift-multus/multus-additional-cni-plugins-btm4f"
Sep 10 02:46:51.174819 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173473    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba9d4732-9d25-4549-98de-12a27aee412f-env-overrides\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.174819 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173476    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-multus-cni-dir\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.174819 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173507    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62603892-88ce-474d-8d32-3b085dbbedd7-system-cni-dir\") pod \"multus-additional-cni-plugins-btm4f\" (UID: \"62603892-88ce-474d-8d32-3b085dbbedd7\") " pod="openshift-multus/multus-additional-cni-plugins-btm4f"
Sep 10 02:46:51.174819 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173528    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-multus-cni-dir\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.174819 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173535    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba9d4732-9d25-4549-98de-12a27aee412f-ovnkube-config\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.174819 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173526    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxjmr\" (UniqueName: \"kubernetes.io/projected/3ab870ac-9723-4beb-b2d1-c043dd831bae-kube-api-access-gxjmr\") pod \"iptables-alerter-9j9fc\" (UID: \"3ab870ac-9723-4beb-b2d1-c043dd831bae\") " pod="openshift-network-operator/iptables-alerter-9j9fc"
Sep 10 02:46:51.174819 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173553    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62603892-88ce-474d-8d32-3b085dbbedd7-cnibin\") pod \"multus-additional-cni-plugins-btm4f\" (UID: \"62603892-88ce-474d-8d32-3b085dbbedd7\") " pod="openshift-multus/multus-additional-cni-plugins-btm4f"
Sep 10 02:46:51.174819 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173566    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-run-openvswitch\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.174819 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173579    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-etc-openvswitch\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.174819 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173598    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bf5695bb-3571-4150-b792-ab3b5d1283c6-multus-daemon-config\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.174819 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173601    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-host-var-lib-cni-bin\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.174819 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173602    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-run-ovn\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.174819 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173622    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-lib-modules\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.174819 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173640    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-log-socket\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.174819 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173649    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-run-ovn\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.174819 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173653    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-host-var-lib-kubelet\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.174819 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173665    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96gxm\" (UniqueName: \"kubernetes.io/projected/e7d69b54-6b09-4496-abe6-6da05c77e958-kube-api-access-96gxm\") pod \"network-check-target-7xn49\" (UID: \"e7d69b54-6b09-4496-abe6-6da05c77e958\") " pod="openshift-network-diagnostics/network-check-target-7xn49"
Sep 10 02:46:51.175754 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173692    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx5r4\" (UniqueName: \"kubernetes.io/projected/494fc5ce-7947-4179-ae56-903c34b716b5-kube-api-access-tx5r4\") pod \"aws-ebs-csi-driver-node-qfz5c\" (UID: \"494fc5ce-7947-4179-ae56-903c34b716b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfz5c"
Sep 10 02:46:51.175754 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173714    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-run\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.175754 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173720    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/494fc5ce-7947-4179-ae56-903c34b716b5-sys-fs\") pod \"aws-ebs-csi-driver-node-qfz5c\" (UID: \"494fc5ce-7947-4179-ae56-903c34b716b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfz5c"
Sep 10 02:46:51.175754 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173735    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-var-lib-kubelet\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.175754 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173740    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-etc-openvswitch\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.175754 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173769    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx99d\" (UniqueName: \"kubernetes.io/projected/aa599110-d0eb-4cc2-b4e9-e3984e9717fe-kube-api-access-sx99d\") pod \"network-metrics-daemon-m5ph4\" (UID: \"aa599110-d0eb-4cc2-b4e9-e3984e9717fe\") " pod="openshift-multus/network-metrics-daemon-m5ph4"
Sep 10 02:46:51.175754 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173798    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-var-lib-kubelet\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.175754 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173805    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-log-socket\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.175754 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173814    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-multus-conf-dir\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.175754 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173839    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/494fc5ce-7947-4179-ae56-903c34b716b5-device-dir\") pod \"aws-ebs-csi-driver-node-qfz5c\" (UID: \"494fc5ce-7947-4179-ae56-903c34b716b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfz5c"
Sep 10 02:46:51.175754 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173863    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-etc-sysconfig\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.175754 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173890    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-sys\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.175754 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173894    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3ab870ac-9723-4beb-b2d1-c043dd831bae-host-slash\") pod \"iptables-alerter-9j9fc\" (UID: \"3ab870ac-9723-4beb-b2d1-c043dd831bae\") " pod="openshift-network-operator/iptables-alerter-9j9fc"
Sep 10 02:46:51.175754 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173918    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-host-run-netns\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.175754 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173937    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-multus-conf-dir\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.175754 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173940    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-host-cni-netd\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.175754 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173956    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-run\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.175754 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173841    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-etc-kubernetes\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.176386 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173971    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba9d4732-9d25-4549-98de-12a27aee412f-ovn-node-metrics-cert\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.176386 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173984    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/494fc5ce-7947-4179-ae56-903c34b716b5-device-dir\") pod \"aws-ebs-csi-driver-node-qfz5c\" (UID: \"494fc5ce-7947-4179-ae56-903c34b716b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfz5c"
Sep 10 02:46:51.176386 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.173993    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-host-kubelet\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.176386 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174002    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-host-run-netns\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.176386 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174003    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-sys\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.176386 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174022    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/82b3bbcd-1ae2-4bbc-88a7-61aff07f409a-agent-certs\") pod \"konnectivity-agent-296rx\" (UID: \"82b3bbcd-1ae2-4bbc-88a7-61aff07f409a\") " pod="kube-system/konnectivity-agent-296rx"
Sep 10 02:46:51.176386 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174024    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-host-cni-netd\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.176386 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174044    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-host-kubelet\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.176386 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174050    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-etc-sysconfig\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.176386 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174050    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/494fc5ce-7947-4179-ae56-903c34b716b5-etc-selinux\") pod \"aws-ebs-csi-driver-node-qfz5c\" (UID: \"494fc5ce-7947-4179-ae56-903c34b716b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfz5c"
Sep 10 02:46:51.176386 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174117    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/494fc5ce-7947-4179-ae56-903c34b716b5-etc-selinux\") pod \"aws-ebs-csi-driver-node-qfz5c\" (UID: \"494fc5ce-7947-4179-ae56-903c34b716b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfz5c"
Sep 10 02:46:51.176386 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174051    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3ab870ac-9723-4beb-b2d1-c043dd831bae-iptables-alerter-script\") pod \"iptables-alerter-9j9fc\" (UID: \"3ab870ac-9723-4beb-b2d1-c043dd831bae\") " pod="openshift-network-operator/iptables-alerter-9j9fc"
Sep 10 02:46:51.176386 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174126    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62603892-88ce-474d-8d32-3b085dbbedd7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-btm4f\" (UID: \"62603892-88ce-474d-8d32-3b085dbbedd7\") " pod="openshift-multus/multus-additional-cni-plugins-btm4f"
Sep 10 02:46:51.176386 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174173    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/494fc5ce-7947-4179-ae56-903c34b716b5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qfz5c\" (UID: \"494fc5ce-7947-4179-ae56-903c34b716b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfz5c"
Sep 10 02:46:51.176386 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174208    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2ch5\" (UniqueName: \"kubernetes.io/projected/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-kube-api-access-b2ch5\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.176386 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174233    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqjjk\" (UniqueName: \"kubernetes.io/projected/ba9d4732-9d25-4549-98de-12a27aee412f-kube-api-access-cqjjk\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.176386 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174258    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-etc-sysctl-d\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.176898 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174281    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/82b3bbcd-1ae2-4bbc-88a7-61aff07f409a-konnectivity-ca\") pod \"konnectivity-agent-296rx\" (UID: \"82b3bbcd-1ae2-4bbc-88a7-61aff07f409a\") " pod="kube-system/konnectivity-agent-296rx"
Sep 10 02:46:51.176898 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174304    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-host-run-k8s-cni-cncf-io\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.176898 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174327    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-hostroot\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.176898 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174352    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-host-run-multus-certs\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.176898 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174375    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/494fc5ce-7947-4179-ae56-903c34b716b5-registration-dir\") pod \"aws-ebs-csi-driver-node-qfz5c\" (UID: \"494fc5ce-7947-4179-ae56-903c34b716b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfz5c"
Sep 10 02:46:51.176898 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174396    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-host-slash\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.176898 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174430    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/494fc5ce-7947-4179-ae56-903c34b716b5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qfz5c\" (UID: \"494fc5ce-7947-4179-ae56-903c34b716b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfz5c"
Sep 10 02:46:51.176898 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174489    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-etc-sysctl-d\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.176898 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174471    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-hostroot\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.176898 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174525    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-host-run-k8s-cni-cncf-io\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.176898 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174539    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-host-run-multus-certs\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.176898 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174569    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/494fc5ce-7947-4179-ae56-903c34b716b5-registration-dir\") pod \"aws-ebs-csi-driver-node-qfz5c\" (UID: \"494fc5ce-7947-4179-ae56-903c34b716b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfz5c"
Sep 10 02:46:51.176898 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174569    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf5695bb-3571-4150-b792-ab3b5d1283c6-cni-binary-copy\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.176898 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174603    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-host-slash\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.176898 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174633    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f90981c4-23c1-4950-8786-2f4bb0d6e3fb-tmp-dir\") pod \"node-resolver-rl4nr\" (UID: \"f90981c4-23c1-4950-8786-2f4bb0d6e3fb\") " pod="openshift-dns/node-resolver-rl4nr"
Sep 10 02:46:51.176898 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174682    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/62603892-88ce-474d-8d32-3b085dbbedd7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-btm4f\" (UID: \"62603892-88ce-474d-8d32-3b085dbbedd7\") " pod="openshift-multus/multus-additional-cni-plugins-btm4f"
Sep 10 02:46:51.176898 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174710    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-systemd-units\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.177372 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174734    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-host-run-ovn-kubernetes\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.177372 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174761    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-host-cni-bin\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.177372 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174784    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.177372 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174810    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-cnibin\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.177372 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174832    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/494fc5ce-7947-4179-ae56-903c34b716b5-socket-dir\") pod \"aws-ebs-csi-driver-node-qfz5c\" (UID: \"494fc5ce-7947-4179-ae56-903c34b716b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfz5c"
Sep 10 02:46:51.177372 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174856    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f90981c4-23c1-4950-8786-2f4bb0d6e3fb-hosts-file\") pod \"node-resolver-rl4nr\" (UID: \"f90981c4-23c1-4950-8786-2f4bb0d6e3fb\") " pod="openshift-dns/node-resolver-rl4nr"
Sep 10 02:46:51.177372 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174880    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-var-lib-openvswitch\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.177372 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174903    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba9d4732-9d25-4549-98de-12a27aee412f-ovnkube-script-lib\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.177372 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174966    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/82b3bbcd-1ae2-4bbc-88a7-61aff07f409a-konnectivity-ca\") pod \"konnectivity-agent-296rx\" (UID: \"82b3bbcd-1ae2-4bbc-88a7-61aff07f409a\") " pod="kube-system/konnectivity-agent-296rx"
Sep 10 02:46:51.177372 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.174968    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa599110-d0eb-4cc2-b4e9-e3984e9717fe-metrics-certs\") pod \"network-metrics-daemon-m5ph4\" (UID: \"aa599110-d0eb-4cc2-b4e9-e3984e9717fe\") " pod="openshift-multus/network-metrics-daemon-m5ph4"
Sep 10 02:46:51.177372 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.175022    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf5695bb-3571-4150-b792-ab3b5d1283c6-cni-binary-copy\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.177372 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:51.175052    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 02:46:51.177372 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.175063    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62603892-88ce-474d-8d32-3b085dbbedd7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-btm4f\" (UID: \"62603892-88ce-474d-8d32-3b085dbbedd7\") " pod="openshift-multus/multus-additional-cni-plugins-btm4f"
Sep 10 02:46:51.177372 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:51.175116    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa599110-d0eb-4cc2-b4e9-e3984e9717fe-metrics-certs podName:aa599110-d0eb-4cc2-b4e9-e3984e9717fe nodeName:}" failed. No retries permitted until 2026-09-10 02:46:51.675083645 +0000 UTC m=+2.050169553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aa599110-d0eb-4cc2-b4e9-e3984e9717fe-metrics-certs") pod "network-metrics-daemon-m5ph4" (UID: "aa599110-d0eb-4cc2-b4e9-e3984e9717fe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 02:46:51.177372 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.175125    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.177372 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.175150    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f90981c4-23c1-4950-8786-2f4bb0d6e3fb-hosts-file\") pod \"node-resolver-rl4nr\" (UID: \"f90981c4-23c1-4950-8786-2f4bb0d6e3fb\") " pod="openshift-dns/node-resolver-rl4nr"
Sep 10 02:46:51.177372 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.175150    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-systemd-units\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.177824 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.175169    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-var-lib-openvswitch\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.177824 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.175178    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-host-run-ovn-kubernetes\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.177824 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.175224    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba9d4732-9d25-4549-98de-12a27aee412f-host-cni-bin\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.177824 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.175348    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf5695bb-3571-4150-b792-ab3b5d1283c6-cnibin\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.177824 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.175417    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f90981c4-23c1-4950-8786-2f4bb0d6e3fb-tmp-dir\") pod \"node-resolver-rl4nr\" (UID: \"f90981c4-23c1-4950-8786-2f4bb0d6e3fb\") " pod="openshift-dns/node-resolver-rl4nr"
Sep 10 02:46:51.177824 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.175463    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/494fc5ce-7947-4179-ae56-903c34b716b5-socket-dir\") pod \"aws-ebs-csi-driver-node-qfz5c\" (UID: \"494fc5ce-7947-4179-ae56-903c34b716b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfz5c"
Sep 10 02:46:51.177824 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.175646    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/62603892-88ce-474d-8d32-3b085dbbedd7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-btm4f\" (UID: \"62603892-88ce-474d-8d32-3b085dbbedd7\") " pod="openshift-multus/multus-additional-cni-plugins-btm4f"
Sep 10 02:46:51.177824 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.175680    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba9d4732-9d25-4549-98de-12a27aee412f-ovnkube-script-lib\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.177824 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.176266    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-tmp\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.177824 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.176307    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-etc-tuned\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.177824 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.176480    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba9d4732-9d25-4549-98de-12a27aee412f-ovn-node-metrics-cert\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.177824 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.177056    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/82b3bbcd-1ae2-4bbc-88a7-61aff07f409a-agent-certs\") pod \"konnectivity-agent-296rx\" (UID: \"82b3bbcd-1ae2-4bbc-88a7-61aff07f409a\") " pod="kube-system/konnectivity-agent-296rx"
Sep 10 02:46:51.184868 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:51.184848    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 02:46:51.184868 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:51.184870    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 02:46:51.185014 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:51.184880    2558 projected.go:196] Error preparing data for projected volume kube-api-access-96gxm for pod openshift-network-diagnostics/network-check-target-7xn49: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 02:46:51.185014 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:51.184931    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7d69b54-6b09-4496-abe6-6da05c77e958-kube-api-access-96gxm podName:e7d69b54-6b09-4496-abe6-6da05c77e958 nodeName:}" failed. No retries permitted until 2026-09-10 02:46:51.68491512 +0000 UTC m=+2.060001019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-96gxm" (UniqueName: "kubernetes.io/projected/e7d69b54-6b09-4496-abe6-6da05c77e958-kube-api-access-96gxm") pod "network-check-target-7xn49" (UID: "e7d69b54-6b09-4496-abe6-6da05c77e958") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 02:46:51.187023 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.186980    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx5r4\" (UniqueName: \"kubernetes.io/projected/494fc5ce-7947-4179-ae56-903c34b716b5-kube-api-access-tx5r4\") pod \"aws-ebs-csi-driver-node-qfz5c\" (UID: \"494fc5ce-7947-4179-ae56-903c34b716b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfz5c"
Sep 10 02:46:51.187290 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.187273    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjf96\" (UniqueName: \"kubernetes.io/projected/bf5695bb-3571-4150-b792-ab3b5d1283c6-kube-api-access-tjf96\") pod \"multus-pptgm\" (UID: \"bf5695bb-3571-4150-b792-ab3b5d1283c6\") " pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.187991 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.187968    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlslm\" (UniqueName: \"kubernetes.io/projected/62603892-88ce-474d-8d32-3b085dbbedd7-kube-api-access-jlslm\") pod \"multus-additional-cni-plugins-btm4f\" (UID: \"62603892-88ce-474d-8d32-3b085dbbedd7\") " pod="openshift-multus/multus-additional-cni-plugins-btm4f"
Sep 10 02:46:51.188081 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.188032    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2ch5\" (UniqueName: \"kubernetes.io/projected/c31a9e4e-75c0-4f5a-87c2-d323be62dfa5-kube-api-access-b2ch5\") pod \"tuned-qhf58\" (UID: \"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5\") " pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.188081 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.188022    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rstwl\" (UniqueName: \"kubernetes.io/projected/f90981c4-23c1-4950-8786-2f4bb0d6e3fb-kube-api-access-rstwl\") pod \"node-resolver-rl4nr\" (UID: \"f90981c4-23c1-4950-8786-2f4bb0d6e3fb\") " pod="openshift-dns/node-resolver-rl4nr"
Sep 10 02:46:51.188081 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.188055    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx99d\" (UniqueName: \"kubernetes.io/projected/aa599110-d0eb-4cc2-b4e9-e3984e9717fe-kube-api-access-sx99d\") pod \"network-metrics-daemon-m5ph4\" (UID: \"aa599110-d0eb-4cc2-b4e9-e3984e9717fe\") " pod="openshift-multus/network-metrics-daemon-m5ph4"
Sep 10 02:46:51.188468 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.188446    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxjmr\" (UniqueName: \"kubernetes.io/projected/3ab870ac-9723-4beb-b2d1-c043dd831bae-kube-api-access-gxjmr\") pod \"iptables-alerter-9j9fc\" (UID: \"3ab870ac-9723-4beb-b2d1-c043dd831bae\") " pod="openshift-network-operator/iptables-alerter-9j9fc"
Sep 10 02:46:51.189430 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.189415    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqjjk\" (UniqueName: \"kubernetes.io/projected/ba9d4732-9d25-4549-98de-12a27aee412f-kube-api-access-cqjjk\") pod \"ovnkube-node-bltxf\" (UID: \"ba9d4732-9d25-4549-98de-12a27aee412f\") " pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.285729 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.285705    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4xpxz"
Sep 10 02:46:51.295491 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:51.295470    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc68ab7ea_a147_44d5_81a1_91eca0da2366.slice/crio-c5acd2fa93eb93b23da059c6875cd79910b7575455d339a6d4a8cd6f88816fcf WatchSource:0}: Error finding container c5acd2fa93eb93b23da059c6875cd79910b7575455d339a6d4a8cd6f88816fcf: Status 404 returned error can't find the container with id c5acd2fa93eb93b23da059c6875cd79910b7575455d339a6d4a8cd6f88816fcf
Sep 10 02:46:51.304529 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.304512    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pptgm"
Sep 10 02:46:51.312384 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:51.312365    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf5695bb_3571_4150_b792_ab3b5d1283c6.slice/crio-43b2c73879d5c33af0587fe0be374727195505d1c15d740ec7c52825fe379310 WatchSource:0}: Error finding container 43b2c73879d5c33af0587fe0be374727195505d1c15d740ec7c52825fe379310: Status 404 returned error can't find the container with id 43b2c73879d5c33af0587fe0be374727195505d1c15d740ec7c52825fe379310
Sep 10 02:46:51.316461 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.316443    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9j9fc"
Sep 10 02:46:51.325438 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:51.325420    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ab870ac_9723_4beb_b2d1_c043dd831bae.slice/crio-aab19bb13d29bb19122136bd9ee1f12981f8ad25aabe7bfc4bf45325259e30bd WatchSource:0}: Error finding container aab19bb13d29bb19122136bd9ee1f12981f8ad25aabe7bfc4bf45325259e30bd: Status 404 returned error can't find the container with id aab19bb13d29bb19122136bd9ee1f12981f8ad25aabe7bfc4bf45325259e30bd
Sep 10 02:46:51.334282 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.334267    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:46:51.341470 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:51.341453    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba9d4732_9d25_4549_98de_12a27aee412f.slice/crio-6a9a98e675a76b05734b9f5bde2a52e064b49707e5eccf99dd31e1fe4e8b9370 WatchSource:0}: Error finding container 6a9a98e675a76b05734b9f5bde2a52e064b49707e5eccf99dd31e1fe4e8b9370: Status 404 returned error can't find the container with id 6a9a98e675a76b05734b9f5bde2a52e064b49707e5eccf99dd31e1fe4e8b9370
Sep 10 02:46:51.351043 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.351029    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-296rx"
Sep 10 02:46:51.356885 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.356867    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfz5c"
Sep 10 02:46:51.358270 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:51.358253    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82b3bbcd_1ae2_4bbc_88a7_61aff07f409a.slice/crio-10cc190f89cbb01f3e00443b31a667d331f7337980437b58104d36f77f0a72b6 WatchSource:0}: Error finding container 10cc190f89cbb01f3e00443b31a667d331f7337980437b58104d36f77f0a72b6: Status 404 returned error can't find the container with id 10cc190f89cbb01f3e00443b31a667d331f7337980437b58104d36f77f0a72b6
Sep 10 02:46:51.364537 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:51.364516    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod494fc5ce_7947_4179_ae56_903c34b716b5.slice/crio-77c03f79a08132de79c6507c0b29a709cf8e4f1338c15b7cee6bd35b65751823 WatchSource:0}: Error finding container 77c03f79a08132de79c6507c0b29a709cf8e4f1338c15b7cee6bd35b65751823: Status 404 returned error can't find the container with id 77c03f79a08132de79c6507c0b29a709cf8e4f1338c15b7cee6bd35b65751823
Sep 10 02:46:51.373085 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.373071    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qhf58"
Sep 10 02:46:51.380457 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:51.380432    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc31a9e4e_75c0_4f5a_87c2_d323be62dfa5.slice/crio-4cfc0395f4a9a03780cee9d0c91fd9c949b20c8a9069731c0e724282a496afc3 WatchSource:0}: Error finding container 4cfc0395f4a9a03780cee9d0c91fd9c949b20c8a9069731c0e724282a496afc3: Status 404 returned error can't find the container with id 4cfc0395f4a9a03780cee9d0c91fd9c949b20c8a9069731c0e724282a496afc3
Sep 10 02:46:51.390033 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.390018    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rl4nr"
Sep 10 02:46:51.393823 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.393804    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-btm4f"
Sep 10 02:46:51.397638 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:51.397620    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90981c4_23c1_4950_8786_2f4bb0d6e3fb.slice/crio-6bf59fcb67cfccff2d4398ebf57fb03fdfe7347d106bb2d451891333faa6bcf4 WatchSource:0}: Error finding container 6bf59fcb67cfccff2d4398ebf57fb03fdfe7347d106bb2d451891333faa6bcf4: Status 404 returned error can't find the container with id 6bf59fcb67cfccff2d4398ebf57fb03fdfe7347d106bb2d451891333faa6bcf4
Sep 10 02:46:51.401533 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:46:51.401517    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62603892_88ce_474d_8d32_3b085dbbedd7.slice/crio-003af4f8fdb8c1bc26ca2ea3936cfa3d8d7238f7498f03ff74fdf163a40ec681 WatchSource:0}: Error finding container 003af4f8fdb8c1bc26ca2ea3936cfa3d8d7238f7498f03ff74fdf163a40ec681: Status 404 returned error can't find the container with id 003af4f8fdb8c1bc26ca2ea3936cfa3d8d7238f7498f03ff74fdf163a40ec681
Sep 10 02:46:51.473268 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.473249    2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 02:46:51.507853 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.507834    2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 02:46:51.577992 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.577932    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de3a2ed4-28df-4dea-8e14-c5dfe2981ba5-original-pull-secret\") pod \"global-pull-secret-syncer-5qtg6\" (UID: \"de3a2ed4-28df-4dea-8e14-c5dfe2981ba5\") " pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:46:51.578117 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:51.578089    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 02:46:51.578197 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:51.578185    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de3a2ed4-28df-4dea-8e14-c5dfe2981ba5-original-pull-secret podName:de3a2ed4-28df-4dea-8e14-c5dfe2981ba5 nodeName:}" failed. No retries permitted until 2026-09-10 02:46:52.578145204 +0000 UTC m=+2.953231109 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de3a2ed4-28df-4dea-8e14-c5dfe2981ba5-original-pull-secret") pod "global-pull-secret-syncer-5qtg6" (UID: "de3a2ed4-28df-4dea-8e14-c5dfe2981ba5") : object "kube-system"/"original-pull-secret" not registered
Sep 10 02:46:51.682814 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.682770    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa599110-d0eb-4cc2-b4e9-e3984e9717fe-metrics-certs\") pod \"network-metrics-daemon-m5ph4\" (UID: \"aa599110-d0eb-4cc2-b4e9-e3984e9717fe\") " pod="openshift-multus/network-metrics-daemon-m5ph4"
Sep 10 02:46:51.682958 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:51.682916    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 02:46:51.683044 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:51.683025    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa599110-d0eb-4cc2-b4e9-e3984e9717fe-metrics-certs podName:aa599110-d0eb-4cc2-b4e9-e3984e9717fe nodeName:}" failed. No retries permitted until 2026-09-10 02:46:52.682971182 +0000 UTC m=+3.058057087 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aa599110-d0eb-4cc2-b4e9-e3984e9717fe-metrics-certs") pod "network-metrics-daemon-m5ph4" (UID: "aa599110-d0eb-4cc2-b4e9-e3984e9717fe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 02:46:51.785686 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:51.785082    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96gxm\" (UniqueName: \"kubernetes.io/projected/e7d69b54-6b09-4496-abe6-6da05c77e958-kube-api-access-96gxm\") pod \"network-check-target-7xn49\" (UID: \"e7d69b54-6b09-4496-abe6-6da05c77e958\") " pod="openshift-network-diagnostics/network-check-target-7xn49"
Sep 10 02:46:51.785686 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:51.785260    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 02:46:51.785686 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:51.785281    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 02:46:51.785686 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:51.785293    2558 projected.go:196] Error preparing data for projected volume kube-api-access-96gxm for pod openshift-network-diagnostics/network-check-target-7xn49: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 02:46:51.785686 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:51.785346    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7d69b54-6b09-4496-abe6-6da05c77e958-kube-api-access-96gxm podName:e7d69b54-6b09-4496-abe6-6da05c77e958 nodeName:}" failed. No retries permitted until 2026-09-10 02:46:52.785328907 +0000 UTC m=+3.160414809 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-96gxm" (UniqueName: "kubernetes.io/projected/e7d69b54-6b09-4496-abe6-6da05c77e958-kube-api-access-96gxm") pod "network-check-target-7xn49" (UID: "e7d69b54-6b09-4496-abe6-6da05c77e958") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 02:46:52.014280 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:52.014127    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 02:41:51 +0000 UTC" deadline="2028-02-14 04:24:39.8209096 +0000 UTC"
Sep 10 02:46:52.014280 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:52.014183    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12529h37m47.806730138s"
Sep 10 02:46:52.064926 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:52.064420    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:46:52.064926 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:52.064551    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5qtg6" podUID="de3a2ed4-28df-4dea-8e14-c5dfe2981ba5"
Sep 10 02:46:52.073644 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:52.073595    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-296rx" event={"ID":"82b3bbcd-1ae2-4bbc-88a7-61aff07f409a","Type":"ContainerStarted","Data":"10cc190f89cbb01f3e00443b31a667d331f7337980437b58104d36f77f0a72b6"}
Sep 10 02:46:52.082266 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:52.082163    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bltxf" event={"ID":"ba9d4732-9d25-4549-98de-12a27aee412f","Type":"ContainerStarted","Data":"6a9a98e675a76b05734b9f5bde2a52e064b49707e5eccf99dd31e1fe4e8b9370"}
Sep 10 02:46:52.092028 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:52.092005    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 02:46:52.095791 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:52.095766    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9j9fc" event={"ID":"3ab870ac-9723-4beb-b2d1-c043dd831bae","Type":"ContainerStarted","Data":"aab19bb13d29bb19122136bd9ee1f12981f8ad25aabe7bfc4bf45325259e30bd"}
Sep 10 02:46:52.103789 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:52.103722    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-101.ec2.internal" event={"ID":"01f46892139817c71935091c0eabf5f5","Type":"ContainerStarted","Data":"b4124b7b15847a4128eddcc546fb7984617e59ce4ce8708b90f0a23dc26f39b8"}
Sep 10 02:46:52.132283 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:52.132258    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-btm4f" event={"ID":"62603892-88ce-474d-8d32-3b085dbbedd7","Type":"ContainerStarted","Data":"003af4f8fdb8c1bc26ca2ea3936cfa3d8d7238f7498f03ff74fdf163a40ec681"}
Sep 10 02:46:52.144409 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:52.144336    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rl4nr" event={"ID":"f90981c4-23c1-4950-8786-2f4bb0d6e3fb","Type":"ContainerStarted","Data":"6bf59fcb67cfccff2d4398ebf57fb03fdfe7347d106bb2d451891333faa6bcf4"}
Sep 10 02:46:52.154952 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:52.154896    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfz5c" event={"ID":"494fc5ce-7947-4179-ae56-903c34b716b5","Type":"ContainerStarted","Data":"77c03f79a08132de79c6507c0b29a709cf8e4f1338c15b7cee6bd35b65751823"}
Sep 10 02:46:52.168898 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:52.168845    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pptgm" event={"ID":"bf5695bb-3571-4150-b792-ab3b5d1283c6","Type":"ContainerStarted","Data":"43b2c73879d5c33af0587fe0be374727195505d1c15d740ec7c52825fe379310"}
Sep 10 02:46:52.177921 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:52.177848    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4xpxz" event={"ID":"c68ab7ea-a147-44d5-81a1-91eca0da2366","Type":"ContainerStarted","Data":"c5acd2fa93eb93b23da059c6875cd79910b7575455d339a6d4a8cd6f88816fcf"}
Sep 10 02:46:52.200470 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:52.200399    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal" event={"ID":"d3279872c23798b7320a2d8a5e9247b5","Type":"ContainerStarted","Data":"8999ec8e822e76309bbc75a1459761b70bf48553933cc924945e7e8cc90ea4c4"}
Sep 10 02:46:52.203376 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:52.203322    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qhf58" event={"ID":"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5","Type":"ContainerStarted","Data":"4cfc0395f4a9a03780cee9d0c91fd9c949b20c8a9069731c0e724282a496afc3"}
Sep 10 02:46:52.592667 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:52.592633    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de3a2ed4-28df-4dea-8e14-c5dfe2981ba5-original-pull-secret\") pod \"global-pull-secret-syncer-5qtg6\" (UID: \"de3a2ed4-28df-4dea-8e14-c5dfe2981ba5\") " pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:46:52.592815 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:52.592804    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 02:46:52.592872 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:52.592862    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de3a2ed4-28df-4dea-8e14-c5dfe2981ba5-original-pull-secret podName:de3a2ed4-28df-4dea-8e14-c5dfe2981ba5 nodeName:}" failed. No retries permitted until 2026-09-10 02:46:54.592845062 +0000 UTC m=+4.967930974 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de3a2ed4-28df-4dea-8e14-c5dfe2981ba5-original-pull-secret") pod "global-pull-secret-syncer-5qtg6" (UID: "de3a2ed4-28df-4dea-8e14-c5dfe2981ba5") : object "kube-system"/"original-pull-secret" not registered
Sep 10 02:46:52.693151 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:52.693115    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa599110-d0eb-4cc2-b4e9-e3984e9717fe-metrics-certs\") pod \"network-metrics-daemon-m5ph4\" (UID: \"aa599110-d0eb-4cc2-b4e9-e3984e9717fe\") " pod="openshift-multus/network-metrics-daemon-m5ph4"
Sep 10 02:46:52.693314 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:52.693251    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 02:46:52.693372 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:52.693318    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa599110-d0eb-4cc2-b4e9-e3984e9717fe-metrics-certs podName:aa599110-d0eb-4cc2-b4e9-e3984e9717fe nodeName:}" failed. No retries permitted until 2026-09-10 02:46:54.693299614 +0000 UTC m=+5.068385518 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aa599110-d0eb-4cc2-b4e9-e3984e9717fe-metrics-certs") pod "network-metrics-daemon-m5ph4" (UID: "aa599110-d0eb-4cc2-b4e9-e3984e9717fe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 02:46:52.778115 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:52.778068    2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 02:46:52.793989 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:52.793958    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96gxm\" (UniqueName: \"kubernetes.io/projected/e7d69b54-6b09-4496-abe6-6da05c77e958-kube-api-access-96gxm\") pod \"network-check-target-7xn49\" (UID: \"e7d69b54-6b09-4496-abe6-6da05c77e958\") " pod="openshift-network-diagnostics/network-check-target-7xn49"
Sep 10 02:46:52.794149 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:52.794098    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 02:46:52.794149 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:52.794139    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 02:46:52.794256 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:52.794150    2558 projected.go:196] Error preparing data for projected volume kube-api-access-96gxm for pod openshift-network-diagnostics/network-check-target-7xn49: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 02:46:52.794256 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:52.794213    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7d69b54-6b09-4496-abe6-6da05c77e958-kube-api-access-96gxm podName:e7d69b54-6b09-4496-abe6-6da05c77e958 nodeName:}" failed. No retries permitted until 2026-09-10 02:46:54.794184449 +0000 UTC m=+5.169270349 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-96gxm" (UniqueName: "kubernetes.io/projected/e7d69b54-6b09-4496-abe6-6da05c77e958-kube-api-access-96gxm") pod "network-check-target-7xn49" (UID: "e7d69b54-6b09-4496-abe6-6da05c77e958") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 02:46:53.014728 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:53.014647    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 02:41:51 +0000 UTC" deadline="2028-03-17 18:45:57.044483844 +0000 UTC"
Sep 10 02:46:53.014728 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:53.014681    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13311h59m4.029806562s"
Sep 10 02:46:53.062288 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:53.062223    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xn49"
Sep 10 02:46:53.062439 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:53.062339    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xn49" podUID="e7d69b54-6b09-4496-abe6-6da05c77e958"
Sep 10 02:46:53.062758 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:53.062732    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5ph4"
Sep 10 02:46:53.062869 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:53.062849    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5ph4" podUID="aa599110-d0eb-4cc2-b4e9-e3984e9717fe"
Sep 10 02:46:54.065486 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:54.064900    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:46:54.065486 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:54.065030    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5qtg6" podUID="de3a2ed4-28df-4dea-8e14-c5dfe2981ba5"
Sep 10 02:46:54.609090 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:54.608318    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de3a2ed4-28df-4dea-8e14-c5dfe2981ba5-original-pull-secret\") pod \"global-pull-secret-syncer-5qtg6\" (UID: \"de3a2ed4-28df-4dea-8e14-c5dfe2981ba5\") " pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:46:54.609090 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:54.608679    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 02:46:54.609090 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:54.608739    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de3a2ed4-28df-4dea-8e14-c5dfe2981ba5-original-pull-secret podName:de3a2ed4-28df-4dea-8e14-c5dfe2981ba5 nodeName:}" failed. No retries permitted until 2026-09-10 02:46:58.608721055 +0000 UTC m=+8.983806971 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de3a2ed4-28df-4dea-8e14-c5dfe2981ba5-original-pull-secret") pod "global-pull-secret-syncer-5qtg6" (UID: "de3a2ed4-28df-4dea-8e14-c5dfe2981ba5") : object "kube-system"/"original-pull-secret" not registered
Sep 10 02:46:54.709479 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:54.709447    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa599110-d0eb-4cc2-b4e9-e3984e9717fe-metrics-certs\") pod \"network-metrics-daemon-m5ph4\" (UID: \"aa599110-d0eb-4cc2-b4e9-e3984e9717fe\") " pod="openshift-multus/network-metrics-daemon-m5ph4"
Sep 10 02:46:54.709647 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:54.709561    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 02:46:54.709647 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:54.709636    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa599110-d0eb-4cc2-b4e9-e3984e9717fe-metrics-certs podName:aa599110-d0eb-4cc2-b4e9-e3984e9717fe nodeName:}" failed. No retries permitted until 2026-09-10 02:46:58.709616415 +0000 UTC m=+9.084702319 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aa599110-d0eb-4cc2-b4e9-e3984e9717fe-metrics-certs") pod "network-metrics-daemon-m5ph4" (UID: "aa599110-d0eb-4cc2-b4e9-e3984e9717fe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 02:46:54.810384 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:54.810311    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96gxm\" (UniqueName: \"kubernetes.io/projected/e7d69b54-6b09-4496-abe6-6da05c77e958-kube-api-access-96gxm\") pod \"network-check-target-7xn49\" (UID: \"e7d69b54-6b09-4496-abe6-6da05c77e958\") " pod="openshift-network-diagnostics/network-check-target-7xn49"
Sep 10 02:46:54.810535 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:54.810462    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 02:46:54.810535 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:54.810485    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 02:46:54.810535 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:54.810497    2558 projected.go:196] Error preparing data for projected volume kube-api-access-96gxm for pod openshift-network-diagnostics/network-check-target-7xn49: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 02:46:54.810691 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:54.810549    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7d69b54-6b09-4496-abe6-6da05c77e958-kube-api-access-96gxm podName:e7d69b54-6b09-4496-abe6-6da05c77e958 nodeName:}" failed. No retries permitted until 2026-09-10 02:46:58.810532417 +0000 UTC m=+9.185618328 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-96gxm" (UniqueName: "kubernetes.io/projected/e7d69b54-6b09-4496-abe6-6da05c77e958-kube-api-access-96gxm") pod "network-check-target-7xn49" (UID: "e7d69b54-6b09-4496-abe6-6da05c77e958") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 02:46:55.063157 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:55.062500    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xn49"
Sep 10 02:46:55.063157 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:55.062620    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xn49" podUID="e7d69b54-6b09-4496-abe6-6da05c77e958"
Sep 10 02:46:55.063157 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:55.062682    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5ph4"
Sep 10 02:46:55.063157 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:55.062767    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5ph4" podUID="aa599110-d0eb-4cc2-b4e9-e3984e9717fe"
Sep 10 02:46:56.062623 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:56.062587    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:46:56.063077 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:56.062724    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5qtg6" podUID="de3a2ed4-28df-4dea-8e14-c5dfe2981ba5"
Sep 10 02:46:57.063242 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:57.062846    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xn49"
Sep 10 02:46:57.063242 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:57.062852    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5ph4"
Sep 10 02:46:57.063242 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:57.062969    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xn49" podUID="e7d69b54-6b09-4496-abe6-6da05c77e958"
Sep 10 02:46:57.063242 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:57.063207    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5ph4" podUID="aa599110-d0eb-4cc2-b4e9-e3984e9717fe"
Sep 10 02:46:58.063226 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:58.063192    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:46:58.063403 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:58.063319    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5qtg6" podUID="de3a2ed4-28df-4dea-8e14-c5dfe2981ba5"
Sep 10 02:46:58.643598 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:58.643561    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de3a2ed4-28df-4dea-8e14-c5dfe2981ba5-original-pull-secret\") pod \"global-pull-secret-syncer-5qtg6\" (UID: \"de3a2ed4-28df-4dea-8e14-c5dfe2981ba5\") " pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:46:58.643801 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:58.643686    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 02:46:58.643801 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:58.643762    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de3a2ed4-28df-4dea-8e14-c5dfe2981ba5-original-pull-secret podName:de3a2ed4-28df-4dea-8e14-c5dfe2981ba5 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:06.643743579 +0000 UTC m=+17.018829495 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de3a2ed4-28df-4dea-8e14-c5dfe2981ba5-original-pull-secret") pod "global-pull-secret-syncer-5qtg6" (UID: "de3a2ed4-28df-4dea-8e14-c5dfe2981ba5") : object "kube-system"/"original-pull-secret" not registered
Sep 10 02:46:58.744461 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:58.744121    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa599110-d0eb-4cc2-b4e9-e3984e9717fe-metrics-certs\") pod \"network-metrics-daemon-m5ph4\" (UID: \"aa599110-d0eb-4cc2-b4e9-e3984e9717fe\") " pod="openshift-multus/network-metrics-daemon-m5ph4"
Sep 10 02:46:58.744461 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:58.744260    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 02:46:58.744461 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:58.744324    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa599110-d0eb-4cc2-b4e9-e3984e9717fe-metrics-certs podName:aa599110-d0eb-4cc2-b4e9-e3984e9717fe nodeName:}" failed. No retries permitted until 2026-09-10 02:47:06.744304198 +0000 UTC m=+17.119390112 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aa599110-d0eb-4cc2-b4e9-e3984e9717fe-metrics-certs") pod "network-metrics-daemon-m5ph4" (UID: "aa599110-d0eb-4cc2-b4e9-e3984e9717fe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 02:46:58.844522 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:58.844445    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96gxm\" (UniqueName: \"kubernetes.io/projected/e7d69b54-6b09-4496-abe6-6da05c77e958-kube-api-access-96gxm\") pod \"network-check-target-7xn49\" (UID: \"e7d69b54-6b09-4496-abe6-6da05c77e958\") " pod="openshift-network-diagnostics/network-check-target-7xn49"
Sep 10 02:46:58.844688 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:58.844590    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 02:46:58.844688 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:58.844616    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 02:46:58.844688 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:58.844638    2558 projected.go:196] Error preparing data for projected volume kube-api-access-96gxm for pod openshift-network-diagnostics/network-check-target-7xn49: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 02:46:58.844857 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:58.844700    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7d69b54-6b09-4496-abe6-6da05c77e958-kube-api-access-96gxm podName:e7d69b54-6b09-4496-abe6-6da05c77e958 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:06.844682279 +0000 UTC m=+17.219768179 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-96gxm" (UniqueName: "kubernetes.io/projected/e7d69b54-6b09-4496-abe6-6da05c77e958-kube-api-access-96gxm") pod "network-check-target-7xn49" (UID: "e7d69b54-6b09-4496-abe6-6da05c77e958") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 02:46:59.063027 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:59.062944    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5ph4"
Sep 10 02:46:59.063027 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:46:59.062979    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xn49"
Sep 10 02:46:59.063256 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:59.063085    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5ph4" podUID="aa599110-d0eb-4cc2-b4e9-e3984e9717fe"
Sep 10 02:46:59.063256 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:46:59.063205    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xn49" podUID="e7d69b54-6b09-4496-abe6-6da05c77e958"
Sep 10 02:47:00.064445 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:00.064056    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:47:00.064445 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:00.064191    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5qtg6" podUID="de3a2ed4-28df-4dea-8e14-c5dfe2981ba5"
Sep 10 02:47:01.062512 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:01.062484    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xn49"
Sep 10 02:47:01.062512 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:01.062509    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5ph4"
Sep 10 02:47:01.062714 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:01.062595    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xn49" podUID="e7d69b54-6b09-4496-abe6-6da05c77e958"
Sep 10 02:47:01.062757 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:01.062736    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5ph4" podUID="aa599110-d0eb-4cc2-b4e9-e3984e9717fe"
Sep 10 02:47:02.062701 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:02.062669    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:47:02.063269 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:02.062801    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5qtg6" podUID="de3a2ed4-28df-4dea-8e14-c5dfe2981ba5"
Sep 10 02:47:03.062964 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:03.062930    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xn49"
Sep 10 02:47:03.062964 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:03.062977    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5ph4"
Sep 10 02:47:03.063460 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:03.063403    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xn49" podUID="e7d69b54-6b09-4496-abe6-6da05c77e958"
Sep 10 02:47:03.063699 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:03.063667    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5ph4" podUID="aa599110-d0eb-4cc2-b4e9-e3984e9717fe"
Sep 10 02:47:04.062446 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:04.062407    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:47:04.062615 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:04.062535    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5qtg6" podUID="de3a2ed4-28df-4dea-8e14-c5dfe2981ba5"
Sep 10 02:47:05.062847 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:05.062812    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xn49"
Sep 10 02:47:05.062847 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:05.062847    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5ph4"
Sep 10 02:47:05.063302 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:05.062931    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xn49" podUID="e7d69b54-6b09-4496-abe6-6da05c77e958"
Sep 10 02:47:05.063302 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:05.063075    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5ph4" podUID="aa599110-d0eb-4cc2-b4e9-e3984e9717fe"
Sep 10 02:47:06.062963 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:06.062928    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:47:06.063375 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:06.063040    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5qtg6" podUID="de3a2ed4-28df-4dea-8e14-c5dfe2981ba5"
Sep 10 02:47:06.708236 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:06.708204    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de3a2ed4-28df-4dea-8e14-c5dfe2981ba5-original-pull-secret\") pod \"global-pull-secret-syncer-5qtg6\" (UID: \"de3a2ed4-28df-4dea-8e14-c5dfe2981ba5\") " pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:47:06.708455 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:06.708361    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 02:47:06.708455 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:06.708419    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de3a2ed4-28df-4dea-8e14-c5dfe2981ba5-original-pull-secret podName:de3a2ed4-28df-4dea-8e14-c5dfe2981ba5 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:22.708405075 +0000 UTC m=+33.083490978 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de3a2ed4-28df-4dea-8e14-c5dfe2981ba5-original-pull-secret") pod "global-pull-secret-syncer-5qtg6" (UID: "de3a2ed4-28df-4dea-8e14-c5dfe2981ba5") : object "kube-system"/"original-pull-secret" not registered
Sep 10 02:47:06.809325 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:06.809295    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa599110-d0eb-4cc2-b4e9-e3984e9717fe-metrics-certs\") pod \"network-metrics-daemon-m5ph4\" (UID: \"aa599110-d0eb-4cc2-b4e9-e3984e9717fe\") " pod="openshift-multus/network-metrics-daemon-m5ph4"
Sep 10 02:47:06.809481 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:06.809467    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 02:47:06.809535 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:06.809526    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa599110-d0eb-4cc2-b4e9-e3984e9717fe-metrics-certs podName:aa599110-d0eb-4cc2-b4e9-e3984e9717fe nodeName:}" failed. No retries permitted until 2026-09-10 02:47:22.809506534 +0000 UTC m=+33.184592443 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aa599110-d0eb-4cc2-b4e9-e3984e9717fe-metrics-certs") pod "network-metrics-daemon-m5ph4" (UID: "aa599110-d0eb-4cc2-b4e9-e3984e9717fe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 02:47:06.910696 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:06.910667    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96gxm\" (UniqueName: \"kubernetes.io/projected/e7d69b54-6b09-4496-abe6-6da05c77e958-kube-api-access-96gxm\") pod \"network-check-target-7xn49\" (UID: \"e7d69b54-6b09-4496-abe6-6da05c77e958\") " pod="openshift-network-diagnostics/network-check-target-7xn49"
Sep 10 02:47:06.910831 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:06.910821    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 02:47:06.910899 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:06.910844    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 02:47:06.910899 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:06.910856    2558 projected.go:196] Error preparing data for projected volume kube-api-access-96gxm for pod openshift-network-diagnostics/network-check-target-7xn49: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 02:47:06.910991 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:06.910912    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7d69b54-6b09-4496-abe6-6da05c77e958-kube-api-access-96gxm podName:e7d69b54-6b09-4496-abe6-6da05c77e958 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:22.910893447 +0000 UTC m=+33.285979350 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-96gxm" (UniqueName: "kubernetes.io/projected/e7d69b54-6b09-4496-abe6-6da05c77e958-kube-api-access-96gxm") pod "network-check-target-7xn49" (UID: "e7d69b54-6b09-4496-abe6-6da05c77e958") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 02:47:07.062922 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:07.062851    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xn49"
Sep 10 02:47:07.063065 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:07.062851    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5ph4"
Sep 10 02:47:07.063065 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:07.062950    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xn49" podUID="e7d69b54-6b09-4496-abe6-6da05c77e958"
Sep 10 02:47:07.063065 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:07.063054    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5ph4" podUID="aa599110-d0eb-4cc2-b4e9-e3984e9717fe"
Sep 10 02:47:08.062224 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:08.062193    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:47:08.062400 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:08.062301    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5qtg6" podUID="de3a2ed4-28df-4dea-8e14-c5dfe2981ba5"
Sep 10 02:47:09.063168 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:09.063138    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5ph4"
Sep 10 02:47:09.063168 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:09.063177    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xn49"
Sep 10 02:47:09.063603 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:09.063260    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5ph4" podUID="aa599110-d0eb-4cc2-b4e9-e3984e9717fe"
Sep 10 02:47:09.063603 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:09.063404    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xn49" podUID="e7d69b54-6b09-4496-abe6-6da05c77e958"
Sep 10 02:47:10.065868 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:10.065840    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:47:10.069491 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:10.068880    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5qtg6" podUID="de3a2ed4-28df-4dea-8e14-c5dfe2981ba5"
Sep 10 02:47:10.263698 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:10.263660    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pptgm" event={"ID":"bf5695bb-3571-4150-b792-ab3b5d1283c6","Type":"ContainerStarted","Data":"00f42be3995fad447f27bb44087e5a8d0e2d95421e255d36c5b8e79c1eb5c8f6"}
Sep 10 02:47:10.265392 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:10.265334    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qhf58" event={"ID":"c31a9e4e-75c0-4f5a-87c2-d323be62dfa5","Type":"ContainerStarted","Data":"2c0bc9ae75034939641e9e0530f63fede6496b6b942e136ffd3cd60a1a22f904"}
Sep 10 02:47:10.268303 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:10.268129    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bltxf" event={"ID":"ba9d4732-9d25-4549-98de-12a27aee412f","Type":"ContainerStarted","Data":"356bd216e018aa3dfdbf1493ab0b283b3771984161f6c9aa61f5b15645df387c"}
Sep 10 02:47:10.268303 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:10.268206    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bltxf" event={"ID":"ba9d4732-9d25-4549-98de-12a27aee412f","Type":"ContainerStarted","Data":"902b36e669f0ababe4aa0a78127b2a92e3ac8206815fcff25465192f8ff69f50"}
Sep 10 02:47:10.268303 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:10.268253    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bltxf" event={"ID":"ba9d4732-9d25-4549-98de-12a27aee412f","Type":"ContainerStarted","Data":"8f5efc63d82b24107adbc006974f50fdc197ee311c9874a0a8d2a91f8d2b095c"}
Sep 10 02:47:10.272342 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:10.271732    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-101.ec2.internal" event={"ID":"01f46892139817c71935091c0eabf5f5","Type":"ContainerStarted","Data":"17d2838edd391a734d7a6504733b5fe237b90bfda5c6db73cb0de32658383073"}
Sep 10 02:47:10.279145 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:10.279087    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-pptgm" podStartSLOduration=1.66181947 podStartE2EDuration="20.279074907s" totalImagesPullingTime="18.617255437s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:10.278729311 +0000 UTC m=+20.653815232" watchObservedRunningTime="2026-09-10 02:47:10.279074907 +0000 UTC m=+20.654160828"
Sep 10 02:47:10.292964 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:10.292926    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qhf58" podStartSLOduration=1.781760943 podStartE2EDuration="20.292913022s" totalImagesPullingTime="18.511152079s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:10.292477484 +0000 UTC m=+20.667563409" watchObservedRunningTime="2026-09-10 02:47:10.292913022 +0000 UTC m=+20.667998944"
Sep 10 02:47:10.306998 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:10.306039    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-101.ec2.internal" podStartSLOduration=20.306000242 podStartE2EDuration="20.306000242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:10.30543602 +0000 UTC m=+20.680521942" watchObservedRunningTime="2026-09-10 02:47:10.306000242 +0000 UTC m=+20.681086156"
Sep 10 02:47:11.062608 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:11.062441    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xn49"
Sep 10 02:47:11.062722 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:11.062504    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5ph4"
Sep 10 02:47:11.062722 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:11.062673    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xn49" podUID="e7d69b54-6b09-4496-abe6-6da05c77e958"
Sep 10 02:47:11.062794 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:11.062752    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5ph4" podUID="aa599110-d0eb-4cc2-b4e9-e3984e9717fe"
Sep 10 02:47:11.275654 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:11.275587    2558 generic.go:356] "Generic (PLEG): container finished" podID="62603892-88ce-474d-8d32-3b085dbbedd7" containerID="40a55cb5f13d7df640bb1809cbcbc7e80557e8dafe320f605d3312865c2a78ed" exitCode=0
Sep 10 02:47:11.276258 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:11.275662    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-btm4f" event={"ID":"62603892-88ce-474d-8d32-3b085dbbedd7","Type":"ContainerDied","Data":"40a55cb5f13d7df640bb1809cbcbc7e80557e8dafe320f605d3312865c2a78ed"}
Sep 10 02:47:11.277346 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:11.277323    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rl4nr" event={"ID":"f90981c4-23c1-4950-8786-2f4bb0d6e3fb","Type":"ContainerStarted","Data":"d9525e9feafbd3c479f04498b2dad932f8fef878800845f34c38aaf504105aeb"}
Sep 10 02:47:11.278735 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:11.278703    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfz5c" event={"ID":"494fc5ce-7947-4179-ae56-903c34b716b5","Type":"ContainerStarted","Data":"e0391dc5268508eb3e013d20a75e018be638a2cfa3bb4c2f0f6697c94ce8ed18"}
Sep 10 02:47:11.280275 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:11.280251    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4xpxz" event={"ID":"c68ab7ea-a147-44d5-81a1-91eca0da2366","Type":"ContainerStarted","Data":"adcb174b5c4b687928fdbf8d4ec0ae7d0c68a45cd1a61d7cc48cb0fcd89fd1de"}
Sep 10 02:47:11.281823 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:11.281801    2558 generic.go:356] "Generic (PLEG): container finished" podID="d3279872c23798b7320a2d8a5e9247b5" containerID="013d165db620f9589949ab0cf959977bdc87659176297eea5932d09d508727a4" exitCode=0
Sep 10 02:47:11.281900 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:11.281869    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal" event={"ID":"d3279872c23798b7320a2d8a5e9247b5","Type":"ContainerDied","Data":"013d165db620f9589949ab0cf959977bdc87659176297eea5932d09d508727a4"}
Sep 10 02:47:11.283717 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:11.283691    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-296rx" event={"ID":"82b3bbcd-1ae2-4bbc-88a7-61aff07f409a","Type":"ContainerStarted","Data":"771c5096cddf7fecced711740c9a2525db8529765afdd69c148349d51dc72cd4"}
Sep 10 02:47:11.286498 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:11.286469    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bltxf" event={"ID":"ba9d4732-9d25-4549-98de-12a27aee412f","Type":"ContainerStarted","Data":"461ad28a124e9581ddc2adb5d1244a2131b8e6acf2608c301e7e85d9ee0a7910"}
Sep 10 02:47:11.286589 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:11.286528    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bltxf" event={"ID":"ba9d4732-9d25-4549-98de-12a27aee412f","Type":"ContainerStarted","Data":"a3c9f990fe3855991ac1904a0718bacfc54e47db9302c7f84397710fb9db611c"}
Sep 10 02:47:11.286589 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:11.286560    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bltxf" event={"ID":"ba9d4732-9d25-4549-98de-12a27aee412f","Type":"ContainerStarted","Data":"f413957a66344c01adfeb202ad3f25514d798555e2b4bd4c9c940ceb033bed3c"}
Sep 10 02:47:11.325725 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:11.325680    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4xpxz" podStartSLOduration=2.75177958 podStartE2EDuration="21.325668379s" totalImagesPullingTime="18.573888799s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:11.310678449 +0000 UTC m=+21.685764370" watchObservedRunningTime="2026-09-10 02:47:11.325668379 +0000 UTC m=+21.700754282"
Sep 10 02:47:11.326087 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:11.326066    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rl4nr" podStartSLOduration=2.834182978 podStartE2EDuration="21.326059247s" totalImagesPullingTime="18.491876269s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:11.325723135 +0000 UTC m=+21.700809055" watchObservedRunningTime="2026-09-10 02:47:11.326059247 +0000 UTC m=+21.701145168"
Sep 10 02:47:11.361151 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:11.361119    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-296rx" podStartSLOduration=2.849775865 podStartE2EDuration="21.361091359s" totalImagesPullingTime="18.511315494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:11.360742773 +0000 UTC m=+21.735828694" watchObservedRunningTime="2026-09-10 02:47:11.361091359 +0000 UTC m=+21.736177279"
Sep 10 02:47:11.409399 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:11.409378    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:11.484648 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:11.484630    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:11.766986 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:11.766955    2558 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 02:47:12.042031 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:12.041946    2558 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T02:47:11.766981677Z","UUID":"3d7f1a82-8699-4af1-9c68-fc8de9823873","Handler":null,"Name":"","Endpoint":""}
Sep 10 02:47:12.043512 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:12.043489    2558 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 02:47:12.043633 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:12.043524    2558 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 02:47:12.062749 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:12.062728    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:47:12.062865 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:12.062844    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5qtg6" podUID="de3a2ed4-28df-4dea-8e14-c5dfe2981ba5"
Sep 10 02:47:12.291359 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:12.291325    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9j9fc" event={"ID":"3ab870ac-9723-4beb-b2d1-c043dd831bae","Type":"ContainerStarted","Data":"e1e405bddd8f7a30ee3cad213c3246fdad9454241fecd40805fbaf52dbeb4d8d"}
Sep 10 02:47:12.294198 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:12.294138    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfz5c" event={"ID":"494fc5ce-7947-4179-ae56-903c34b716b5","Type":"ContainerStarted","Data":"de888dec10e12e5f72fca3985e6b48df1747d226055ed546e0fe1be56e7acb1a"}
Sep 10 02:47:12.296790 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:12.296764    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal" event={"ID":"d3279872c23798b7320a2d8a5e9247b5","Type":"ContainerStarted","Data":"0de8d366784ee9789f260c29759c7c0f7c0ee1669d3c2d83015ecd1299dba549"}
Sep 10 02:47:12.305544 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:12.305497    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9j9fc" podStartSLOduration=3.74093676 podStartE2EDuration="22.305481037s" totalImagesPullingTime="18.564544277s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:12.305225153 +0000 UTC m=+22.680311090" watchObservedRunningTime="2026-09-10 02:47:12.305481037 +0000 UTC m=+22.680566958"
Sep 10 02:47:12.324676 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:12.324624    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-101.ec2.internal" podStartSLOduration=22.32460724 podStartE2EDuration="22.32460724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:12.32460078 +0000 UTC m=+22.699686702" watchObservedRunningTime="2026-09-10 02:47:12.32460724 +0000 UTC m=+22.699693163"
Sep 10 02:47:12.527343 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:12.527098    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:12.589556 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:12.589480    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:13.062637 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:13.062609    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xn49"
Sep 10 02:47:13.062760 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:13.062718    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xn49" podUID="e7d69b54-6b09-4496-abe6-6da05c77e958"
Sep 10 02:47:13.062826 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:13.062769    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5ph4"
Sep 10 02:47:13.062878 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:13.062863    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5ph4" podUID="aa599110-d0eb-4cc2-b4e9-e3984e9717fe"
Sep 10 02:47:13.302914 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:13.302833    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bltxf" event={"ID":"ba9d4732-9d25-4549-98de-12a27aee412f","Type":"ContainerStarted","Data":"7a741d9b4df4965ce28bdcdfbc8fc47b114e52dfdebfc706543bc774d2272251"}
Sep 10 02:47:13.305361 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:13.305330    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfz5c" event={"ID":"494fc5ce-7947-4179-ae56-903c34b716b5","Type":"ContainerStarted","Data":"4722c84a96cd8c42f5c5d636c8041b97fab26a119b1c96a6186f837976d7a862"}
Sep 10 02:47:13.321825 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:13.321784    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qfz5c" podStartSLOduration=2.424485964 podStartE2EDuration="23.321767287s" totalImagesPullingTime="20.897281323s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:46:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:13.321242741 +0000 UTC m=+23.696328662" watchObservedRunningTime="2026-09-10 02:47:13.321767287 +0000 UTC m=+23.696853212"
Sep 10 02:47:14.063116 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:14.063074    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:47:14.063337 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:14.063223    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5qtg6" podUID="de3a2ed4-28df-4dea-8e14-c5dfe2981ba5"
Sep 10 02:47:15.062403 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:15.062326    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xn49"
Sep 10 02:47:15.062826 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:15.062435    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xn49" podUID="e7d69b54-6b09-4496-abe6-6da05c77e958"
Sep 10 02:47:15.062826 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:15.062500    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5ph4"
Sep 10 02:47:15.062826 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:15.062626    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5ph4" podUID="aa599110-d0eb-4cc2-b4e9-e3984e9717fe"
Sep 10 02:47:15.474158 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:15.474128    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-296rx"
Sep 10 02:47:15.474725 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:15.474703    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-296rx"
Sep 10 02:47:16.063312 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:16.063169    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:47:16.064313 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:16.064275    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5qtg6" podUID="de3a2ed4-28df-4dea-8e14-c5dfe2981ba5"
Sep 10 02:47:16.313400 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:16.313342    2558 generic.go:356] "Generic (PLEG): container finished" podID="62603892-88ce-474d-8d32-3b085dbbedd7" containerID="c2231979723497e614322412d4fa253b1e88f580307938db71b87bc18f9254ed" exitCode=0
Sep 10 02:47:16.313506 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:16.313433    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-btm4f" event={"ID":"62603892-88ce-474d-8d32-3b085dbbedd7","Type":"ContainerDied","Data":"c2231979723497e614322412d4fa253b1e88f580307938db71b87bc18f9254ed"}
Sep 10 02:47:16.317391 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:16.317365    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bltxf" event={"ID":"ba9d4732-9d25-4549-98de-12a27aee412f","Type":"ContainerStarted","Data":"be6b6e066dfdddd72b6cef218ea9a2e2f6ae131f5ca0f2e73bf7613a9c4af218"}
Sep 10 02:47:16.317580 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:16.317557    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-296rx"
Sep 10 02:47:16.317951 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:16.317937    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-296rx"
Sep 10 02:47:16.353910 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:16.353871    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bltxf" podStartSLOduration=7.739815745 podStartE2EDuration="26.353859308s" totalImagesPullingTime="18.614043563s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:16.35352211 +0000 UTC m=+26.728608034" watchObservedRunningTime="2026-09-10 02:47:16.353859308 +0000 UTC m=+26.728945226"
Sep 10 02:47:16.412249 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:16.412229    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:16.701976 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:16.701950    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:16.786790 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:16.786763    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:16.872635 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:16.872611    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:16.973956 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:16.973924    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:17.062213 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:17.062189    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5ph4"
Sep 10 02:47:17.062338 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:17.062189    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xn49"
Sep 10 02:47:17.062338 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:17.062318    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5ph4" podUID="aa599110-d0eb-4cc2-b4e9-e3984e9717fe"
Sep 10 02:47:17.062458 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:17.062353    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xn49" podUID="e7d69b54-6b09-4496-abe6-6da05c77e958"
Sep 10 02:47:17.321945 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:17.321868    2558 generic.go:356] "Generic (PLEG): container finished" podID="62603892-88ce-474d-8d32-3b085dbbedd7" containerID="46496d88bfb41534ec8ed7292ddbb868111447a94c55595cfaa8cb9bfba51899" exitCode=0
Sep 10 02:47:17.321945 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:17.321930    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-btm4f" event={"ID":"62603892-88ce-474d-8d32-3b085dbbedd7","Type":"ContainerDied","Data":"46496d88bfb41534ec8ed7292ddbb868111447a94c55595cfaa8cb9bfba51899"}
Sep 10 02:47:17.322867 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:17.322743    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:47:17.322867 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:17.322808    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:47:17.322867 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:17.322858    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:47:17.337997 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:17.337974    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:47:17.338258 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:17.338242    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:47:17.664726 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:17.664658    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5qtg6"]
Sep 10 02:47:17.664827 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:17.664791    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:47:17.664888 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:17.664869    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5qtg6" podUID="de3a2ed4-28df-4dea-8e14-c5dfe2981ba5"
Sep 10 02:47:17.668005 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:17.667979    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7xn49"]
Sep 10 02:47:17.668138 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:17.668078    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xn49"
Sep 10 02:47:17.668199 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:17.668184    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xn49" podUID="e7d69b54-6b09-4496-abe6-6da05c77e958"
Sep 10 02:47:17.668654 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:17.668622    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m5ph4"]
Sep 10 02:47:17.668735 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:17.668721    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5ph4"
Sep 10 02:47:17.668838 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:17.668820    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5ph4" podUID="aa599110-d0eb-4cc2-b4e9-e3984e9717fe"
Sep 10 02:47:18.326617 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:18.326416    2558 generic.go:356] "Generic (PLEG): container finished" podID="62603892-88ce-474d-8d32-3b085dbbedd7" containerID="735cc5f3853d25aa2783562456f456464c2be51e9fc6b1935dd209031429a3d6" exitCode=0
Sep 10 02:47:18.327126 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:18.326481    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-btm4f" event={"ID":"62603892-88ce-474d-8d32-3b085dbbedd7","Type":"ContainerDied","Data":"735cc5f3853d25aa2783562456f456464c2be51e9fc6b1935dd209031429a3d6"}
Sep 10 02:47:18.336811 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:18.336793    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rl4nr_f90981c4-23c1-4950-8786-2f4bb0d6e3fb/dns-node-resolver/0.log"
Sep 10 02:47:18.554489 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:18.554464    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:18.617297 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:18.617230    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:18.938795 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:18.938771    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4xpxz_c68ab7ea-a147-44d5-81a1-91eca0da2366/node-ca/0.log"
Sep 10 02:47:19.063194 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:19.063166    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xn49"
Sep 10 02:47:19.063352 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:19.063167    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:47:19.063352 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:19.063290    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xn49" podUID="e7d69b54-6b09-4496-abe6-6da05c77e958"
Sep 10 02:47:19.063469 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:19.063174    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5ph4"
Sep 10 02:47:19.063469 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:19.063362    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5qtg6" podUID="de3a2ed4-28df-4dea-8e14-c5dfe2981ba5"
Sep 10 02:47:19.063469 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:19.063460    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5ph4" podUID="aa599110-d0eb-4cc2-b4e9-e3984e9717fe"
Sep 10 02:47:19.674933 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:19.674847    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:19.774774 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:19.774734    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:20.609902 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:20.609868    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:20.681939 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:20.681908    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:21.063026 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:21.062997    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xn49"
Sep 10 02:47:21.063211 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:21.063032    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5ph4"
Sep 10 02:47:21.063211 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:21.063092    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:47:21.063211 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:21.063125    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xn49" podUID="e7d69b54-6b09-4496-abe6-6da05c77e958"
Sep 10 02:47:21.063211 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:21.063177    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5qtg6" podUID="de3a2ed4-28df-4dea-8e14-c5dfe2981ba5"
Sep 10 02:47:21.063432 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:21.063262    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5ph4" podUID="aa599110-d0eb-4cc2-b4e9-e3984e9717fe"
Sep 10 02:47:21.539534 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:21.539504    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:21.639795 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:21.639765    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:21.695004 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:21.694972    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:21.973545 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:21.973514    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:22.069081 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:22.069048    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:22.169875 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:22.169849    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:22.288044 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:22.287962    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:22.368873 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:22.368849    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:22.655664 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:22.655611    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:22.716367 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:22.716348    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:22.725401 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:22.725377    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de3a2ed4-28df-4dea-8e14-c5dfe2981ba5-original-pull-secret\") pod \"global-pull-secret-syncer-5qtg6\" (UID: \"de3a2ed4-28df-4dea-8e14-c5dfe2981ba5\") " pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:47:22.725495 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:22.725483    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 02:47:22.725537 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:22.725529    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de3a2ed4-28df-4dea-8e14-c5dfe2981ba5-original-pull-secret podName:de3a2ed4-28df-4dea-8e14-c5dfe2981ba5 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:54.725516901 +0000 UTC m=+65.100602799 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de3a2ed4-28df-4dea-8e14-c5dfe2981ba5-original-pull-secret") pod "global-pull-secret-syncer-5qtg6" (UID: "de3a2ed4-28df-4dea-8e14-c5dfe2981ba5") : object "kube-system"/"original-pull-secret" not registered
Sep 10 02:47:22.825944 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:22.825923    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa599110-d0eb-4cc2-b4e9-e3984e9717fe-metrics-certs\") pod \"network-metrics-daemon-m5ph4\" (UID: \"aa599110-d0eb-4cc2-b4e9-e3984e9717fe\") " pod="openshift-multus/network-metrics-daemon-m5ph4"
Sep 10 02:47:22.826031 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:22.826016    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 02:47:22.826070 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:22.826060    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa599110-d0eb-4cc2-b4e9-e3984e9717fe-metrics-certs podName:aa599110-d0eb-4cc2-b4e9-e3984e9717fe nodeName:}" failed. No retries permitted until 2026-09-10 02:47:54.826047866 +0000 UTC m=+65.201133765 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aa599110-d0eb-4cc2-b4e9-e3984e9717fe-metrics-certs") pod "network-metrics-daemon-m5ph4" (UID: "aa599110-d0eb-4cc2-b4e9-e3984e9717fe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 02:47:22.926821 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:22.926752    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96gxm\" (UniqueName: \"kubernetes.io/projected/e7d69b54-6b09-4496-abe6-6da05c77e958-kube-api-access-96gxm\") pod \"network-check-target-7xn49\" (UID: \"e7d69b54-6b09-4496-abe6-6da05c77e958\") " pod="openshift-network-diagnostics/network-check-target-7xn49"
Sep 10 02:47:22.926943 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:22.926923    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 02:47:22.926993 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:22.926949    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 02:47:22.926993 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:22.926964    2558 projected.go:196] Error preparing data for projected volume kube-api-access-96gxm for pod openshift-network-diagnostics/network-check-target-7xn49: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 02:47:22.927051 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:22.927021    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7d69b54-6b09-4496-abe6-6da05c77e958-kube-api-access-96gxm podName:e7d69b54-6b09-4496-abe6-6da05c77e958 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:54.927007105 +0000 UTC m=+65.302093004 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-96gxm" (UniqueName: "kubernetes.io/projected/e7d69b54-6b09-4496-abe6-6da05c77e958-kube-api-access-96gxm") pod "network-check-target-7xn49" (UID: "e7d69b54-6b09-4496-abe6-6da05c77e958") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 02:47:23.062304 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:23.062285    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:47:23.062403 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:23.062386    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5qtg6" podUID="de3a2ed4-28df-4dea-8e14-c5dfe2981ba5"
Sep 10 02:47:23.062487 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:23.062471    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5ph4"
Sep 10 02:47:23.062588 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:23.062572    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m5ph4" podUID="aa599110-d0eb-4cc2-b4e9-e3984e9717fe"
Sep 10 02:47:23.062624 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:23.062613    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xn49"
Sep 10 02:47:23.062673 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:23.062661    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7xn49" podUID="e7d69b54-6b09-4496-abe6-6da05c77e958"
Sep 10 02:47:23.343420 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:23.343387    2558 generic.go:356] "Generic (PLEG): container finished" podID="62603892-88ce-474d-8d32-3b085dbbedd7" containerID="939963304dcd02eb45835ec033cc6ee05858e876987af15545fb2b68f6e193e2" exitCode=0
Sep 10 02:47:23.343580 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:23.343432    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-btm4f" event={"ID":"62603892-88ce-474d-8d32-3b085dbbedd7","Type":"ContainerDied","Data":"939963304dcd02eb45835ec033cc6ee05858e876987af15545fb2b68f6e193e2"}
Sep 10 02:47:23.876453 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:23.876415    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:23.938064 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:23.938036    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:23.949727 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:23.949708    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-101.ec2.internal" event="NodeReady"
Sep 10 02:47:23.949812 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:23.949790    2558 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 02:47:23.993893 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:23.993874    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pwzr2"]
Sep 10 02:47:24.016742 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.016723    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qqd6h"]
Sep 10 02:47:24.016859 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.016840    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pwzr2"
Sep 10 02:47:24.018750 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.018731    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 02:47:24.019187 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.019170    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gq58s\""
Sep 10 02:47:24.019267 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.019192    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 02:47:24.036828 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.036803    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pwzr2"]
Sep 10 02:47:24.036930 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.036837    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pfnd8"]
Sep 10 02:47:24.036930 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.036914    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qqd6h"
Sep 10 02:47:24.039137 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.039117    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 02:47:24.039230 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.039214    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 02:47:24.039315 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.039301    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5sdzz\""
Sep 10 02:47:24.039396 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.039384    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 02:47:24.062283 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.062258    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qqd6h"]
Sep 10 02:47:24.062283 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.062286    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pfnd8"]
Sep 10 02:47:24.062399 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.062363    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pfnd8"
Sep 10 02:47:24.064342 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.064322    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 02:47:24.064440 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.064354    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 02:47:24.064440 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.064422    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fq77l\""
Sep 10 02:47:24.064583 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.064567    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 02:47:24.064663 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.064646    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 02:47:24.082298 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.082278    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:24.136347 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.136294    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/27053ebb-2737-4ae1-9aee-c36be0c9ec67-metrics-tls\") pod \"dns-default-pwzr2\" (UID: \"27053ebb-2737-4ae1-9aee-c36be0c9ec67\") " pod="openshift-dns/dns-default-pwzr2"
Sep 10 02:47:24.136347 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.136326    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cae06c98-aa57-4981-9318-77f97d4aeba6-cert\") pod \"ingress-canary-qqd6h\" (UID: \"cae06c98-aa57-4981-9318-77f97d4aeba6\") " pod="openshift-ingress-canary/ingress-canary-qqd6h"
Sep 10 02:47:24.136470 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.136351    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/27053ebb-2737-4ae1-9aee-c36be0c9ec67-config-volume\") pod \"dns-default-pwzr2\" (UID: \"27053ebb-2737-4ae1-9aee-c36be0c9ec67\") " pod="openshift-dns/dns-default-pwzr2"
Sep 10 02:47:24.136470 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.136395    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27053ebb-2737-4ae1-9aee-c36be0c9ec67-tmp-dir\") pod \"dns-default-pwzr2\" (UID: \"27053ebb-2737-4ae1-9aee-c36be0c9ec67\") " pod="openshift-dns/dns-default-pwzr2"
Sep 10 02:47:24.136470 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.136423    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpmjd\" (UniqueName: \"kubernetes.io/projected/cae06c98-aa57-4981-9318-77f97d4aeba6-kube-api-access-fpmjd\") pod \"ingress-canary-qqd6h\" (UID: \"cae06c98-aa57-4981-9318-77f97d4aeba6\") " pod="openshift-ingress-canary/ingress-canary-qqd6h"
Sep 10 02:47:24.136470 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.136463    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv7wz\" (UniqueName: \"kubernetes.io/projected/27053ebb-2737-4ae1-9aee-c36be0c9ec67-kube-api-access-bv7wz\") pod \"dns-default-pwzr2\" (UID: \"27053ebb-2737-4ae1-9aee-c36be0c9ec67\") " pod="openshift-dns/dns-default-pwzr2"
Sep 10 02:47:24.145270 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.145245    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:24.236882 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.236859    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dszzf\" (UniqueName: \"kubernetes.io/projected/82088748-d4dc-4c0b-939c-62b49e6d19d7-kube-api-access-dszzf\") pod \"insights-runtime-extractor-pfnd8\" (UID: \"82088748-d4dc-4c0b-939c-62b49e6d19d7\") " pod="openshift-insights/insights-runtime-extractor-pfnd8"
Sep 10 02:47:24.236968 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.236900    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/27053ebb-2737-4ae1-9aee-c36be0c9ec67-metrics-tls\") pod \"dns-default-pwzr2\" (UID: \"27053ebb-2737-4ae1-9aee-c36be0c9ec67\") " pod="openshift-dns/dns-default-pwzr2"
Sep 10 02:47:24.236968 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.236919    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82088748-d4dc-4c0b-939c-62b49e6d19d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pfnd8\" (UID: \"82088748-d4dc-4c0b-939c-62b49e6d19d7\") " pod="openshift-insights/insights-runtime-extractor-pfnd8"
Sep 10 02:47:24.237078 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:24.236985    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 02:47:24.237078 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.237000    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cae06c98-aa57-4981-9318-77f97d4aeba6-cert\") pod \"ingress-canary-qqd6h\" (UID: \"cae06c98-aa57-4981-9318-77f97d4aeba6\") " pod="openshift-ingress-canary/ingress-canary-qqd6h"
Sep 10 02:47:24.237078 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:24.237031    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27053ebb-2737-4ae1-9aee-c36be0c9ec67-metrics-tls podName:27053ebb-2737-4ae1-9aee-c36be0c9ec67 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:24.737019135 +0000 UTC m=+35.112105069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/27053ebb-2737-4ae1-9aee-c36be0c9ec67-metrics-tls") pod "dns-default-pwzr2" (UID: "27053ebb-2737-4ae1-9aee-c36be0c9ec67") : secret "dns-default-metrics-tls" not found
Sep 10 02:47:24.237078 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.237043    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/27053ebb-2737-4ae1-9aee-c36be0c9ec67-config-volume\") pod \"dns-default-pwzr2\" (UID: \"27053ebb-2737-4ae1-9aee-c36be0c9ec67\") " pod="openshift-dns/dns-default-pwzr2"
Sep 10 02:47:24.237078 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.237062    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27053ebb-2737-4ae1-9aee-c36be0c9ec67-tmp-dir\") pod \"dns-default-pwzr2\" (UID: \"27053ebb-2737-4ae1-9aee-c36be0c9ec67\") " pod="openshift-dns/dns-default-pwzr2"
Sep 10 02:47:24.237078 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.237078    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpmjd\" (UniqueName: \"kubernetes.io/projected/cae06c98-aa57-4981-9318-77f97d4aeba6-kube-api-access-fpmjd\") pod \"ingress-canary-qqd6h\" (UID: \"cae06c98-aa57-4981-9318-77f97d4aeba6\") " pod="openshift-ingress-canary/ingress-canary-qqd6h"
Sep 10 02:47:24.237379 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.237127    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv7wz\" (UniqueName: \"kubernetes.io/projected/27053ebb-2737-4ae1-9aee-c36be0c9ec67-kube-api-access-bv7wz\") pod \"dns-default-pwzr2\" (UID: \"27053ebb-2737-4ae1-9aee-c36be0c9ec67\") " pod="openshift-dns/dns-default-pwzr2"
Sep 10 02:47:24.237379 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.237156    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82088748-d4dc-4c0b-939c-62b49e6d19d7-crio-socket\") pod \"insights-runtime-extractor-pfnd8\" (UID: \"82088748-d4dc-4c0b-939c-62b49e6d19d7\") " pod="openshift-insights/insights-runtime-extractor-pfnd8"
Sep 10 02:47:24.237379 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.237186    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/82088748-d4dc-4c0b-939c-62b49e6d19d7-data-volume\") pod \"insights-runtime-extractor-pfnd8\" (UID: \"82088748-d4dc-4c0b-939c-62b49e6d19d7\") " pod="openshift-insights/insights-runtime-extractor-pfnd8"
Sep 10 02:47:24.237379 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.237217    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82088748-d4dc-4c0b-939c-62b49e6d19d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pfnd8\" (UID: \"82088748-d4dc-4c0b-939c-62b49e6d19d7\") " pod="openshift-insights/insights-runtime-extractor-pfnd8"
Sep 10 02:47:24.237811 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.237772    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27053ebb-2737-4ae1-9aee-c36be0c9ec67-tmp-dir\") pod \"dns-default-pwzr2\" (UID: \"27053ebb-2737-4ae1-9aee-c36be0c9ec67\") " pod="openshift-dns/dns-default-pwzr2"
Sep 10 02:47:24.238177 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.238149    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/27053ebb-2737-4ae1-9aee-c36be0c9ec67-config-volume\") pod \"dns-default-pwzr2\" (UID: \"27053ebb-2737-4ae1-9aee-c36be0c9ec67\") " pod="openshift-dns/dns-default-pwzr2"
Sep 10 02:47:24.241987 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.241963    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cae06c98-aa57-4981-9318-77f97d4aeba6-cert\") pod \"ingress-canary-qqd6h\" (UID: \"cae06c98-aa57-4981-9318-77f97d4aeba6\") " pod="openshift-ingress-canary/ingress-canary-qqd6h"
Sep 10 02:47:24.245420 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.245394    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv7wz\" (UniqueName: \"kubernetes.io/projected/27053ebb-2737-4ae1-9aee-c36be0c9ec67-kube-api-access-bv7wz\") pod \"dns-default-pwzr2\" (UID: \"27053ebb-2737-4ae1-9aee-c36be0c9ec67\") " pod="openshift-dns/dns-default-pwzr2"
Sep 10 02:47:24.245598 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.245582    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpmjd\" (UniqueName: \"kubernetes.io/projected/cae06c98-aa57-4981-9318-77f97d4aeba6-kube-api-access-fpmjd\") pod \"ingress-canary-qqd6h\" (UID: \"cae06c98-aa57-4981-9318-77f97d4aeba6\") " pod="openshift-ingress-canary/ingress-canary-qqd6h"
Sep 10 02:47:24.338008 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.337985    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82088748-d4dc-4c0b-939c-62b49e6d19d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pfnd8\" (UID: \"82088748-d4dc-4c0b-939c-62b49e6d19d7\") " pod="openshift-insights/insights-runtime-extractor-pfnd8"
Sep 10 02:47:24.338092 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.338027    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dszzf\" (UniqueName: \"kubernetes.io/projected/82088748-d4dc-4c0b-939c-62b49e6d19d7-kube-api-access-dszzf\") pod \"insights-runtime-extractor-pfnd8\" (UID: \"82088748-d4dc-4c0b-939c-62b49e6d19d7\") " pod="openshift-insights/insights-runtime-extractor-pfnd8"
Sep 10 02:47:24.338159 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:24.338091    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 02:47:24.338194 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.338171    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82088748-d4dc-4c0b-939c-62b49e6d19d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pfnd8\" (UID: \"82088748-d4dc-4c0b-939c-62b49e6d19d7\") " pod="openshift-insights/insights-runtime-extractor-pfnd8"
Sep 10 02:47:24.338265 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:24.338251    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 02:47:24.338300 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.338247    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82088748-d4dc-4c0b-939c-62b49e6d19d7-crio-socket\") pod \"insights-runtime-extractor-pfnd8\" (UID: \"82088748-d4dc-4c0b-939c-62b49e6d19d7\") " pod="openshift-insights/insights-runtime-extractor-pfnd8"
Sep 10 02:47:24.338340 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:24.338300    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82088748-d4dc-4c0b-939c-62b49e6d19d7-kube-rbac-proxy-cm podName:82088748-d4dc-4c0b-939c-62b49e6d19d7 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:24.838279691 +0000 UTC m=+35.213365603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82088748-d4dc-4c0b-939c-62b49e6d19d7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pfnd8" (UID: "82088748-d4dc-4c0b-939c-62b49e6d19d7") : configmap "kube-rbac-proxy" not found
Sep 10 02:47:24.338340 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:24.338323    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82088748-d4dc-4c0b-939c-62b49e6d19d7-insights-runtime-extractor-tls podName:82088748-d4dc-4c0b-939c-62b49e6d19d7 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:24.838316918 +0000 UTC m=+35.213402822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82088748-d4dc-4c0b-939c-62b49e6d19d7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pfnd8" (UID: "82088748-d4dc-4c0b-939c-62b49e6d19d7") : secret "insights-runtime-extractor-tls" not found
Sep 10 02:47:24.338411 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.338353    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82088748-d4dc-4c0b-939c-62b49e6d19d7-crio-socket\") pod \"insights-runtime-extractor-pfnd8\" (UID: \"82088748-d4dc-4c0b-939c-62b49e6d19d7\") " pod="openshift-insights/insights-runtime-extractor-pfnd8"
Sep 10 02:47:24.338411 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.338354    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/82088748-d4dc-4c0b-939c-62b49e6d19d7-data-volume\") pod \"insights-runtime-extractor-pfnd8\" (UID: \"82088748-d4dc-4c0b-939c-62b49e6d19d7\") " pod="openshift-insights/insights-runtime-extractor-pfnd8"
Sep 10 02:47:24.338554 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.338540    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/82088748-d4dc-4c0b-939c-62b49e6d19d7-data-volume\") pod \"insights-runtime-extractor-pfnd8\" (UID: \"82088748-d4dc-4c0b-939c-62b49e6d19d7\") " pod="openshift-insights/insights-runtime-extractor-pfnd8"
Sep 10 02:47:24.344302 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.344285    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qqd6h"
Sep 10 02:47:24.345773 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.345752    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dszzf\" (UniqueName: \"kubernetes.io/projected/82088748-d4dc-4c0b-939c-62b49e6d19d7-kube-api-access-dszzf\") pod \"insights-runtime-extractor-pfnd8\" (UID: \"82088748-d4dc-4c0b-939c-62b49e6d19d7\") " pod="openshift-insights/insights-runtime-extractor-pfnd8"
Sep 10 02:47:24.348593 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.348551    2558 generic.go:356] "Generic (PLEG): container finished" podID="62603892-88ce-474d-8d32-3b085dbbedd7" containerID="e70d66e67f82885063ec768afd0a7a6d432c4b026edae7f761e655f2510c3283" exitCode=0
Sep 10 02:47:24.348593 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.348591    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-btm4f" event={"ID":"62603892-88ce-474d-8d32-3b085dbbedd7","Type":"ContainerDied","Data":"e70d66e67f82885063ec768afd0a7a6d432c4b026edae7f761e655f2510c3283"}
Sep 10 02:47:24.665928 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.665866    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qqd6h"]
Sep 10 02:47:24.669487 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:47:24.669462    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcae06c98_aa57_4981_9318_77f97d4aeba6.slice/crio-625064337f0a6ae5ecd6b4f7cb37be432372c9c367f4f7848bb46d3746d66c5a WatchSource:0}: Error finding container 625064337f0a6ae5ecd6b4f7cb37be432372c9c367f4f7848bb46d3746d66c5a: Status 404 returned error can't find the container with id 625064337f0a6ae5ecd6b4f7cb37be432372c9c367f4f7848bb46d3746d66c5a
Sep 10 02:47:24.741450 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.741427    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/27053ebb-2737-4ae1-9aee-c36be0c9ec67-metrics-tls\") pod \"dns-default-pwzr2\" (UID: \"27053ebb-2737-4ae1-9aee-c36be0c9ec67\") " pod="openshift-dns/dns-default-pwzr2"
Sep 10 02:47:24.741559 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:24.741547    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 02:47:24.741610 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:24.741592    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27053ebb-2737-4ae1-9aee-c36be0c9ec67-metrics-tls podName:27053ebb-2737-4ae1-9aee-c36be0c9ec67 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:25.741578615 +0000 UTC m=+36.116664515 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/27053ebb-2737-4ae1-9aee-c36be0c9ec67-metrics-tls") pod "dns-default-pwzr2" (UID: "27053ebb-2737-4ae1-9aee-c36be0c9ec67") : secret "dns-default-metrics-tls" not found
Sep 10 02:47:24.842672 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.842649    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82088748-d4dc-4c0b-939c-62b49e6d19d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pfnd8\" (UID: \"82088748-d4dc-4c0b-939c-62b49e6d19d7\") " pod="openshift-insights/insights-runtime-extractor-pfnd8"
Sep 10 02:47:24.842774 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.842698    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82088748-d4dc-4c0b-939c-62b49e6d19d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pfnd8\" (UID: \"82088748-d4dc-4c0b-939c-62b49e6d19d7\") " pod="openshift-insights/insights-runtime-extractor-pfnd8"
Sep 10 02:47:24.842774 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:24.842752    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 02:47:24.842845 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:24.842785    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 02:47:24.842845 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:24.842790    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82088748-d4dc-4c0b-939c-62b49e6d19d7-kube-rbac-proxy-cm podName:82088748-d4dc-4c0b-939c-62b49e6d19d7 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:25.842780542 +0000 UTC m=+36.217866440 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82088748-d4dc-4c0b-939c-62b49e6d19d7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pfnd8" (UID: "82088748-d4dc-4c0b-939c-62b49e6d19d7") : configmap "kube-rbac-proxy" not found
Sep 10 02:47:24.842922 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:24.842847    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82088748-d4dc-4c0b-939c-62b49e6d19d7-insights-runtime-extractor-tls podName:82088748-d4dc-4c0b-939c-62b49e6d19d7 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:25.842830276 +0000 UTC m=+36.217916176 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82088748-d4dc-4c0b-939c-62b49e6d19d7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pfnd8" (UID: "82088748-d4dc-4c0b-939c-62b49e6d19d7") : secret "insights-runtime-extractor-tls" not found
Sep 10 02:47:24.915300 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:24.915269    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:25.062860 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:25.062836    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:47:25.062954 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:25.062924    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xn49"
Sep 10 02:47:25.062954 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:25.062948    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5ph4"
Sep 10 02:47:25.066221 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:25.066199    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 02:47:25.066221 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:25.066211    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 02:47:25.066406 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:25.066262    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 02:47:25.066406 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:25.066210    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 02:47:25.066406 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:25.066328    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-t98bg\""
Sep 10 02:47:25.066406 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:25.066361    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mtnpz\""
Sep 10 02:47:25.355452 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:25.355380    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-btm4f" event={"ID":"62603892-88ce-474d-8d32-3b085dbbedd7","Type":"ContainerStarted","Data":"3e9c8f32f2325b01f7f8386e5720a4056ddc25baf73bbd150746b4397b1461a8"}
Sep 10 02:47:25.356718 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:25.356695    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qqd6h" event={"ID":"cae06c98-aa57-4981-9318-77f97d4aeba6","Type":"ContainerStarted","Data":"625064337f0a6ae5ecd6b4f7cb37be432372c9c367f4f7848bb46d3746d66c5a"}
Sep 10 02:47:25.393399 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:25.393350    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-btm4f" podStartSLOduration=6.564135729 podStartE2EDuration="35.39333781s" totalImagesPullingTime="28.829202081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:46:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:25.392489265 +0000 UTC m=+35.767575187" watchObservedRunningTime="2026-09-10 02:47:25.39333781 +0000 UTC m=+35.768423731"
Sep 10 02:47:25.749387 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:25.749360    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/27053ebb-2737-4ae1-9aee-c36be0c9ec67-metrics-tls\") pod \"dns-default-pwzr2\" (UID: \"27053ebb-2737-4ae1-9aee-c36be0c9ec67\") " pod="openshift-dns/dns-default-pwzr2"
Sep 10 02:47:25.749543 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:25.749519    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 02:47:25.749604 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:25.749597    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27053ebb-2737-4ae1-9aee-c36be0c9ec67-metrics-tls podName:27053ebb-2737-4ae1-9aee-c36be0c9ec67 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:27.749577175 +0000 UTC m=+38.124663076 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/27053ebb-2737-4ae1-9aee-c36be0c9ec67-metrics-tls") pod "dns-default-pwzr2" (UID: "27053ebb-2737-4ae1-9aee-c36be0c9ec67") : secret "dns-default-metrics-tls" not found
Sep 10 02:47:25.850595 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:25.850570    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82088748-d4dc-4c0b-939c-62b49e6d19d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pfnd8\" (UID: \"82088748-d4dc-4c0b-939c-62b49e6d19d7\") " pod="openshift-insights/insights-runtime-extractor-pfnd8"
Sep 10 02:47:25.850726 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:25.850653    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82088748-d4dc-4c0b-939c-62b49e6d19d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pfnd8\" (UID: \"82088748-d4dc-4c0b-939c-62b49e6d19d7\") " pod="openshift-insights/insights-runtime-extractor-pfnd8"
Sep 10 02:47:25.850726 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:25.850711    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 02:47:25.850813 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:25.850774    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82088748-d4dc-4c0b-939c-62b49e6d19d7-insights-runtime-extractor-tls podName:82088748-d4dc-4c0b-939c-62b49e6d19d7 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:27.850760155 +0000 UTC m=+38.225846074 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82088748-d4dc-4c0b-939c-62b49e6d19d7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pfnd8" (UID: "82088748-d4dc-4c0b-939c-62b49e6d19d7") : secret "insights-runtime-extractor-tls" not found
Sep 10 02:47:25.851307 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:25.851287    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82088748-d4dc-4c0b-939c-62b49e6d19d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pfnd8\" (UID: \"82088748-d4dc-4c0b-939c-62b49e6d19d7\") " pod="openshift-insights/insights-runtime-extractor-pfnd8"
Sep 10 02:47:25.927113 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:25.927075    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:27.764706 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:27.764679    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/27053ebb-2737-4ae1-9aee-c36be0c9ec67-metrics-tls\") pod \"dns-default-pwzr2\" (UID: \"27053ebb-2737-4ae1-9aee-c36be0c9ec67\") " pod="openshift-dns/dns-default-pwzr2"
Sep 10 02:47:27.764986 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:27.764838    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 02:47:27.764986 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:27.764902    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27053ebb-2737-4ae1-9aee-c36be0c9ec67-metrics-tls podName:27053ebb-2737-4ae1-9aee-c36be0c9ec67 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:31.764885923 +0000 UTC m=+42.139971822 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/27053ebb-2737-4ae1-9aee-c36be0c9ec67-metrics-tls") pod "dns-default-pwzr2" (UID: "27053ebb-2737-4ae1-9aee-c36be0c9ec67") : secret "dns-default-metrics-tls" not found
Sep 10 02:47:27.865343 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:27.865195    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82088748-d4dc-4c0b-939c-62b49e6d19d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pfnd8\" (UID: \"82088748-d4dc-4c0b-939c-62b49e6d19d7\") " pod="openshift-insights/insights-runtime-extractor-pfnd8"
Sep 10 02:47:27.865442 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:27.865332    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 02:47:27.865442 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:27.865422    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82088748-d4dc-4c0b-939c-62b49e6d19d7-insights-runtime-extractor-tls podName:82088748-d4dc-4c0b-939c-62b49e6d19d7 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:31.865408092 +0000 UTC m=+42.240493994 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82088748-d4dc-4c0b-939c-62b49e6d19d7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pfnd8" (UID: "82088748-d4dc-4c0b-939c-62b49e6d19d7") : secret "insights-runtime-extractor-tls" not found
Sep 10 02:47:27.966176 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:27.966140    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:28.365018 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:28.364980    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qqd6h" event={"ID":"cae06c98-aa57-4981-9318-77f97d4aeba6","Type":"ContainerStarted","Data":"df6bea2c31b0e6866f550e0c0bbf2ecc874906d9945b0084f7846c1c1ccb4d73"}
Sep 10 02:47:28.386721 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:28.386695    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:28.458435 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:28.458412    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:28.824198 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:28.824169    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:29.184123 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:29.184031    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:29.284266 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:29.284243    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:29.385138 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:29.385117    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:29.486386 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:29.486361    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:29.598099 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:29.598080    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:29.723464 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:29.723439    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:29.830352 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:29.830294    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:30.784492 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:30.784466    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:30.988996 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:30.988966    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:31.185714 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:31.185662    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:31.391981 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:31.391956    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:31.456979 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:31.456951    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:31.588435 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:31.588407    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:31.792197 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:31.792136    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/27053ebb-2737-4ae1-9aee-c36be0c9ec67-metrics-tls\") pod \"dns-default-pwzr2\" (UID: \"27053ebb-2737-4ae1-9aee-c36be0c9ec67\") " pod="openshift-dns/dns-default-pwzr2"
Sep 10 02:47:31.792301 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:31.792283    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 02:47:31.792351 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:31.792344    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27053ebb-2737-4ae1-9aee-c36be0c9ec67-metrics-tls podName:27053ebb-2737-4ae1-9aee-c36be0c9ec67 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:39.79232949 +0000 UTC m=+50.167415389 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/27053ebb-2737-4ae1-9aee-c36be0c9ec67-metrics-tls") pod "dns-default-pwzr2" (UID: "27053ebb-2737-4ae1-9aee-c36be0c9ec67") : secret "dns-default-metrics-tls" not found
Sep 10 02:47:31.892796 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:31.892771    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82088748-d4dc-4c0b-939c-62b49e6d19d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pfnd8\" (UID: \"82088748-d4dc-4c0b-939c-62b49e6d19d7\") " pod="openshift-insights/insights-runtime-extractor-pfnd8"
Sep 10 02:47:31.892942 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:31.892923    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 02:47:31.892994 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:47:31.892987    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82088748-d4dc-4c0b-939c-62b49e6d19d7-insights-runtime-extractor-tls podName:82088748-d4dc-4c0b-939c-62b49e6d19d7 nodeName:}" failed. No retries permitted until 2026-09-10 02:47:39.892970871 +0000 UTC m=+50.268056769 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82088748-d4dc-4c0b-939c-62b49e6d19d7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pfnd8" (UID: "82088748-d4dc-4c0b-939c-62b49e6d19d7") : secret "insights-runtime-extractor-tls" not found
Sep 10 02:47:32.485131 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:32.485087    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:32.585419 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:32.585391    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:32.644703 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:32.644669    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:32.982715 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:32.982689    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:33.179017 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:33.178988    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:33.377838 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:33.377785    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:33.434678 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:33.434655    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:33.586946 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:33.586919    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:34.392336 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:34.392298    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:34.581472 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:34.581446    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:34.682632 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:34.682579    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:34.737229 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:34.737210    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:35.090052 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:35.090024    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:35.186952 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:35.186926    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:35.394709 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:35.394655    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:35.980709 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:35.980684    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:36.179900 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:36.179872    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:36.782439 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:36.782410    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:37.179393 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:37.179321    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:37.579592 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:37.579560    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:37.787368 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:37.787339    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:38.182566 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:38.182542    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:38.774170 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:38.774142    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:39.200880 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:39.200850    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:39.383842 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:39.383808    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:39.578467 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:39.578406    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:39.853400 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:39.853343    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/27053ebb-2737-4ae1-9aee-c36be0c9ec67-metrics-tls\") pod \"dns-default-pwzr2\" (UID: \"27053ebb-2737-4ae1-9aee-c36be0c9ec67\") " pod="openshift-dns/dns-default-pwzr2"
Sep 10 02:47:39.857293 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:39.857259    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/27053ebb-2737-4ae1-9aee-c36be0c9ec67-metrics-tls\") pod \"dns-default-pwzr2\" (UID: \"27053ebb-2737-4ae1-9aee-c36be0c9ec67\") " pod="openshift-dns/dns-default-pwzr2"
Sep 10 02:47:39.926525 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:39.926497    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pwzr2"
Sep 10 02:47:39.954008 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:39.953978    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82088748-d4dc-4c0b-939c-62b49e6d19d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pfnd8\" (UID: \"82088748-d4dc-4c0b-939c-62b49e6d19d7\") " pod="openshift-insights/insights-runtime-extractor-pfnd8"
Sep 10 02:47:39.956301 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:39.956277    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82088748-d4dc-4c0b-939c-62b49e6d19d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pfnd8\" (UID: \"82088748-d4dc-4c0b-939c-62b49e6d19d7\") " pod="openshift-insights/insights-runtime-extractor-pfnd8"
Sep 10 02:47:39.970331 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:39.970297    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pfnd8"
Sep 10 02:47:40.229412 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:40.229364    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qqd6h" podStartSLOduration=14.174510307 podStartE2EDuration="17.229348615s" totalImagesPullingTime="3.054838308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:47:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:28.381912948 +0000 UTC m=+38.756998871" watchObservedRunningTime="2026-09-10 02:47:40.229348615 +0000 UTC m=+50.604434536"
Sep 10 02:47:40.229863 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:40.229841    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pwzr2"]
Sep 10 02:47:40.232740 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:47:40.232720    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27053ebb_2737_4ae1_9aee_c36be0c9ec67.slice/crio-e7886f553c6a3edaa6be4b7090906da493b18429a836b1b52da6a0d90a70124c WatchSource:0}: Error finding container e7886f553c6a3edaa6be4b7090906da493b18429a836b1b52da6a0d90a70124c: Status 404 returned error can't find the container with id e7886f553c6a3edaa6be4b7090906da493b18429a836b1b52da6a0d90a70124c
Sep 10 02:47:40.272286 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:40.272264    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pfnd8"]
Sep 10 02:47:40.275091 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:47:40.275069    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82088748_d4dc_4c0b_939c_62b49e6d19d7.slice/crio-ab15a1864404d388ce055ea9390345e0633004e55323eb209be7f715f22f10ab WatchSource:0}: Error finding container ab15a1864404d388ce055ea9390345e0633004e55323eb209be7f715f22f10ab: Status 404 returned error can't find the container with id ab15a1864404d388ce055ea9390345e0633004e55323eb209be7f715f22f10ab
Sep 10 02:47:40.394747 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:40.394711    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pfnd8" event={"ID":"82088748-d4dc-4c0b-939c-62b49e6d19d7","Type":"ContainerStarted","Data":"3ee998ff55a2903f88fb989cec5a6c2fd272b513517ce8acd327775b6379f7da"}
Sep 10 02:47:40.394861 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:40.394768    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pfnd8" event={"ID":"82088748-d4dc-4c0b-939c-62b49e6d19d7","Type":"ContainerStarted","Data":"ab15a1864404d388ce055ea9390345e0633004e55323eb209be7f715f22f10ab"}
Sep 10 02:47:40.395751 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:40.395730    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pwzr2" event={"ID":"27053ebb-2737-4ae1-9aee-c36be0c9ec67","Type":"ContainerStarted","Data":"e7886f553c6a3edaa6be4b7090906da493b18429a836b1b52da6a0d90a70124c"}
Sep 10 02:47:41.401219 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:41.401139    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pfnd8" event={"ID":"82088748-d4dc-4c0b-939c-62b49e6d19d7","Type":"ContainerStarted","Data":"4466ae5f309ce5fd3988d6c99bc96b6de1703801339b3c45ec1be4af3e19fe88"}
Sep 10 02:47:42.406035 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:42.405992    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pwzr2" event={"ID":"27053ebb-2737-4ae1-9aee-c36be0c9ec67","Type":"ContainerStarted","Data":"91619a46fdda56b726ee21bb5f8ee96404fe91c6da249c83059b63717ff371d2"}
Sep 10 02:47:42.406502 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:42.406083    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pwzr2" event={"ID":"27053ebb-2737-4ae1-9aee-c36be0c9ec67","Type":"ContainerStarted","Data":"4fbd771db3241ac61a1fe803041fe01c7b88a3f62167fe1aa9cd8a80952bf592"}
Sep 10 02:47:42.406502 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:42.406168    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pwzr2"
Sep 10 02:47:42.424319 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:42.424268    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pwzr2" podStartSLOduration=17.925496252 podStartE2EDuration="19.42425549s" totalImagesPullingTime="1.498759238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:47:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:42.423921107 +0000 UTC m=+52.799007030" watchObservedRunningTime="2026-09-10 02:47:42.42425549 +0000 UTC m=+52.799341414"
Sep 10 02:47:43.411621 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.411586    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pfnd8" event={"ID":"82088748-d4dc-4c0b-939c-62b49e6d19d7","Type":"ContainerStarted","Data":"d4577609267ccc94bf5c7eda44454f29ee2f50108a0ee8bef46406ffd8203007"}
Sep 10 02:47:43.444739 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.444689    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pfnd8" podStartSLOduration=18.159585963 podStartE2EDuration="20.444673781s" totalImagesPullingTime="2.285087818s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:47:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:43.443373162 +0000 UTC m=+53.818459084" watchObservedRunningTime="2026-09-10 02:47:43.444673781 +0000 UTC m=+53.819759717"
Sep 10 02:47:43.688449 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.688387    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-tcjdj"]
Sep 10 02:47:43.690033 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.690016    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-tcjdj"
Sep 10 02:47:43.692891 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.692862    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 10 02:47:43.693045 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.693024    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 10 02:47:43.693141 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.693043    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-lgbgn\""
Sep 10 02:47:43.693306 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.693291    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8559c5d4-h5g2l"]
Sep 10 02:47:43.694948 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.694930    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66fdc7db5b-75rst"]
Sep 10 02:47:43.695145 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.695097    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8559c5d4-h5g2l"
Sep 10 02:47:43.696631 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.696610    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66fdc7db5b-75rst"
Sep 10 02:47:43.697850 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.697745    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 02:47:43.698939 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.698914    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 02:47:43.699046 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.698923    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 02:47:43.699046 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.698957    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-m4kj5\""
Sep 10 02:47:43.699273 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.699251    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 10 02:47:43.699432 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.699407    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 10 02:47:43.703615 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.703592    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-tcjdj"]
Sep 10 02:47:43.706798 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.706778    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bd45fc999-stnd6"]
Sep 10 02:47:43.709431 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.708881    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bd45fc999-stnd6"
Sep 10 02:47:43.709913 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.709895    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8559c5d4-h5g2l"]
Sep 10 02:47:43.711196 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.711171    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 10 02:47:43.711338 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.711257    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 10 02:47:43.711405 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.711388    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 10 02:47:43.711494 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.711473    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 10 02:47:43.723554 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.723531    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bd45fc999-stnd6"]
Sep 10 02:47:43.725500 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.725478    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66fdc7db5b-75rst"]
Sep 10 02:47:43.783903 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.783877    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/39430887-0027-4101-86e6-be7bf438d62b-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-tcjdj\" (UID: \"39430887-0027-4101-86e6-be7bf438d62b\") " pod="openshift-network-console/networking-console-plugin-79977f9844-tcjdj"
Sep 10 02:47:43.784013 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.783924    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/709daf29-7cf3-4cb1-bc5d-6c6fd7e0d8a4-tmp\") pod \"klusterlet-addon-workmgr-d8559c5d4-h5g2l\" (UID: \"709daf29-7cf3-4cb1-bc5d-6c6fd7e0d8a4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8559c5d4-h5g2l"
Sep 10 02:47:43.784013 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.783941    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ed24b9bd-58ec-4a2f-97e9-13491402b73b-hub\") pod \"cluster-proxy-proxy-agent-5bd45fc999-stnd6\" (UID: \"ed24b9bd-58ec-4a2f-97e9-13491402b73b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bd45fc999-stnd6"
Sep 10 02:47:43.784013 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.783959    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ed24b9bd-58ec-4a2f-97e9-13491402b73b-ca\") pod \"cluster-proxy-proxy-agent-5bd45fc999-stnd6\" (UID: \"ed24b9bd-58ec-4a2f-97e9-13491402b73b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bd45fc999-stnd6"
Sep 10 02:47:43.784142 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.784010    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ed24b9bd-58ec-4a2f-97e9-13491402b73b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5bd45fc999-stnd6\" (UID: \"ed24b9bd-58ec-4a2f-97e9-13491402b73b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bd45fc999-stnd6"
Sep 10 02:47:43.784142 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.784075    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/709daf29-7cf3-4cb1-bc5d-6c6fd7e0d8a4-klusterlet-config\") pod \"klusterlet-addon-workmgr-d8559c5d4-h5g2l\" (UID: \"709daf29-7cf3-4cb1-bc5d-6c6fd7e0d8a4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8559c5d4-h5g2l"
Sep 10 02:47:43.784142 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.784095    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed24b9bd-58ec-4a2f-97e9-13491402b73b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5bd45fc999-stnd6\" (UID: \"ed24b9bd-58ec-4a2f-97e9-13491402b73b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bd45fc999-stnd6"
Sep 10 02:47:43.784250 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.784144    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtkk4\" (UniqueName: \"kubernetes.io/projected/709daf29-7cf3-4cb1-bc5d-6c6fd7e0d8a4-kube-api-access-vtkk4\") pod \"klusterlet-addon-workmgr-d8559c5d4-h5g2l\" (UID: \"709daf29-7cf3-4cb1-bc5d-6c6fd7e0d8a4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8559c5d4-h5g2l"
Sep 10 02:47:43.784250 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.784164    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8dxh\" (UniqueName: \"kubernetes.io/projected/15085d98-fc88-429b-8d18-f3719e7fec4d-kube-api-access-j8dxh\") pod \"managed-serviceaccount-addon-agent-66fdc7db5b-75rst\" (UID: \"15085d98-fc88-429b-8d18-f3719e7fec4d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66fdc7db5b-75rst"
Sep 10 02:47:43.784250 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.784180    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxjt4\" (UniqueName: \"kubernetes.io/projected/ed24b9bd-58ec-4a2f-97e9-13491402b73b-kube-api-access-qxjt4\") pod \"cluster-proxy-proxy-agent-5bd45fc999-stnd6\" (UID: \"ed24b9bd-58ec-4a2f-97e9-13491402b73b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bd45fc999-stnd6"
Sep 10 02:47:43.784250 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.784195    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/39430887-0027-4101-86e6-be7bf438d62b-nginx-conf\") pod \"networking-console-plugin-79977f9844-tcjdj\" (UID: \"39430887-0027-4101-86e6-be7bf438d62b\") " pod="openshift-network-console/networking-console-plugin-79977f9844-tcjdj"
Sep 10 02:47:43.784250 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.784221    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/15085d98-fc88-429b-8d18-f3719e7fec4d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-66fdc7db5b-75rst\" (UID: \"15085d98-fc88-429b-8d18-f3719e7fec4d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66fdc7db5b-75rst"
Sep 10 02:47:43.784250 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.784244    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ed24b9bd-58ec-4a2f-97e9-13491402b73b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5bd45fc999-stnd6\" (UID: \"ed24b9bd-58ec-4a2f-97e9-13491402b73b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bd45fc999-stnd6"
Sep 10 02:47:43.826331 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.826311    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:43.885053 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.885030    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/709daf29-7cf3-4cb1-bc5d-6c6fd7e0d8a4-klusterlet-config\") pod \"klusterlet-addon-workmgr-d8559c5d4-h5g2l\" (UID: \"709daf29-7cf3-4cb1-bc5d-6c6fd7e0d8a4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8559c5d4-h5g2l"
Sep 10 02:47:43.885166 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.885062    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed24b9bd-58ec-4a2f-97e9-13491402b73b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5bd45fc999-stnd6\" (UID: \"ed24b9bd-58ec-4a2f-97e9-13491402b73b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bd45fc999-stnd6"
Sep 10 02:47:43.885166 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.885087    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtkk4\" (UniqueName: \"kubernetes.io/projected/709daf29-7cf3-4cb1-bc5d-6c6fd7e0d8a4-kube-api-access-vtkk4\") pod \"klusterlet-addon-workmgr-d8559c5d4-h5g2l\" (UID: \"709daf29-7cf3-4cb1-bc5d-6c6fd7e0d8a4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8559c5d4-h5g2l"
Sep 10 02:47:43.885166 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.885121    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8dxh\" (UniqueName: \"kubernetes.io/projected/15085d98-fc88-429b-8d18-f3719e7fec4d-kube-api-access-j8dxh\") pod \"managed-serviceaccount-addon-agent-66fdc7db5b-75rst\" (UID: \"15085d98-fc88-429b-8d18-f3719e7fec4d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66fdc7db5b-75rst"
Sep 10 02:47:43.885166 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.885144    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxjt4\" (UniqueName: \"kubernetes.io/projected/ed24b9bd-58ec-4a2f-97e9-13491402b73b-kube-api-access-qxjt4\") pod \"cluster-proxy-proxy-agent-5bd45fc999-stnd6\" (UID: \"ed24b9bd-58ec-4a2f-97e9-13491402b73b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bd45fc999-stnd6"
Sep 10 02:47:43.885166 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.885162    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/39430887-0027-4101-86e6-be7bf438d62b-nginx-conf\") pod \"networking-console-plugin-79977f9844-tcjdj\" (UID: \"39430887-0027-4101-86e6-be7bf438d62b\") " pod="openshift-network-console/networking-console-plugin-79977f9844-tcjdj"
Sep 10 02:47:43.885412 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.885184    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/15085d98-fc88-429b-8d18-f3719e7fec4d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-66fdc7db5b-75rst\" (UID: \"15085d98-fc88-429b-8d18-f3719e7fec4d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66fdc7db5b-75rst"
Sep 10 02:47:43.885412 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.885206    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ed24b9bd-58ec-4a2f-97e9-13491402b73b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5bd45fc999-stnd6\" (UID: \"ed24b9bd-58ec-4a2f-97e9-13491402b73b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bd45fc999-stnd6"
Sep 10 02:47:43.885412 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.885239    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/39430887-0027-4101-86e6-be7bf438d62b-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-tcjdj\" (UID: \"39430887-0027-4101-86e6-be7bf438d62b\") " pod="openshift-network-console/networking-console-plugin-79977f9844-tcjdj"
Sep 10 02:47:43.885412 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.885291    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/709daf29-7cf3-4cb1-bc5d-6c6fd7e0d8a4-tmp\") pod \"klusterlet-addon-workmgr-d8559c5d4-h5g2l\" (UID: \"709daf29-7cf3-4cb1-bc5d-6c6fd7e0d8a4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8559c5d4-h5g2l"
Sep 10 02:47:43.885412 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.885313    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ed24b9bd-58ec-4a2f-97e9-13491402b73b-hub\") pod \"cluster-proxy-proxy-agent-5bd45fc999-stnd6\" (UID: \"ed24b9bd-58ec-4a2f-97e9-13491402b73b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bd45fc999-stnd6"
Sep 10 02:47:43.885650 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.885576    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ed24b9bd-58ec-4a2f-97e9-13491402b73b-ca\") pod \"cluster-proxy-proxy-agent-5bd45fc999-stnd6\" (UID: \"ed24b9bd-58ec-4a2f-97e9-13491402b73b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bd45fc999-stnd6"
Sep 10 02:47:43.885650 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.885625    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ed24b9bd-58ec-4a2f-97e9-13491402b73b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5bd45fc999-stnd6\" (UID: \"ed24b9bd-58ec-4a2f-97e9-13491402b73b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bd45fc999-stnd6"
Sep 10 02:47:43.885880 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.885857    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed24b9bd-58ec-4a2f-97e9-13491402b73b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5bd45fc999-stnd6\" (UID: \"ed24b9bd-58ec-4a2f-97e9-13491402b73b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bd45fc999-stnd6"
Sep 10 02:47:43.886049 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.886024    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/709daf29-7cf3-4cb1-bc5d-6c6fd7e0d8a4-tmp\") pod \"klusterlet-addon-workmgr-d8559c5d4-h5g2l\" (UID: \"709daf29-7cf3-4cb1-bc5d-6c6fd7e0d8a4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8559c5d4-h5g2l"
Sep 10 02:47:43.887132 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.887091    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/39430887-0027-4101-86e6-be7bf438d62b-nginx-conf\") pod \"networking-console-plugin-79977f9844-tcjdj\" (UID: \"39430887-0027-4101-86e6-be7bf438d62b\") " pod="openshift-network-console/networking-console-plugin-79977f9844-tcjdj"
Sep 10 02:47:43.888330 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.888299    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/709daf29-7cf3-4cb1-bc5d-6c6fd7e0d8a4-klusterlet-config\") pod \"klusterlet-addon-workmgr-d8559c5d4-h5g2l\" (UID: \"709daf29-7cf3-4cb1-bc5d-6c6fd7e0d8a4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8559c5d4-h5g2l"
Sep 10 02:47:43.888547 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.888520    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/39430887-0027-4101-86e6-be7bf438d62b-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-tcjdj\" (UID: \"39430887-0027-4101-86e6-be7bf438d62b\") " pod="openshift-network-console/networking-console-plugin-79977f9844-tcjdj"
Sep 10 02:47:43.888612 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.888577    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ed24b9bd-58ec-4a2f-97e9-13491402b73b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5bd45fc999-stnd6\" (UID: \"ed24b9bd-58ec-4a2f-97e9-13491402b73b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bd45fc999-stnd6"
Sep 10 02:47:43.888612 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.888601    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ed24b9bd-58ec-4a2f-97e9-13491402b73b-hub\") pod \"cluster-proxy-proxy-agent-5bd45fc999-stnd6\" (UID: \"ed24b9bd-58ec-4a2f-97e9-13491402b73b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bd45fc999-stnd6"
Sep 10 02:47:43.888695 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.888644    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/15085d98-fc88-429b-8d18-f3719e7fec4d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-66fdc7db5b-75rst\" (UID: \"15085d98-fc88-429b-8d18-f3719e7fec4d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66fdc7db5b-75rst"
Sep 10 02:47:43.888862 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.888846    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ed24b9bd-58ec-4a2f-97e9-13491402b73b-ca\") pod \"cluster-proxy-proxy-agent-5bd45fc999-stnd6\" (UID: \"ed24b9bd-58ec-4a2f-97e9-13491402b73b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bd45fc999-stnd6"
Sep 10 02:47:43.889042 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.889022    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ed24b9bd-58ec-4a2f-97e9-13491402b73b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5bd45fc999-stnd6\" (UID: \"ed24b9bd-58ec-4a2f-97e9-13491402b73b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bd45fc999-stnd6"
Sep 10 02:47:43.894015 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.893997    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxjt4\" (UniqueName: \"kubernetes.io/projected/ed24b9bd-58ec-4a2f-97e9-13491402b73b-kube-api-access-qxjt4\") pod \"cluster-proxy-proxy-agent-5bd45fc999-stnd6\" (UID: \"ed24b9bd-58ec-4a2f-97e9-13491402b73b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bd45fc999-stnd6"
Sep 10 02:47:43.894407 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.894387    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8dxh\" (UniqueName: \"kubernetes.io/projected/15085d98-fc88-429b-8d18-f3719e7fec4d-kube-api-access-j8dxh\") pod \"managed-serviceaccount-addon-agent-66fdc7db5b-75rst\" (UID: \"15085d98-fc88-429b-8d18-f3719e7fec4d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66fdc7db5b-75rst"
Sep 10 02:47:43.894473 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.894455    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtkk4\" (UniqueName: \"kubernetes.io/projected/709daf29-7cf3-4cb1-bc5d-6c6fd7e0d8a4-kube-api-access-vtkk4\") pod \"klusterlet-addon-workmgr-d8559c5d4-h5g2l\" (UID: \"709daf29-7cf3-4cb1-bc5d-6c6fd7e0d8a4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8559c5d4-h5g2l"
Sep 10 02:47:43.969316 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:43.969291    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:44.001848 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:44.001828    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-tcjdj"
Sep 10 02:47:44.010158 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:44.010140    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8559c5d4-h5g2l"
Sep 10 02:47:44.028944 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:44.028923    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66fdc7db5b-75rst"
Sep 10 02:47:44.033080 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:44.033056    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bd45fc999-stnd6"
Sep 10 02:47:44.070731 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:44.070665    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:44.273937 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:44.273873    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-tcjdj"]
Sep 10 02:47:44.276991 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:47:44.276968    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39430887_0027_4101_86e6_be7bf438d62b.slice/crio-475607cb154d1a839aa1bdfa1b3351eb8fabeb85e24ede9829ee29afcda11d4a WatchSource:0}: Error finding container 475607cb154d1a839aa1bdfa1b3351eb8fabeb85e24ede9829ee29afcda11d4a: Status 404 returned error can't find the container with id 475607cb154d1a839aa1bdfa1b3351eb8fabeb85e24ede9829ee29afcda11d4a
Sep 10 02:47:44.299173 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:44.299149    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8559c5d4-h5g2l"]
Sep 10 02:47:44.301433 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:47:44.301409    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod709daf29_7cf3_4cb1_bc5d_6c6fd7e0d8a4.slice/crio-5b05fe19afafcca54e38f0f58336fd270401d203f07c20947dd96b397e523807 WatchSource:0}: Error finding container 5b05fe19afafcca54e38f0f58336fd270401d203f07c20947dd96b397e523807: Status 404 returned error can't find the container with id 5b05fe19afafcca54e38f0f58336fd270401d203f07c20947dd96b397e523807
Sep 10 02:47:44.318572 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:44.318550    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66fdc7db5b-75rst"]
Sep 10 02:47:44.321336 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:47:44.320980    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15085d98_fc88_429b_8d18_f3719e7fec4d.slice/crio-1400354419fb81b6ae795e7def4f52e5bca4228be41df6003b2ec36b482d1b2a WatchSource:0}: Error finding container 1400354419fb81b6ae795e7def4f52e5bca4228be41df6003b2ec36b482d1b2a: Status 404 returned error can't find the container with id 1400354419fb81b6ae795e7def4f52e5bca4228be41df6003b2ec36b482d1b2a
Sep 10 02:47:44.322258 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:44.322240    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bd45fc999-stnd6"]
Sep 10 02:47:44.325086 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:47:44.325066    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded24b9bd_58ec_4a2f_97e9_13491402b73b.slice/crio-931f205f7052bc4dccea00c0714c1b713283ed04b98dcdc8ef508059754fad38 WatchSource:0}: Error finding container 931f205f7052bc4dccea00c0714c1b713283ed04b98dcdc8ef508059754fad38: Status 404 returned error can't find the container with id 931f205f7052bc4dccea00c0714c1b713283ed04b98dcdc8ef508059754fad38
Sep 10 02:47:44.416175 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:44.416142    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bd45fc999-stnd6" event={"ID":"ed24b9bd-58ec-4a2f-97e9-13491402b73b","Type":"ContainerStarted","Data":"931f205f7052bc4dccea00c0714c1b713283ed04b98dcdc8ef508059754fad38"}
Sep 10 02:47:44.417259 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:44.417232    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8559c5d4-h5g2l" event={"ID":"709daf29-7cf3-4cb1-bc5d-6c6fd7e0d8a4","Type":"ContainerStarted","Data":"5b05fe19afafcca54e38f0f58336fd270401d203f07c20947dd96b397e523807"}
Sep 10 02:47:44.418332 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:44.418314    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66fdc7db5b-75rst" event={"ID":"15085d98-fc88-429b-8d18-f3719e7fec4d","Type":"ContainerStarted","Data":"1400354419fb81b6ae795e7def4f52e5bca4228be41df6003b2ec36b482d1b2a"}
Sep 10 02:47:44.419351 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:44.419330    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-tcjdj" event={"ID":"39430887-0027-4101-86e6-be7bf438d62b","Type":"ContainerStarted","Data":"475607cb154d1a839aa1bdfa1b3351eb8fabeb85e24ede9829ee29afcda11d4a"}
Sep 10 02:47:44.713058 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:44.713030    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:44.963585 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:44.963514    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:45.063146 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:45.063099    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:48.839793 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:48.839715    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:48.940524 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:48.940492    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:49.048446 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:49.048411    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:49.156656 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:49.156573    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:49.340712 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:49.340688    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bltxf"
Sep 10 02:47:49.511171 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:49.511137    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:49.617369 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:49.617331    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:51.447082 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:51.447049    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8559c5d4-h5g2l" event={"ID":"709daf29-7cf3-4cb1-bc5d-6c6fd7e0d8a4","Type":"ContainerStarted","Data":"b820163206c0e910f81440ddbc481d38dcd6a972f9e999c373630851b3c292d3"}
Sep 10 02:47:51.447524 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:51.447251    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8559c5d4-h5g2l"
Sep 10 02:47:51.448859 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:51.448840    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8559c5d4-h5g2l"
Sep 10 02:47:51.449043 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:51.449020    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66fdc7db5b-75rst" event={"ID":"15085d98-fc88-429b-8d18-f3719e7fec4d","Type":"ContainerStarted","Data":"a85901d094d4858f05ae21c4d524769bf186dcef2285b0df4a7519618326a1ee"}
Sep 10 02:47:51.450574 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:51.450547    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-tcjdj" event={"ID":"39430887-0027-4101-86e6-be7bf438d62b","Type":"ContainerStarted","Data":"b99ae658d871f2d2fe255aff4b143138cc5347301be20885e645240702931712"}
Sep 10 02:47:51.452715 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:51.452697    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bd45fc999-stnd6" event={"ID":"ed24b9bd-58ec-4a2f-97e9-13491402b73b","Type":"ContainerStarted","Data":"eb4b00818553b00a2a291a5c18ff958d49748bf90bf9f6c7a5f4720cf437f0a5"}
Sep 10 02:47:51.452795 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:51.452722    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bd45fc999-stnd6" event={"ID":"ed24b9bd-58ec-4a2f-97e9-13491402b73b","Type":"ContainerStarted","Data":"99e957bc93fefcf7b97a35c09bfc283ca9bac1f6501f836785a94436ad1224ad"}
Sep 10 02:47:51.452795 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:51.452735    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bd45fc999-stnd6" event={"ID":"ed24b9bd-58ec-4a2f-97e9-13491402b73b","Type":"ContainerStarted","Data":"e418bdaa6e2dfc93adc5ffdb331a9eee7d498056a69f2a4a61aaabfbf45237a4"}
Sep 10 02:47:51.464668 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:51.464625    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8559c5d4-h5g2l" podStartSLOduration=2.043701588 podStartE2EDuration="8.464611016s" totalImagesPullingTime="6.420909428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:51.464500065 +0000 UTC m=+61.839585987" watchObservedRunningTime="2026-09-10 02:47:51.464611016 +0000 UTC m=+61.839696935"
Sep 10 02:47:51.481113 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:51.481074    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66fdc7db5b-75rst" podStartSLOduration=2.0681795 podStartE2EDuration="8.481063322s" totalImagesPullingTime="6.412883822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:51.480410313 +0000 UTC m=+61.855496233" watchObservedRunningTime="2026-09-10 02:47:51.481063322 +0000 UTC m=+61.856149272"
Sep 10 02:47:51.516744 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:51.516706    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bd45fc999-stnd6" podStartSLOduration=2.100937387 podStartE2EDuration="8.516694367s" totalImagesPullingTime="6.41575698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:51.515857998 +0000 UTC m=+61.890943918" watchObservedRunningTime="2026-09-10 02:47:51.516694367 +0000 UTC m=+61.891780288"
Sep 10 02:47:51.531595 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:51.531553    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-tcjdj" podStartSLOduration=6.353555053 podStartE2EDuration="8.531543311s" totalImagesPullingTime="2.177988258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:51.530871015 +0000 UTC m=+61.905956940" watchObservedRunningTime="2026-09-10 02:47:51.531543311 +0000 UTC m=+61.906629229"
Sep 10 02:47:51.631730 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:51.631703    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:51.705124 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:51.705089    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:51.765663 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:51.765641    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:51.868477 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:51.868451    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:52.072643 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:52.072585    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:52.193966 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:52.193939    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:52.270735 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:52.270703    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:52.333902 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:52.333839    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:52.414325 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:52.414305    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pwzr2"
Sep 10 02:47:54.078363 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:54.078328    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:54.178437 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:54.178409    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:54.731776 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:54.731738    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:54.767895 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:54.767866    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de3a2ed4-28df-4dea-8e14-c5dfe2981ba5-original-pull-secret\") pod \"global-pull-secret-syncer-5qtg6\" (UID: \"de3a2ed4-28df-4dea-8e14-c5dfe2981ba5\") " pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:47:54.770345 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:54.770327    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 02:47:54.780675 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:54.780647    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de3a2ed4-28df-4dea-8e14-c5dfe2981ba5-original-pull-secret\") pod \"global-pull-secret-syncer-5qtg6\" (UID: \"de3a2ed4-28df-4dea-8e14-c5dfe2981ba5\") " pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:47:54.833171 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:54.833140    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:54.868188 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:54.868160    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa599110-d0eb-4cc2-b4e9-e3984e9717fe-metrics-certs\") pod \"network-metrics-daemon-m5ph4\" (UID: \"aa599110-d0eb-4cc2-b4e9-e3984e9717fe\") " pod="openshift-multus/network-metrics-daemon-m5ph4"
Sep 10 02:47:54.870508 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:54.870489    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 02:47:54.880505 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:54.880484    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa599110-d0eb-4cc2-b4e9-e3984e9717fe-metrics-certs\") pod \"network-metrics-daemon-m5ph4\" (UID: \"aa599110-d0eb-4cc2-b4e9-e3984e9717fe\") " pod="openshift-multus/network-metrics-daemon-m5ph4"
Sep 10 02:47:54.969050 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:54.969022    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96gxm\" (UniqueName: \"kubernetes.io/projected/e7d69b54-6b09-4496-abe6-6da05c77e958-kube-api-access-96gxm\") pod \"network-check-target-7xn49\" (UID: \"e7d69b54-6b09-4496-abe6-6da05c77e958\") " pod="openshift-network-diagnostics/network-check-target-7xn49"
Sep 10 02:47:54.971460 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:54.971434    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 02:47:54.982351 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:54.982301    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 02:47:54.992358 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:54.992337    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96gxm\" (UniqueName: \"kubernetes.io/projected/e7d69b54-6b09-4496-abe6-6da05c77e958-kube-api-access-96gxm\") pod \"network-check-target-7xn49\" (UID: \"e7d69b54-6b09-4496-abe6-6da05c77e958\") " pod="openshift-network-diagnostics/network-check-target-7xn49"
Sep 10 02:47:55.072479 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:55.072443    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5qtg6"
Sep 10 02:47:55.081322 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:55.081299    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-t98bg\""
Sep 10 02:47:55.088123 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:55.088087    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7xn49"
Sep 10 02:47:55.088243 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:55.088183    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mtnpz\""
Sep 10 02:47:55.093683 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:55.093660    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m5ph4"
Sep 10 02:47:55.357283 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:55.357208    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5qtg6"]
Sep 10 02:47:55.361228 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:47:55.361196    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde3a2ed4_28df_4dea_8e14_c5dfe2981ba5.slice/crio-dc3319652b1cbf7344f5b6250f255676678f2dced4e2812464e51d50d1d10b25 WatchSource:0}: Error finding container dc3319652b1cbf7344f5b6250f255676678f2dced4e2812464e51d50d1d10b25: Status 404 returned error can't find the container with id dc3319652b1cbf7344f5b6250f255676678f2dced4e2812464e51d50d1d10b25
Sep 10 02:47:55.375989 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:55.375964    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:55.387427 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:55.387404    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7xn49"]
Sep 10 02:47:55.389435 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:47:55.389410    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d69b54_6b09_4496_abe6_6da05c77e958.slice/crio-cb2f6c4b660e364612622655cbde306962022393efc0d7b21fa4287259763333 WatchSource:0}: Error finding container cb2f6c4b660e364612622655cbde306962022393efc0d7b21fa4287259763333: Status 404 returned error can't find the container with id cb2f6c4b660e364612622655cbde306962022393efc0d7b21fa4287259763333
Sep 10 02:47:55.393096 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:55.393031    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m5ph4"]
Sep 10 02:47:55.395685 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:47:55.395662    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa599110_d0eb_4cc2_b4e9_e3984e9717fe.slice/crio-fc9968594e54777061e2d8330cf2e8e6d7b96683b53d148bec5bb69291910104 WatchSource:0}: Error finding container fc9968594e54777061e2d8330cf2e8e6d7b96683b53d148bec5bb69291910104: Status 404 returned error can't find the container with id fc9968594e54777061e2d8330cf2e8e6d7b96683b53d148bec5bb69291910104
Sep 10 02:47:55.466136 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:55.466096    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7xn49" event={"ID":"e7d69b54-6b09-4496-abe6-6da05c77e958","Type":"ContainerStarted","Data":"cb2f6c4b660e364612622655cbde306962022393efc0d7b21fa4287259763333"}
Sep 10 02:47:55.467237 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:55.467215    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m5ph4" event={"ID":"aa599110-d0eb-4cc2-b4e9-e3984e9717fe","Type":"ContainerStarted","Data":"fc9968594e54777061e2d8330cf2e8e6d7b96683b53d148bec5bb69291910104"}
Sep 10 02:47:55.468276 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:55.468260    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5qtg6" event={"ID":"de3a2ed4-28df-4dea-8e14-c5dfe2981ba5","Type":"ContainerStarted","Data":"dc3319652b1cbf7344f5b6250f255676678f2dced4e2812464e51d50d1d10b25"}
Sep 10 02:47:55.483495 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:55.483472    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:56.475967 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:56.475939    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m5ph4" event={"ID":"aa599110-d0eb-4cc2-b4e9-e3984e9717fe","Type":"ContainerStarted","Data":"f727c54b1bb456bef297ed9e5e46f3e5eb8b1127eb2a783a517ae5dee49ff6e6"}
Sep 10 02:47:57.219466 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.219341    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:57.309133 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.309082    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:57.431335 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.431299    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:57.461041 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.461009    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4bt8z"]
Sep 10 02:47:57.464412 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.464386    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4bt8z"
Sep 10 02:47:57.471926 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.471869    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 02:47:57.472042 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.471871    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9ltsg\""
Sep 10 02:47:57.473504 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.473342    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 02:47:57.473504 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.473364    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 02:47:57.473504 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.473414    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 02:47:57.473737 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.473663    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 02:47:57.475746 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.475724    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 02:47:57.485496 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.485465    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m5ph4" event={"ID":"aa599110-d0eb-4cc2-b4e9-e3984e9717fe","Type":"ContainerStarted","Data":"8d17f384252864b8abc2c658a0db6633bb3ac1e5b6ed829ef55952e19677525b"}
Sep 10 02:47:57.514251 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.514198    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-m5ph4" podStartSLOduration=66.537211038 podStartE2EDuration="1m7.514179921s" totalImagesPullingTime="976.968883ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:47:57.513181871 +0000 UTC m=+67.888267794" watchObservedRunningTime="2026-09-10 02:47:57.514179921 +0000 UTC m=+67.889265843"
Sep 10 02:47:57.522348 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.522324    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:57.591144 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.590954    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9b99eefb-4544-4d4c-9087-2bd1723c04b6-root\") pod \"node-exporter-4bt8z\" (UID: \"9b99eefb-4544-4d4c-9087-2bd1723c04b6\") " pod="openshift-monitoring/node-exporter-4bt8z"
Sep 10 02:47:57.591144 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.590993    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnxdx\" (UniqueName: \"kubernetes.io/projected/9b99eefb-4544-4d4c-9087-2bd1723c04b6-kube-api-access-wnxdx\") pod \"node-exporter-4bt8z\" (UID: \"9b99eefb-4544-4d4c-9087-2bd1723c04b6\") " pod="openshift-monitoring/node-exporter-4bt8z"
Sep 10 02:47:57.591144 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.591042    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b99eefb-4544-4d4c-9087-2bd1723c04b6-sys\") pod \"node-exporter-4bt8z\" (UID: \"9b99eefb-4544-4d4c-9087-2bd1723c04b6\") " pod="openshift-monitoring/node-exporter-4bt8z"
Sep 10 02:47:57.591144 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.591073    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b99eefb-4544-4d4c-9087-2bd1723c04b6-node-exporter-tls\") pod \"node-exporter-4bt8z\" (UID: \"9b99eefb-4544-4d4c-9087-2bd1723c04b6\") " pod="openshift-monitoring/node-exporter-4bt8z"
Sep 10 02:47:57.591396 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.591145    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9b99eefb-4544-4d4c-9087-2bd1723c04b6-node-exporter-wtmp\") pod \"node-exporter-4bt8z\" (UID: \"9b99eefb-4544-4d4c-9087-2bd1723c04b6\") " pod="openshift-monitoring/node-exporter-4bt8z"
Sep 10 02:47:57.591396 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.591191    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b99eefb-4544-4d4c-9087-2bd1723c04b6-metrics-client-ca\") pod \"node-exporter-4bt8z\" (UID: \"9b99eefb-4544-4d4c-9087-2bd1723c04b6\") " pod="openshift-monitoring/node-exporter-4bt8z"
Sep 10 02:47:57.591396 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.591284    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9b99eefb-4544-4d4c-9087-2bd1723c04b6-node-exporter-textfile\") pod \"node-exporter-4bt8z\" (UID: \"9b99eefb-4544-4d4c-9087-2bd1723c04b6\") " pod="openshift-monitoring/node-exporter-4bt8z"
Sep 10 02:47:57.591396 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.591359    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b99eefb-4544-4d4c-9087-2bd1723c04b6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4bt8z\" (UID: \"9b99eefb-4544-4d4c-9087-2bd1723c04b6\") " pod="openshift-monitoring/node-exporter-4bt8z"
Sep 10 02:47:57.591396 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.591386    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9b99eefb-4544-4d4c-9087-2bd1723c04b6-node-exporter-accelerators-collector-config\") pod \"node-exporter-4bt8z\" (UID: \"9b99eefb-4544-4d4c-9087-2bd1723c04b6\") " pod="openshift-monitoring/node-exporter-4bt8z"
Sep 10 02:47:57.592410 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.592386    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:57.692060 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.692035    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b99eefb-4544-4d4c-9087-2bd1723c04b6-sys\") pod \"node-exporter-4bt8z\" (UID: \"9b99eefb-4544-4d4c-9087-2bd1723c04b6\") " pod="openshift-monitoring/node-exporter-4bt8z"
Sep 10 02:47:57.692194 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.692084    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b99eefb-4544-4d4c-9087-2bd1723c04b6-node-exporter-tls\") pod \"node-exporter-4bt8z\" (UID: \"9b99eefb-4544-4d4c-9087-2bd1723c04b6\") " pod="openshift-monitoring/node-exporter-4bt8z"
Sep 10 02:47:57.692194 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.692132    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b99eefb-4544-4d4c-9087-2bd1723c04b6-sys\") pod \"node-exporter-4bt8z\" (UID: \"9b99eefb-4544-4d4c-9087-2bd1723c04b6\") " pod="openshift-monitoring/node-exporter-4bt8z"
Sep 10 02:47:57.692194 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.692134    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9b99eefb-4544-4d4c-9087-2bd1723c04b6-node-exporter-wtmp\") pod \"node-exporter-4bt8z\" (UID: \"9b99eefb-4544-4d4c-9087-2bd1723c04b6\") " pod="openshift-monitoring/node-exporter-4bt8z"
Sep 10 02:47:57.692365 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.692223    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b99eefb-4544-4d4c-9087-2bd1723c04b6-metrics-client-ca\") pod \"node-exporter-4bt8z\" (UID: \"9b99eefb-4544-4d4c-9087-2bd1723c04b6\") " pod="openshift-monitoring/node-exporter-4bt8z"
Sep 10 02:47:57.692365 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.692257    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9b99eefb-4544-4d4c-9087-2bd1723c04b6-node-exporter-textfile\") pod \"node-exporter-4bt8z\" (UID: \"9b99eefb-4544-4d4c-9087-2bd1723c04b6\") " pod="openshift-monitoring/node-exporter-4bt8z"
Sep 10 02:47:57.692365 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.692267    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9b99eefb-4544-4d4c-9087-2bd1723c04b6-node-exporter-wtmp\") pod \"node-exporter-4bt8z\" (UID: \"9b99eefb-4544-4d4c-9087-2bd1723c04b6\") " pod="openshift-monitoring/node-exporter-4bt8z"
Sep 10 02:47:57.692365 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.692288    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b99eefb-4544-4d4c-9087-2bd1723c04b6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4bt8z\" (UID: \"9b99eefb-4544-4d4c-9087-2bd1723c04b6\") " pod="openshift-monitoring/node-exporter-4bt8z"
Sep 10 02:47:57.692365 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.692314    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9b99eefb-4544-4d4c-9087-2bd1723c04b6-node-exporter-accelerators-collector-config\") pod \"node-exporter-4bt8z\" (UID: \"9b99eefb-4544-4d4c-9087-2bd1723c04b6\") " pod="openshift-monitoring/node-exporter-4bt8z"
Sep 10 02:47:57.692365 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.692356    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9b99eefb-4544-4d4c-9087-2bd1723c04b6-root\") pod \"node-exporter-4bt8z\" (UID: \"9b99eefb-4544-4d4c-9087-2bd1723c04b6\") " pod="openshift-monitoring/node-exporter-4bt8z"
Sep 10 02:47:57.692650 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.692381    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnxdx\" (UniqueName: \"kubernetes.io/projected/9b99eefb-4544-4d4c-9087-2bd1723c04b6-kube-api-access-wnxdx\") pod \"node-exporter-4bt8z\" (UID: \"9b99eefb-4544-4d4c-9087-2bd1723c04b6\") " pod="openshift-monitoring/node-exporter-4bt8z"
Sep 10 02:47:57.692828 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.692807    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b99eefb-4544-4d4c-9087-2bd1723c04b6-metrics-client-ca\") pod \"node-exporter-4bt8z\" (UID: \"9b99eefb-4544-4d4c-9087-2bd1723c04b6\") " pod="openshift-monitoring/node-exporter-4bt8z"
Sep 10 02:47:57.692960 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.692931    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9b99eefb-4544-4d4c-9087-2bd1723c04b6-node-exporter-textfile\") pod \"node-exporter-4bt8z\" (UID: \"9b99eefb-4544-4d4c-9087-2bd1723c04b6\") " pod="openshift-monitoring/node-exporter-4bt8z"
Sep 10 02:47:57.693040 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.693004    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9b99eefb-4544-4d4c-9087-2bd1723c04b6-root\") pod \"node-exporter-4bt8z\" (UID: \"9b99eefb-4544-4d4c-9087-2bd1723c04b6\") " pod="openshift-monitoring/node-exporter-4bt8z"
Sep 10 02:47:57.693315 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.693294    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9b99eefb-4544-4d4c-9087-2bd1723c04b6-node-exporter-accelerators-collector-config\") pod \"node-exporter-4bt8z\" (UID: \"9b99eefb-4544-4d4c-9087-2bd1723c04b6\") " pod="openshift-monitoring/node-exporter-4bt8z"
Sep 10 02:47:57.694430 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.694361    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:57.696168 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.695775    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b99eefb-4544-4d4c-9087-2bd1723c04b6-node-exporter-tls\") pod \"node-exporter-4bt8z\" (UID: \"9b99eefb-4544-4d4c-9087-2bd1723c04b6\") " pod="openshift-monitoring/node-exporter-4bt8z"
Sep 10 02:47:57.696168 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.696126    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b99eefb-4544-4d4c-9087-2bd1723c04b6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4bt8z\" (UID: \"9b99eefb-4544-4d4c-9087-2bd1723c04b6\") " pod="openshift-monitoring/node-exporter-4bt8z"
Sep 10 02:47:57.703382 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.703338    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnxdx\" (UniqueName: \"kubernetes.io/projected/9b99eefb-4544-4d4c-9087-2bd1723c04b6-kube-api-access-wnxdx\") pod \"node-exporter-4bt8z\" (UID: \"9b99eefb-4544-4d4c-9087-2bd1723c04b6\") " pod="openshift-monitoring/node-exporter-4bt8z"
Sep 10 02:47:57.781283 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.781219    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4bt8z"
Sep 10 02:47:57.794010 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.793985    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:57.893612 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.893532    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:57.993805 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:57.993774    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:58.432031 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:58.431998    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:58.511481 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:58.511447    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:58.659422 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:47:58.659391    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:47:59.867193 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:47:59.867166    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b99eefb_4544_4d4c_9087_2bd1723c04b6.slice/crio-b3854d5217025c64a01e87c9bfe59ceced635ca8755a2a80c5f744b444f2fae7 WatchSource:0}: Error finding container b3854d5217025c64a01e87c9bfe59ceced635ca8755a2a80c5f744b444f2fae7: Status 404 returned error can't find the container with id b3854d5217025c64a01e87c9bfe59ceced635ca8755a2a80c5f744b444f2fae7
Sep 10 02:48:00.498369 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:00.498334    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5qtg6" event={"ID":"de3a2ed4-28df-4dea-8e14-c5dfe2981ba5","Type":"ContainerStarted","Data":"0528367af566c12a6dcda87db6dc829a0fcefeabf2cc802d0a3286accc42fd40"}
Sep 10 02:48:00.499767 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:00.499740    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4bt8z" event={"ID":"9b99eefb-4544-4d4c-9087-2bd1723c04b6","Type":"ContainerStarted","Data":"b3854d5217025c64a01e87c9bfe59ceced635ca8755a2a80c5f744b444f2fae7"}
Sep 10 02:48:00.501614 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:00.501583    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7xn49" event={"ID":"e7d69b54-6b09-4496-abe6-6da05c77e958","Type":"ContainerStarted","Data":"e9d5e5afd492d15ada814d8caeab088af4c897dd9c50b2315f9d8f7beda35174"}
Sep 10 02:48:00.501723 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:00.501713    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7xn49"
Sep 10 02:48:00.509351 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:00.509330    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:00.518962 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:00.518912    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5qtg6" podStartSLOduration=66.000812474 podStartE2EDuration="1m10.518893808s" totalImagesPullingTime="4.518081334s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:48:00.517905954 +0000 UTC m=+70.892991874" watchObservedRunningTime="2026-09-10 02:48:00.518893808 +0000 UTC m=+70.893979733"
Sep 10 02:48:00.536365 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:00.536320    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7xn49" podStartSLOduration=66.057977126 podStartE2EDuration="1m10.536304925s" totalImagesPullingTime="4.478327799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:48:00.535721766 +0000 UTC m=+70.910807698" watchObservedRunningTime="2026-09-10 02:48:00.536304925 +0000 UTC m=+70.911390846"
Sep 10 02:48:00.682731 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:00.682702    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:00.933495 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:00.933468    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:01.507248 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:01.507209    2558 generic.go:356] "Generic (PLEG): container finished" podID="9b99eefb-4544-4d4c-9087-2bd1723c04b6" containerID="9429b7131dffa132d0b07767e412dc6421af937631e826070c3fef554242c007" exitCode=0
Sep 10 02:48:01.507422 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:01.507304    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4bt8z" event={"ID":"9b99eefb-4544-4d4c-9087-2bd1723c04b6","Type":"ContainerDied","Data":"9429b7131dffa132d0b07767e412dc6421af937631e826070c3fef554242c007"}
Sep 10 02:48:02.227050 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:02.227020    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:02.326545 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:02.326520    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:02.513587 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:02.513515    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4bt8z" event={"ID":"9b99eefb-4544-4d4c-9087-2bd1723c04b6","Type":"ContainerStarted","Data":"dd8b760fb3a5d2e6164750da9d0dbcd612fb80670c5f310fa22603c70d98da5e"}
Sep 10 02:48:02.513587 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:02.513567    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4bt8z" event={"ID":"9b99eefb-4544-4d4c-9087-2bd1723c04b6","Type":"ContainerStarted","Data":"6dac45e03a248e37af083be4d7ac9b8c974008121dce59005928b4dbd9ff959c"}
Sep 10 02:48:02.542272 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:02.542224    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4bt8z" podStartSLOduration=4.656814139 podStartE2EDuration="5.542208544s" totalImagesPullingTime="824.794255ms" totalInitContainerRuntime="60.60015ms" isStatefulPod=true podCreationTimestamp="2026-09-10 02:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:48:02.540800042 +0000 UTC m=+72.915885966" watchObservedRunningTime="2026-09-10 02:48:02.542208544 +0000 UTC m=+72.917294465"
Sep 10 02:48:03.883591 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:03.883555    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:03.966032 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:03.966000    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:04.456943 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:04.456914    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:04.557590 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:04.557553    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:05.780998 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:05.780960    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:05.860205 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:05.860163    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:09.061121 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:09.061079    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:09.126784 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:09.126745    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:11.832160 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:11.832071    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:11.890979 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:11.890948    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:11.952440 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:11.952400    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:12.101243 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:12.101163    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:12.480963 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:12.480932    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:12.582581 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:12.582551    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:12.782494 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:12.782356    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:12.841899 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:12.841872    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:12.902709 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:12.902684    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:13.110690 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:13.110625    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:13.209748 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:13.209722    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:13.310223 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:13.310202    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:13.492260 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:13.492231    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:13.638949 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:13.638901    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:14.120351 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:14.120315    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:14.221417 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:14.221389    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:15.105124 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:15.105072    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:18.582395 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:18.582362    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:18.688848 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:18.688818    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:18.738704 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:18.738679    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:21.009904 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:21.009852    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:24.115469 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:24.115432    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:24.215912 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:24.215876    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:28.115005 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:28.114970    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:31.510020 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:31.509981    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7xn49"
Sep 10 02:48:31.750760 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:31.750730    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:31.843923 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:31.843858    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:34.509486 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:34.509446    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:34.610464 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:34.610431    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:39.029432 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:39.029400    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:39.086513 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:39.086485    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:39.339076 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:39.339004    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:39.402211 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:39.402176    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:39.464140 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:39.464096    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:54.073990 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:54.073944    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:48:54.174389 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:48:54.174351    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:49:13.681275 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:49:13.681235    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:49:13.785809 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:49:13.785769    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:49:24.631683 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:49:24.631650    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:49:24.701405 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:49:24.701373    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:49:25.624547 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:49:25.624489    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:49:25.687861 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:49:25.687830    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:49:35.281389 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:49:35.281352    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:49:35.381549 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:49:35.381508    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:49:35.440353 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:49:35.440322    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:49:39.076229 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:49:39.076188    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:49:39.177045 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:49:39.176999    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:50:02.717273 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:50:02.717187    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:50:02.908086 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:50:02.908051    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:50:31.918821 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:50:31.918788    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:50:32.027563 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:50:32.027533    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:50:54.078898 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:50:54.078862    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:50:54.467671 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:50:54.467642    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:50:57.914294 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:50:57.914247    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:50:58.037393 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:50:58.037364    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:50:59.765416 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:50:59.765359    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:50:59.902040 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:50:59.902012    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:51:00.141005 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:00.140923    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:51:00.308066 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:00.308034    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:51:00.340839 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:00.340809    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6q5jq"]
Sep 10 02:51:00.342893 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:00.342875    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6q5jq"
Sep 10 02:51:00.345566 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:00.345540    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 02:51:00.345655 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:00.345545    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xllqd\""
Sep 10 02:51:00.346226 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:00.346205    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 02:51:00.359472 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:00.359443    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6q5jq"]
Sep 10 02:51:00.449681 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:00.449655    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbm5r\" (UniqueName: \"kubernetes.io/projected/65487812-a756-4f2b-bd2e-5b8cea81a75a-kube-api-access-dbm5r\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6q5jq\" (UID: \"65487812-a756-4f2b-bd2e-5b8cea81a75a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6q5jq"
Sep 10 02:51:00.449767 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:00.449718    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/65487812-a756-4f2b-bd2e-5b8cea81a75a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6q5jq\" (UID: \"65487812-a756-4f2b-bd2e-5b8cea81a75a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6q5jq"
Sep 10 02:51:00.449767 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:00.449738    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/65487812-a756-4f2b-bd2e-5b8cea81a75a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6q5jq\" (UID: \"65487812-a756-4f2b-bd2e-5b8cea81a75a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6q5jq"
Sep 10 02:51:00.550953 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:00.550927    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/65487812-a756-4f2b-bd2e-5b8cea81a75a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6q5jq\" (UID: \"65487812-a756-4f2b-bd2e-5b8cea81a75a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6q5jq"
Sep 10 02:51:00.551071 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:00.550955    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/65487812-a756-4f2b-bd2e-5b8cea81a75a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6q5jq\" (UID: \"65487812-a756-4f2b-bd2e-5b8cea81a75a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6q5jq"
Sep 10 02:51:00.551071 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:00.550989    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbm5r\" (UniqueName: \"kubernetes.io/projected/65487812-a756-4f2b-bd2e-5b8cea81a75a-kube-api-access-dbm5r\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6q5jq\" (UID: \"65487812-a756-4f2b-bd2e-5b8cea81a75a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6q5jq"
Sep 10 02:51:00.551305 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:00.551286    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/65487812-a756-4f2b-bd2e-5b8cea81a75a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6q5jq\" (UID: \"65487812-a756-4f2b-bd2e-5b8cea81a75a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6q5jq"
Sep 10 02:51:00.551371 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:00.551344    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/65487812-a756-4f2b-bd2e-5b8cea81a75a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6q5jq\" (UID: \"65487812-a756-4f2b-bd2e-5b8cea81a75a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6q5jq"
Sep 10 02:51:00.573535 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:00.573508    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbm5r\" (UniqueName: \"kubernetes.io/projected/65487812-a756-4f2b-bd2e-5b8cea81a75a-kube-api-access-dbm5r\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6q5jq\" (UID: \"65487812-a756-4f2b-bd2e-5b8cea81a75a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6q5jq"
Sep 10 02:51:00.652543 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:00.652522    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6q5jq"
Sep 10 02:51:00.944184 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:00.944092    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6q5jq"]
Sep 10 02:51:00.946487 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:51:00.946452    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65487812_a756_4f2b_bd2e_5b8cea81a75a.slice/crio-d276f5de2830e2f8ba5059ad0f90b2b32407b88a147fb164bd64c1dbcae0ddcb WatchSource:0}: Error finding container d276f5de2830e2f8ba5059ad0f90b2b32407b88a147fb164bd64c1dbcae0ddcb: Status 404 returned error can't find the container with id d276f5de2830e2f8ba5059ad0f90b2b32407b88a147fb164bd64c1dbcae0ddcb
Sep 10 02:51:01.144144 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:01.144092    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6q5jq" event={"ID":"65487812-a756-4f2b-bd2e-5b8cea81a75a","Type":"ContainerStarted","Data":"d276f5de2830e2f8ba5059ad0f90b2b32407b88a147fb164bd64c1dbcae0ddcb"}
Sep 10 02:51:01.959808 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:01.959772    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:51:02.581504 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:02.581470    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:51:04.501521 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:04.501489    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:51:04.608678 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:04.608647    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:51:07.166755 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:07.166724    2558 generic.go:356] "Generic (PLEG): container finished" podID="65487812-a756-4f2b-bd2e-5b8cea81a75a" containerID="c456432313a42a3df0d395139b51fe28efd725af0cc114b151959bf26e5fa879" exitCode=0
Sep 10 02:51:07.167049 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:07.166765    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6q5jq" event={"ID":"65487812-a756-4f2b-bd2e-5b8cea81a75a","Type":"ContainerDied","Data":"c456432313a42a3df0d395139b51fe28efd725af0cc114b151959bf26e5fa879"}
Sep 10 02:51:07.777860 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:07.777827    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:51:07.875139 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:07.875091    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:51:09.175442 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:09.175410    2558 generic.go:356] "Generic (PLEG): container finished" podID="65487812-a756-4f2b-bd2e-5b8cea81a75a" containerID="311a99c7319e5a4880eb29bce07586c8c5dd28915bd013a0fa713dea0203870b" exitCode=0
Sep 10 02:51:09.175811 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:09.175454    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6q5jq" event={"ID":"65487812-a756-4f2b-bd2e-5b8cea81a75a","Type":"ContainerDied","Data":"311a99c7319e5a4880eb29bce07586c8c5dd28915bd013a0fa713dea0203870b"}
Sep 10 02:51:10.369896 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:10.369849    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:51:16.205949 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:16.205922    2558 generic.go:356] "Generic (PLEG): container finished" podID="65487812-a756-4f2b-bd2e-5b8cea81a75a" containerID="ea6aa85feea35b26f5bf79266991e06dc97122fa49bee8602586af867a35d5e5" exitCode=0
Sep 10 02:51:16.206326 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:16.205993    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6q5jq" event={"ID":"65487812-a756-4f2b-bd2e-5b8cea81a75a","Type":"ContainerDied","Data":"ea6aa85feea35b26f5bf79266991e06dc97122fa49bee8602586af867a35d5e5"}
Sep 10 02:51:17.327314 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:17.327282    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6q5jq"
Sep 10 02:51:17.372216 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:17.372187    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/65487812-a756-4f2b-bd2e-5b8cea81a75a-bundle\") pod \"65487812-a756-4f2b-bd2e-5b8cea81a75a\" (UID: \"65487812-a756-4f2b-bd2e-5b8cea81a75a\") "
Sep 10 02:51:17.372341 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:17.372253    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/65487812-a756-4f2b-bd2e-5b8cea81a75a-util\") pod \"65487812-a756-4f2b-bd2e-5b8cea81a75a\" (UID: \"65487812-a756-4f2b-bd2e-5b8cea81a75a\") "
Sep 10 02:51:17.372341 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:17.372293    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbm5r\" (UniqueName: \"kubernetes.io/projected/65487812-a756-4f2b-bd2e-5b8cea81a75a-kube-api-access-dbm5r\") pod \"65487812-a756-4f2b-bd2e-5b8cea81a75a\" (UID: \"65487812-a756-4f2b-bd2e-5b8cea81a75a\") "
Sep 10 02:51:17.372504 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:17.372482    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65487812-a756-4f2b-bd2e-5b8cea81a75a-bundle" (OuterVolumeSpecName: "bundle") pod "65487812-a756-4f2b-bd2e-5b8cea81a75a" (UID: "65487812-a756-4f2b-bd2e-5b8cea81a75a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 02:51:17.374232 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:17.374210    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65487812-a756-4f2b-bd2e-5b8cea81a75a-kube-api-access-dbm5r" (OuterVolumeSpecName: "kube-api-access-dbm5r") pod "65487812-a756-4f2b-bd2e-5b8cea81a75a" (UID: "65487812-a756-4f2b-bd2e-5b8cea81a75a"). InnerVolumeSpecName "kube-api-access-dbm5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 02:51:17.378243 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:17.378218    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65487812-a756-4f2b-bd2e-5b8cea81a75a-util" (OuterVolumeSpecName: "util") pod "65487812-a756-4f2b-bd2e-5b8cea81a75a" (UID: "65487812-a756-4f2b-bd2e-5b8cea81a75a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 02:51:17.472828 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:17.472781    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/65487812-a756-4f2b-bd2e-5b8cea81a75a-util\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:51:17.472828 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:17.472800    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbm5r\" (UniqueName: \"kubernetes.io/projected/65487812-a756-4f2b-bd2e-5b8cea81a75a-kube-api-access-dbm5r\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:51:17.472828 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:17.472811    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/65487812-a756-4f2b-bd2e-5b8cea81a75a-bundle\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:51:18.214764 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:18.214743    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6q5jq"
Sep 10 02:51:18.214857 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:18.214736    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6q5jq" event={"ID":"65487812-a756-4f2b-bd2e-5b8cea81a75a","Type":"ContainerDied","Data":"d276f5de2830e2f8ba5059ad0f90b2b32407b88a147fb164bd64c1dbcae0ddcb"}
Sep 10 02:51:18.214911 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:18.214890    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d276f5de2830e2f8ba5059ad0f90b2b32407b88a147fb164bd64c1dbcae0ddcb"
Sep 10 02:51:21.840247 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:21.840208    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:51:21.940971 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:21.940940    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:51:23.668301 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:23.668261    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-62lbv"]
Sep 10 02:51:23.668740 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:23.668500    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65487812-a756-4f2b-bd2e-5b8cea81a75a" containerName="util"
Sep 10 02:51:23.668740 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:23.668512    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="65487812-a756-4f2b-bd2e-5b8cea81a75a" containerName="util"
Sep 10 02:51:23.668740 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:23.668523    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65487812-a756-4f2b-bd2e-5b8cea81a75a" containerName="pull"
Sep 10 02:51:23.668740 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:23.668530    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="65487812-a756-4f2b-bd2e-5b8cea81a75a" containerName="pull"
Sep 10 02:51:23.668740 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:23.668546    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65487812-a756-4f2b-bd2e-5b8cea81a75a" containerName="extract"
Sep 10 02:51:23.668740 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:23.668552    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="65487812-a756-4f2b-bd2e-5b8cea81a75a" containerName="extract"
Sep 10 02:51:23.668740 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:23.668597    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="65487812-a756-4f2b-bd2e-5b8cea81a75a" containerName="extract"
Sep 10 02:51:23.671511 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:23.671491    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-62lbv"
Sep 10 02:51:23.674456 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:23.674431    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-blfp2\""
Sep 10 02:51:23.675222 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:23.675186    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 02:51:23.675346 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:23.675192    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 02:51:23.681088 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:23.681062    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-62lbv"]
Sep 10 02:51:23.813213 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:23.813189    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/737e32f5-3b9d-4cf8-addd-70f13b10c6f4-tmp\") pod \"openshift-lws-operator-fd8ccff4c-62lbv\" (UID: \"737e32f5-3b9d-4cf8-addd-70f13b10c6f4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-62lbv"
Sep 10 02:51:23.813334 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:23.813223    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5dgd\" (UniqueName: \"kubernetes.io/projected/737e32f5-3b9d-4cf8-addd-70f13b10c6f4-kube-api-access-m5dgd\") pod \"openshift-lws-operator-fd8ccff4c-62lbv\" (UID: \"737e32f5-3b9d-4cf8-addd-70f13b10c6f4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-62lbv"
Sep 10 02:51:23.913730 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:23.913700    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/737e32f5-3b9d-4cf8-addd-70f13b10c6f4-tmp\") pod \"openshift-lws-operator-fd8ccff4c-62lbv\" (UID: \"737e32f5-3b9d-4cf8-addd-70f13b10c6f4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-62lbv"
Sep 10 02:51:23.913822 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:23.913745    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5dgd\" (UniqueName: \"kubernetes.io/projected/737e32f5-3b9d-4cf8-addd-70f13b10c6f4-kube-api-access-m5dgd\") pod \"openshift-lws-operator-fd8ccff4c-62lbv\" (UID: \"737e32f5-3b9d-4cf8-addd-70f13b10c6f4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-62lbv"
Sep 10 02:51:23.914070 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:23.914048    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/737e32f5-3b9d-4cf8-addd-70f13b10c6f4-tmp\") pod \"openshift-lws-operator-fd8ccff4c-62lbv\" (UID: \"737e32f5-3b9d-4cf8-addd-70f13b10c6f4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-62lbv"
Sep 10 02:51:23.922348 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:23.922280    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5dgd\" (UniqueName: \"kubernetes.io/projected/737e32f5-3b9d-4cf8-addd-70f13b10c6f4-kube-api-access-m5dgd\") pod \"openshift-lws-operator-fd8ccff4c-62lbv\" (UID: \"737e32f5-3b9d-4cf8-addd-70f13b10c6f4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-62lbv"
Sep 10 02:51:23.981555 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:23.981523    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-62lbv"
Sep 10 02:51:24.261537 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:24.261512    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-62lbv"]
Sep 10 02:51:24.263323 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:51:24.263297    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod737e32f5_3b9d_4cf8_addd_70f13b10c6f4.slice/crio-24630ef0be3ef01f6a617b197a9647b4c720001a5355fc2515ce43917b245416 WatchSource:0}: Error finding container 24630ef0be3ef01f6a617b197a9647b4c720001a5355fc2515ce43917b245416: Status 404 returned error can't find the container with id 24630ef0be3ef01f6a617b197a9647b4c720001a5355fc2515ce43917b245416
Sep 10 02:51:25.240433 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:25.240389    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-62lbv" event={"ID":"737e32f5-3b9d-4cf8-addd-70f13b10c6f4","Type":"ContainerStarted","Data":"24630ef0be3ef01f6a617b197a9647b4c720001a5355fc2515ce43917b245416"}
Sep 10 02:51:27.249053 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:27.249016    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-62lbv" event={"ID":"737e32f5-3b9d-4cf8-addd-70f13b10c6f4","Type":"ContainerStarted","Data":"12bf1e45b722cde3b954d47e87ebcd69fd7ab8b9c7c286722bd6674c7de8f293"}
Sep 10 02:51:27.265214 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:27.265165    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-62lbv" podStartSLOduration=2.288641936 podStartE2EDuration="4.265149442s" totalImagesPullingTime="1.976507506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:51:27.264060675 +0000 UTC m=+277.639146597" watchObservedRunningTime="2026-09-10 02:51:27.265149442 +0000 UTC m=+277.640235363"
Sep 10 02:51:27.975773 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:27.975739    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:51:28.076175 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:28.076142    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:51:29.588146 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:29.588091    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq9wp"]
Sep 10 02:51:29.591562 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:29.591543    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq9wp"
Sep 10 02:51:29.593874 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:29.593847    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 02:51:29.594035 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:29.593892    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 02:51:29.594340 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:29.594323    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xllqd\""
Sep 10 02:51:29.598939 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:29.598915    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq9wp"]
Sep 10 02:51:29.649568 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:29.649546    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7aa2263-afbe-470c-9410-91d9431afe00-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq9wp\" (UID: \"e7aa2263-afbe-470c-9410-91d9431afe00\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq9wp"
Sep 10 02:51:29.649685 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:29.649585    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7aa2263-afbe-470c-9410-91d9431afe00-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq9wp\" (UID: \"e7aa2263-afbe-470c-9410-91d9431afe00\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq9wp"
Sep 10 02:51:29.649685 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:29.649619    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rqwt\" (UniqueName: \"kubernetes.io/projected/e7aa2263-afbe-470c-9410-91d9431afe00-kube-api-access-8rqwt\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq9wp\" (UID: \"e7aa2263-afbe-470c-9410-91d9431afe00\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq9wp"
Sep 10 02:51:29.750092 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:29.750066    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7aa2263-afbe-470c-9410-91d9431afe00-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq9wp\" (UID: \"e7aa2263-afbe-470c-9410-91d9431afe00\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq9wp"
Sep 10 02:51:29.750199 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:29.750116    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7aa2263-afbe-470c-9410-91d9431afe00-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq9wp\" (UID: \"e7aa2263-afbe-470c-9410-91d9431afe00\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq9wp"
Sep 10 02:51:29.750199 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:29.750146    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rqwt\" (UniqueName: \"kubernetes.io/projected/e7aa2263-afbe-470c-9410-91d9431afe00-kube-api-access-8rqwt\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq9wp\" (UID: \"e7aa2263-afbe-470c-9410-91d9431afe00\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq9wp"
Sep 10 02:51:29.750437 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:29.750413    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7aa2263-afbe-470c-9410-91d9431afe00-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq9wp\" (UID: \"e7aa2263-afbe-470c-9410-91d9431afe00\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq9wp"
Sep 10 02:51:29.750515 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:29.750498    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7aa2263-afbe-470c-9410-91d9431afe00-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq9wp\" (UID: \"e7aa2263-afbe-470c-9410-91d9431afe00\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq9wp"
Sep 10 02:51:29.758177 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:29.758153    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rqwt\" (UniqueName: \"kubernetes.io/projected/e7aa2263-afbe-470c-9410-91d9431afe00-kube-api-access-8rqwt\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq9wp\" (UID: \"e7aa2263-afbe-470c-9410-91d9431afe00\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq9wp"
Sep 10 02:51:29.782706 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:29.782682    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:51:29.901077 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:29.901028    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq9wp"
Sep 10 02:51:30.172259 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:30.172194    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq9wp"]
Sep 10 02:51:30.175384 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:51:30.175359    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7aa2263_afbe_470c_9410_91d9431afe00.slice/crio-b9efa3c20aec0bfcd296773e3ad77ed8fd4d72568a5f07850b6bfb8232133cbe WatchSource:0}: Error finding container b9efa3c20aec0bfcd296773e3ad77ed8fd4d72568a5f07850b6bfb8232133cbe: Status 404 returned error can't find the container with id b9efa3c20aec0bfcd296773e3ad77ed8fd4d72568a5f07850b6bfb8232133cbe
Sep 10 02:51:30.260596 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:30.260567    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq9wp" event={"ID":"e7aa2263-afbe-470c-9410-91d9431afe00","Type":"ContainerStarted","Data":"aeb7958961b3c0c4833addd9fb43300f9f668e9e100fff0edc4f96e74eb85610"}
Sep 10 02:51:30.260723 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:30.260618    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq9wp" event={"ID":"e7aa2263-afbe-470c-9410-91d9431afe00","Type":"ContainerStarted","Data":"b9efa3c20aec0bfcd296773e3ad77ed8fd4d72568a5f07850b6bfb8232133cbe"}
Sep 10 02:51:31.266747 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:31.266706    2558 generic.go:356] "Generic (PLEG): container finished" podID="e7aa2263-afbe-470c-9410-91d9431afe00" containerID="aeb7958961b3c0c4833addd9fb43300f9f668e9e100fff0edc4f96e74eb85610" exitCode=0
Sep 10 02:51:31.267188 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:31.266801    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq9wp" event={"ID":"e7aa2263-afbe-470c-9410-91d9431afe00","Type":"ContainerDied","Data":"aeb7958961b3c0c4833addd9fb43300f9f668e9e100fff0edc4f96e74eb85610"}
Sep 10 02:51:32.271649 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:32.271623    2558 generic.go:356] "Generic (PLEG): container finished" podID="e7aa2263-afbe-470c-9410-91d9431afe00" containerID="d5cabecfcfad6038af4470b0c0c0c1302cec4c72efcf70f6ea40511ebb1e6f10" exitCode=0
Sep 10 02:51:32.271980 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:32.271712    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq9wp" event={"ID":"e7aa2263-afbe-470c-9410-91d9431afe00","Type":"ContainerDied","Data":"d5cabecfcfad6038af4470b0c0c0c1302cec4c72efcf70f6ea40511ebb1e6f10"}
Sep 10 02:51:33.278605 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:33.278570    2558 generic.go:356] "Generic (PLEG): container finished" podID="e7aa2263-afbe-470c-9410-91d9431afe00" containerID="8f65192e739047deeef918595f3b07145148eb59a640484d94d777a9975d5098" exitCode=0
Sep 10 02:51:33.278996 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:33.278635    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq9wp" event={"ID":"e7aa2263-afbe-470c-9410-91d9431afe00","Type":"ContainerDied","Data":"8f65192e739047deeef918595f3b07145148eb59a640484d94d777a9975d5098"}
Sep 10 02:51:34.059962 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:34.059926    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:51:34.166221 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:34.166194    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:51:34.403160 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:34.403135    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq9wp"
Sep 10 02:51:34.481280 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:34.481255    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7aa2263-afbe-470c-9410-91d9431afe00-util\") pod \"e7aa2263-afbe-470c-9410-91d9431afe00\" (UID: \"e7aa2263-afbe-470c-9410-91d9431afe00\") "
Sep 10 02:51:34.481383 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:34.481314    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7aa2263-afbe-470c-9410-91d9431afe00-bundle\") pod \"e7aa2263-afbe-470c-9410-91d9431afe00\" (UID: \"e7aa2263-afbe-470c-9410-91d9431afe00\") "
Sep 10 02:51:34.481383 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:34.481347    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rqwt\" (UniqueName: \"kubernetes.io/projected/e7aa2263-afbe-470c-9410-91d9431afe00-kube-api-access-8rqwt\") pod \"e7aa2263-afbe-470c-9410-91d9431afe00\" (UID: \"e7aa2263-afbe-470c-9410-91d9431afe00\") "
Sep 10 02:51:34.481974 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:34.481950    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7aa2263-afbe-470c-9410-91d9431afe00-bundle" (OuterVolumeSpecName: "bundle") pod "e7aa2263-afbe-470c-9410-91d9431afe00" (UID: "e7aa2263-afbe-470c-9410-91d9431afe00"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 02:51:34.483346 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:34.483323    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7aa2263-afbe-470c-9410-91d9431afe00-kube-api-access-8rqwt" (OuterVolumeSpecName: "kube-api-access-8rqwt") pod "e7aa2263-afbe-470c-9410-91d9431afe00" (UID: "e7aa2263-afbe-470c-9410-91d9431afe00"). InnerVolumeSpecName "kube-api-access-8rqwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 02:51:34.487115 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:34.487078    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7aa2263-afbe-470c-9410-91d9431afe00-util" (OuterVolumeSpecName: "util") pod "e7aa2263-afbe-470c-9410-91d9431afe00" (UID: "e7aa2263-afbe-470c-9410-91d9431afe00"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 02:51:34.582414 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:34.582367    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7aa2263-afbe-470c-9410-91d9431afe00-bundle\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:51:34.582414 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:34.582385    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rqwt\" (UniqueName: \"kubernetes.io/projected/e7aa2263-afbe-470c-9410-91d9431afe00-kube-api-access-8rqwt\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:51:34.582414 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:34.582394    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7aa2263-afbe-470c-9410-91d9431afe00-util\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:51:35.291139 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:35.291080    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq9wp" event={"ID":"e7aa2263-afbe-470c-9410-91d9431afe00","Type":"ContainerDied","Data":"b9efa3c20aec0bfcd296773e3ad77ed8fd4d72568a5f07850b6bfb8232133cbe"}
Sep 10 02:51:35.291139 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:35.291138    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq9wp"
Sep 10 02:51:35.291310 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:35.291174    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9efa3c20aec0bfcd296773e3ad77ed8fd4d72568a5f07850b6bfb8232133cbe"
Sep 10 02:51:45.995141 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:45.995084    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:51:46.797755 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:46.797717    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rtjkf"]
Sep 10 02:51:46.798072 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:46.798053    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7aa2263-afbe-470c-9410-91d9431afe00" containerName="util"
Sep 10 02:51:46.798208 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:46.798076    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7aa2263-afbe-470c-9410-91d9431afe00" containerName="util"
Sep 10 02:51:46.798208 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:46.798089    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7aa2263-afbe-470c-9410-91d9431afe00" containerName="extract"
Sep 10 02:51:46.798208 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:46.798097    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7aa2263-afbe-470c-9410-91d9431afe00" containerName="extract"
Sep 10 02:51:46.798208 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:46.798127    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7aa2263-afbe-470c-9410-91d9431afe00" containerName="pull"
Sep 10 02:51:46.798208 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:46.798135    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7aa2263-afbe-470c-9410-91d9431afe00" containerName="pull"
Sep 10 02:51:46.798208 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:46.798204    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7aa2263-afbe-470c-9410-91d9431afe00" containerName="extract"
Sep 10 02:51:46.817736 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:46.817700    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rtjkf"]
Sep 10 02:51:46.817931 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:46.817859    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rtjkf"
Sep 10 02:51:46.820127 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:46.820085    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 02:51:46.820262 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:46.820147    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xllqd\""
Sep 10 02:51:46.820671 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:46.820645    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 02:51:46.960218 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:46.960187    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a649285-85c9-4258-b410-7e3367973970-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rtjkf\" (UID: \"1a649285-85c9-4258-b410-7e3367973970\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rtjkf"
Sep 10 02:51:46.960336 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:46.960227    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bvjp\" (UniqueName: \"kubernetes.io/projected/1a649285-85c9-4258-b410-7e3367973970-kube-api-access-8bvjp\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rtjkf\" (UID: \"1a649285-85c9-4258-b410-7e3367973970\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rtjkf"
Sep 10 02:51:46.960336 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:46.960283    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a649285-85c9-4258-b410-7e3367973970-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rtjkf\" (UID: \"1a649285-85c9-4258-b410-7e3367973970\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rtjkf"
Sep 10 02:51:47.020590 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.020562    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-tzqhc"]
Sep 10 02:51:47.042061 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.042034    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-tzqhc"
Sep 10 02:51:47.046077 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.046053    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 02:51:47.046281 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.046160    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 02:51:47.046281 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.046269    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-fcm22\""
Sep 10 02:51:47.046569 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.046159    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 02:51:47.046569 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.046502    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 02:51:47.046691 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.046569    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-tzqhc"]
Sep 10 02:51:47.061172 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.061152    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a649285-85c9-4258-b410-7e3367973970-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rtjkf\" (UID: \"1a649285-85c9-4258-b410-7e3367973970\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rtjkf"
Sep 10 02:51:47.061260 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.061181    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bvjp\" (UniqueName: \"kubernetes.io/projected/1a649285-85c9-4258-b410-7e3367973970-kube-api-access-8bvjp\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rtjkf\" (UID: \"1a649285-85c9-4258-b410-7e3367973970\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rtjkf"
Sep 10 02:51:47.061260 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.061209    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a649285-85c9-4258-b410-7e3367973970-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rtjkf\" (UID: \"1a649285-85c9-4258-b410-7e3367973970\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rtjkf"
Sep 10 02:51:47.061493 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.061478    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a649285-85c9-4258-b410-7e3367973970-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rtjkf\" (UID: \"1a649285-85c9-4258-b410-7e3367973970\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rtjkf"
Sep 10 02:51:47.061553 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.061535    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a649285-85c9-4258-b410-7e3367973970-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rtjkf\" (UID: \"1a649285-85c9-4258-b410-7e3367973970\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rtjkf"
Sep 10 02:51:47.076387 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.076358    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bvjp\" (UniqueName: \"kubernetes.io/projected/1a649285-85c9-4258-b410-7e3367973970-kube-api-access-8bvjp\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rtjkf\" (UID: \"1a649285-85c9-4258-b410-7e3367973970\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rtjkf"
Sep 10 02:51:47.129085 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.129065    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rtjkf"
Sep 10 02:51:47.161744 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.161721    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/eb6047a4-0717-4515-b95e-cbaea1712fe3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-tzqhc\" (UID: \"eb6047a4-0717-4515-b95e-cbaea1712fe3\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-tzqhc"
Sep 10 02:51:47.161860 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.161792    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb6047a4-0717-4515-b95e-cbaea1712fe3-webhook-cert\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-tzqhc\" (UID: \"eb6047a4-0717-4515-b95e-cbaea1712fe3\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-tzqhc"
Sep 10 02:51:47.161860 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.161826    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eb6047a4-0717-4515-b95e-cbaea1712fe3-manifests\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-tzqhc\" (UID: \"eb6047a4-0717-4515-b95e-cbaea1712fe3\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-tzqhc"
Sep 10 02:51:47.161860 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.161853    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb6047a4-0717-4515-b95e-cbaea1712fe3-tmp\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-tzqhc\" (UID: \"eb6047a4-0717-4515-b95e-cbaea1712fe3\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-tzqhc"
Sep 10 02:51:47.162021 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.161973    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhw2z\" (UniqueName: \"kubernetes.io/projected/eb6047a4-0717-4515-b95e-cbaea1712fe3-kube-api-access-dhw2z\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-tzqhc\" (UID: \"eb6047a4-0717-4515-b95e-cbaea1712fe3\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-tzqhc"
Sep 10 02:51:47.220808 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.220777    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:51:47.262898 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.262874    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb6047a4-0717-4515-b95e-cbaea1712fe3-webhook-cert\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-tzqhc\" (UID: \"eb6047a4-0717-4515-b95e-cbaea1712fe3\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-tzqhc"
Sep 10 02:51:47.262977 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.262903    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eb6047a4-0717-4515-b95e-cbaea1712fe3-manifests\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-tzqhc\" (UID: \"eb6047a4-0717-4515-b95e-cbaea1712fe3\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-tzqhc"
Sep 10 02:51:47.262977 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.262920    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb6047a4-0717-4515-b95e-cbaea1712fe3-tmp\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-tzqhc\" (UID: \"eb6047a4-0717-4515-b95e-cbaea1712fe3\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-tzqhc"
Sep 10 02:51:47.262977 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.262954    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhw2z\" (UniqueName: \"kubernetes.io/projected/eb6047a4-0717-4515-b95e-cbaea1712fe3-kube-api-access-dhw2z\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-tzqhc\" (UID: \"eb6047a4-0717-4515-b95e-cbaea1712fe3\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-tzqhc"
Sep 10 02:51:47.263135 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.263001    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/eb6047a4-0717-4515-b95e-cbaea1712fe3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-tzqhc\" (UID: \"eb6047a4-0717-4515-b95e-cbaea1712fe3\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-tzqhc"
Sep 10 02:51:47.263384 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.263360    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eb6047a4-0717-4515-b95e-cbaea1712fe3-manifests\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-tzqhc\" (UID: \"eb6047a4-0717-4515-b95e-cbaea1712fe3\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-tzqhc"
Sep 10 02:51:47.265080 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.265050    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb6047a4-0717-4515-b95e-cbaea1712fe3-tmp\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-tzqhc\" (UID: \"eb6047a4-0717-4515-b95e-cbaea1712fe3\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-tzqhc"
Sep 10 02:51:47.265332 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.265311    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb6047a4-0717-4515-b95e-cbaea1712fe3-webhook-cert\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-tzqhc\" (UID: \"eb6047a4-0717-4515-b95e-cbaea1712fe3\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-tzqhc"
Sep 10 02:51:47.265379 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.265330    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/eb6047a4-0717-4515-b95e-cbaea1712fe3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-tzqhc\" (UID: \"eb6047a4-0717-4515-b95e-cbaea1712fe3\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-tzqhc"
Sep 10 02:51:47.271672 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.271653    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhw2z\" (UniqueName: \"kubernetes.io/projected/eb6047a4-0717-4515-b95e-cbaea1712fe3-kube-api-access-dhw2z\") pod \"opendatahub-operator-controller-manager-84d77ffc9d-tzqhc\" (UID: \"eb6047a4-0717-4515-b95e-cbaea1712fe3\") " pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-tzqhc"
Sep 10 02:51:47.321042 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.320995    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:51:47.352992 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.352973    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-tzqhc"
Sep 10 02:51:47.417501 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.417479    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rtjkf"]
Sep 10 02:51:47.419852 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:51:47.419823    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a649285_85c9_4258_b410_7e3367973970.slice/crio-053e011c2c522c0cd738547b8cf692b84838916119b24f8fc2ad66f414ff924e WatchSource:0}: Error finding container 053e011c2c522c0cd738547b8cf692b84838916119b24f8fc2ad66f414ff924e: Status 404 returned error can't find the container with id 053e011c2c522c0cd738547b8cf692b84838916119b24f8fc2ad66f414ff924e
Sep 10 02:51:47.420128 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.420092    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:51:47.493735 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.493711    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:51:47.647477 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.647450    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:51:47.669613 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:47.669590    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-tzqhc"]
Sep 10 02:51:47.672515 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:51:47.672495    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb6047a4_0717_4515_b95e_cbaea1712fe3.slice/crio-f47d49467f82dcaf21c8ba49963b7b723eadb0a432929087d725da32cd683e5a WatchSource:0}: Error finding container f47d49467f82dcaf21c8ba49963b7b723eadb0a432929087d725da32cd683e5a: Status 404 returned error can't find the container with id f47d49467f82dcaf21c8ba49963b7b723eadb0a432929087d725da32cd683e5a
Sep 10 02:51:48.343023 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:48.342938    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-tzqhc" event={"ID":"eb6047a4-0717-4515-b95e-cbaea1712fe3","Type":"ContainerStarted","Data":"f47d49467f82dcaf21c8ba49963b7b723eadb0a432929087d725da32cd683e5a"}
Sep 10 02:51:48.344959 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:48.344929    2558 generic.go:356] "Generic (PLEG): container finished" podID="1a649285-85c9-4258-b410-7e3367973970" containerID="0ff213d702c77c1ba0edf58fc7fb5009023d6fdbc2d014c3f8c1736b553f9469" exitCode=0
Sep 10 02:51:48.345132 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:48.345014    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rtjkf" event={"ID":"1a649285-85c9-4258-b410-7e3367973970","Type":"ContainerDied","Data":"0ff213d702c77c1ba0edf58fc7fb5009023d6fdbc2d014c3f8c1736b553f9469"}
Sep 10 02:51:48.345132 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:48.345050    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rtjkf" event={"ID":"1a649285-85c9-4258-b410-7e3367973970","Type":"ContainerStarted","Data":"053e011c2c522c0cd738547b8cf692b84838916119b24f8fc2ad66f414ff924e"}
Sep 10 02:51:49.350954 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:49.350920    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rtjkf" event={"ID":"1a649285-85c9-4258-b410-7e3367973970","Type":"ContainerStarted","Data":"429fff7914c1e4e50078e65b9208958b70dcc87374082620e495d8c51516b6f3"}
Sep 10 02:51:50.622205 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:50.622180    2558 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 02:51:51.362048 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:51.362010    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-tzqhc" event={"ID":"eb6047a4-0717-4515-b95e-cbaea1712fe3","Type":"ContainerStarted","Data":"39ebffa081bfb1dfa8826f8c7d8e1d020e9d9cb263e8b4977bf0b42cbdd47097"}
Sep 10 02:51:51.364082 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:51.364050    2558 generic.go:356] "Generic (PLEG): container finished" podID="1a649285-85c9-4258-b410-7e3367973970" containerID="429fff7914c1e4e50078e65b9208958b70dcc87374082620e495d8c51516b6f3" exitCode=0
Sep 10 02:51:51.364217 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:51.364128    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rtjkf" event={"ID":"1a649285-85c9-4258-b410-7e3367973970","Type":"ContainerDied","Data":"429fff7914c1e4e50078e65b9208958b70dcc87374082620e495d8c51516b6f3"}
Sep 10 02:51:52.370596 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:52.370560    2558 generic.go:356] "Generic (PLEG): container finished" podID="eb6047a4-0717-4515-b95e-cbaea1712fe3" containerID="39ebffa081bfb1dfa8826f8c7d8e1d020e9d9cb263e8b4977bf0b42cbdd47097" exitCode=0
Sep 10 02:51:52.370996 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:52.370652    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-tzqhc" event={"ID":"eb6047a4-0717-4515-b95e-cbaea1712fe3","Type":"ContainerDied","Data":"39ebffa081bfb1dfa8826f8c7d8e1d020e9d9cb263e8b4977bf0b42cbdd47097"}
Sep 10 02:51:52.371213 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:52.371191    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 02:51:52.373608 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:52.373581    2558 generic.go:356] "Generic (PLEG): container finished" podID="1a649285-85c9-4258-b410-7e3367973970" containerID="975a8b96540fbdb9cc41993f7e99faf415e5a3c51dafaa83aeca4c55ea4d767e" exitCode=0
Sep 10 02:51:52.373720 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:52.373658    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rtjkf" event={"ID":"1a649285-85c9-4258-b410-7e3367973970","Type":"ContainerDied","Data":"975a8b96540fbdb9cc41993f7e99faf415e5a3c51dafaa83aeca4c55ea4d767e"}
Sep 10 02:51:53.380804 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:53.380758    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-tzqhc" event={"ID":"eb6047a4-0717-4515-b95e-cbaea1712fe3","Type":"ContainerStarted","Data":"c5ee5393bc560f4aa91cdd6a92bc086d4be01a91e77ad891086bf0feefa7ddc2"}
Sep 10 02:51:53.381290 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:53.381081    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-tzqhc"
Sep 10 02:51:53.400426 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:53.400368    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-tzqhc" podStartSLOduration=3.0353322719999998 podStartE2EDuration="7.400352533s" totalImagesPullingTime="3.284782723s" totalInitContainerRuntime="1.080237538s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:51:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:51:53.399567934 +0000 UTC m=+303.774653883" watchObservedRunningTime="2026-09-10 02:51:53.400352533 +0000 UTC m=+303.775438453"
Sep 10 02:51:53.443774 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:53.443747    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:51:53.510296 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:53.510274    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rtjkf"
Sep 10 02:51:53.613572 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:53.613547    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a649285-85c9-4258-b410-7e3367973970-bundle\") pod \"1a649285-85c9-4258-b410-7e3367973970\" (UID: \"1a649285-85c9-4258-b410-7e3367973970\") "
Sep 10 02:51:53.613691 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:53.613583    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a649285-85c9-4258-b410-7e3367973970-util\") pod \"1a649285-85c9-4258-b410-7e3367973970\" (UID: \"1a649285-85c9-4258-b410-7e3367973970\") "
Sep 10 02:51:53.613691 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:53.613647    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bvjp\" (UniqueName: \"kubernetes.io/projected/1a649285-85c9-4258-b410-7e3367973970-kube-api-access-8bvjp\") pod \"1a649285-85c9-4258-b410-7e3367973970\" (UID: \"1a649285-85c9-4258-b410-7e3367973970\") "
Sep 10 02:51:53.614327 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:53.614300    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a649285-85c9-4258-b410-7e3367973970-bundle" (OuterVolumeSpecName: "bundle") pod "1a649285-85c9-4258-b410-7e3367973970" (UID: "1a649285-85c9-4258-b410-7e3367973970"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 02:51:53.615665 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:53.615633    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a649285-85c9-4258-b410-7e3367973970-kube-api-access-8bvjp" (OuterVolumeSpecName: "kube-api-access-8bvjp") pod "1a649285-85c9-4258-b410-7e3367973970" (UID: "1a649285-85c9-4258-b410-7e3367973970"). InnerVolumeSpecName "kube-api-access-8bvjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 02:51:53.617994 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:53.617973    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a649285-85c9-4258-b410-7e3367973970-util" (OuterVolumeSpecName: "util") pod "1a649285-85c9-4258-b410-7e3367973970" (UID: "1a649285-85c9-4258-b410-7e3367973970"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 02:51:53.714158 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:53.714135    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bvjp\" (UniqueName: \"kubernetes.io/projected/1a649285-85c9-4258-b410-7e3367973970-kube-api-access-8bvjp\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:51:53.714158 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:53.714157    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a649285-85c9-4258-b410-7e3367973970-bundle\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:51:53.714288 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:53.714166    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a649285-85c9-4258-b410-7e3367973970-util\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:51:54.388443 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:54.388369    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rtjkf"
Sep 10 02:51:54.388443 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:54.388403    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rtjkf" event={"ID":"1a649285-85c9-4258-b410-7e3367973970","Type":"ContainerDied","Data":"053e011c2c522c0cd738547b8cf692b84838916119b24f8fc2ad66f414ff924e"}
Sep 10 02:51:54.388941 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:54.388456    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="053e011c2c522c0cd738547b8cf692b84838916119b24f8fc2ad66f414ff924e"
Sep 10 02:51:59.187822 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:59.187784    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:51:59.292678 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:51:59.292636    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:52:04.391073 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:04.391039    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-84d77ffc9d-tzqhc"
Sep 10 02:52:06.285318 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:06.285289    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:52:06.477884 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:06.477855    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:52:06.696460 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:06.696381    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-759bbcf6c6-xgrqq"]
Sep 10 02:52:06.696641 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:06.696627    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a649285-85c9-4258-b410-7e3367973970" containerName="extract"
Sep 10 02:52:06.696685 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:06.696644    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a649285-85c9-4258-b410-7e3367973970" containerName="extract"
Sep 10 02:52:06.696685 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:06.696661    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a649285-85c9-4258-b410-7e3367973970" containerName="util"
Sep 10 02:52:06.696685 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:06.696667    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a649285-85c9-4258-b410-7e3367973970" containerName="util"
Sep 10 02:52:06.696685 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:06.696674    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a649285-85c9-4258-b410-7e3367973970" containerName="pull"
Sep 10 02:52:06.696685 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:06.696679    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a649285-85c9-4258-b410-7e3367973970" containerName="pull"
Sep 10 02:52:06.696883 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:06.696728    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a649285-85c9-4258-b410-7e3367973970" containerName="extract"
Sep 10 02:52:06.698041 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:06.698014    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:52:06.705324 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:06.705305    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-759bbcf6c6-xgrqq"
Sep 10 02:52:06.708914 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:06.708893    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 02:52:06.709040 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:06.709019    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 02:52:06.709700 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:06.709682    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 02:52:06.709700 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:06.709683    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 02:52:06.715954 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:06.715933    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-759bbcf6c6-xgrqq"]
Sep 10 02:52:06.789436 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:06.789411    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:52:06.803545 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:06.803524    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8968a45-059c-4148-b1fc-195e51b13330-tls-certs\") pod \"kube-auth-proxy-759bbcf6c6-xgrqq\" (UID: \"e8968a45-059c-4148-b1fc-195e51b13330\") " pod="openshift-ingress/kube-auth-proxy-759bbcf6c6-xgrqq"
Sep 10 02:52:06.803654 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:06.803565    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8968a45-059c-4148-b1fc-195e51b13330-tmp\") pod \"kube-auth-proxy-759bbcf6c6-xgrqq\" (UID: \"e8968a45-059c-4148-b1fc-195e51b13330\") " pod="openshift-ingress/kube-auth-proxy-759bbcf6c6-xgrqq"
Sep 10 02:52:06.803654 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:06.803596    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjdqq\" (UniqueName: \"kubernetes.io/projected/e8968a45-059c-4148-b1fc-195e51b13330-kube-api-access-tjdqq\") pod \"kube-auth-proxy-759bbcf6c6-xgrqq\" (UID: \"e8968a45-059c-4148-b1fc-195e51b13330\") " pod="openshift-ingress/kube-auth-proxy-759bbcf6c6-xgrqq"
Sep 10 02:52:06.904740 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:06.904718    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8968a45-059c-4148-b1fc-195e51b13330-tls-certs\") pod \"kube-auth-proxy-759bbcf6c6-xgrqq\" (UID: \"e8968a45-059c-4148-b1fc-195e51b13330\") " pod="openshift-ingress/kube-auth-proxy-759bbcf6c6-xgrqq"
Sep 10 02:52:06.904841 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:06.904749    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8968a45-059c-4148-b1fc-195e51b13330-tmp\") pod \"kube-auth-proxy-759bbcf6c6-xgrqq\" (UID: \"e8968a45-059c-4148-b1fc-195e51b13330\") " pod="openshift-ingress/kube-auth-proxy-759bbcf6c6-xgrqq"
Sep 10 02:52:06.904841 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:06.904767    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjdqq\" (UniqueName: \"kubernetes.io/projected/e8968a45-059c-4148-b1fc-195e51b13330-kube-api-access-tjdqq\") pod \"kube-auth-proxy-759bbcf6c6-xgrqq\" (UID: \"e8968a45-059c-4148-b1fc-195e51b13330\") " pod="openshift-ingress/kube-auth-proxy-759bbcf6c6-xgrqq"
Sep 10 02:52:06.907179 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:06.907143    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8968a45-059c-4148-b1fc-195e51b13330-tmp\") pod \"kube-auth-proxy-759bbcf6c6-xgrqq\" (UID: \"e8968a45-059c-4148-b1fc-195e51b13330\") " pod="openshift-ingress/kube-auth-proxy-759bbcf6c6-xgrqq"
Sep 10 02:52:06.907281 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:06.907272    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8968a45-059c-4148-b1fc-195e51b13330-tls-certs\") pod \"kube-auth-proxy-759bbcf6c6-xgrqq\" (UID: \"e8968a45-059c-4148-b1fc-195e51b13330\") " pod="openshift-ingress/kube-auth-proxy-759bbcf6c6-xgrqq"
Sep 10 02:52:06.912449 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:06.912428    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjdqq\" (UniqueName: \"kubernetes.io/projected/e8968a45-059c-4148-b1fc-195e51b13330-kube-api-access-tjdqq\") pod \"kube-auth-proxy-759bbcf6c6-xgrqq\" (UID: \"e8968a45-059c-4148-b1fc-195e51b13330\") " pod="openshift-ingress/kube-auth-proxy-759bbcf6c6-xgrqq"
Sep 10 02:52:07.015330 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:07.015308    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-759bbcf6c6-xgrqq"
Sep 10 02:52:07.306165 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:07.306088    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-759bbcf6c6-xgrqq"]
Sep 10 02:52:07.308911 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:52:07.308877    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8968a45_059c_4148_b1fc_195e51b13330.slice/crio-53735a733a08bc087c04b071abf2df5812782394f6c250a82fec444287ed6c16 WatchSource:0}: Error finding container 53735a733a08bc087c04b071abf2df5812782394f6c250a82fec444287ed6c16: Status 404 returned error can't find the container with id 53735a733a08bc087c04b071abf2df5812782394f6c250a82fec444287ed6c16
Sep 10 02:52:07.439427 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:07.439399    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-759bbcf6c6-xgrqq" event={"ID":"e8968a45-059c-4148-b1fc-195e51b13330","Type":"ContainerStarted","Data":"53735a733a08bc087c04b071abf2df5812782394f6c250a82fec444287ed6c16"}
Sep 10 02:52:07.936283 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:07.936243    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:52:07.941575 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:07.941488    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg"]
Sep 10 02:52:07.944945 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:07.944921    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg"
Sep 10 02:52:07.947288 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:07.946988    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 10 02:52:07.947288 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:07.947053    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 10 02:52:07.947288 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:07.947215    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 10 02:52:07.947491 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:07.947291    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 02:52:07.947491 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:07.947218    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-tg48t\""
Sep 10 02:52:07.958365 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:07.958308    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg"]
Sep 10 02:52:08.036496 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:08.036466    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:52:08.113382 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:08.113348    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b594797c-654d-4b37-818b-a134d01ae19c-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-82qfg\" (UID: \"b594797c-654d-4b37-818b-a134d01ae19c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg"
Sep 10 02:52:08.113538 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:08.113416    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b594797c-654d-4b37-818b-a134d01ae19c-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-82qfg\" (UID: \"b594797c-654d-4b37-818b-a134d01ae19c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg"
Sep 10 02:52:08.113538 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:08.113455    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b594797c-654d-4b37-818b-a134d01ae19c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-82qfg\" (UID: \"b594797c-654d-4b37-818b-a134d01ae19c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg"
Sep 10 02:52:08.113538 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:08.113493    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b594797c-654d-4b37-818b-a134d01ae19c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-82qfg\" (UID: \"b594797c-654d-4b37-818b-a134d01ae19c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg"
Sep 10 02:52:08.113538 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:08.113518    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b594797c-654d-4b37-818b-a134d01ae19c-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-82qfg\" (UID: \"b594797c-654d-4b37-818b-a134d01ae19c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg"
Sep 10 02:52:08.113752 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:08.113542    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b594797c-654d-4b37-818b-a134d01ae19c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-82qfg\" (UID: \"b594797c-654d-4b37-818b-a134d01ae19c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg"
Sep 10 02:52:08.113752 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:08.113562    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfvx2\" (UniqueName: \"kubernetes.io/projected/b594797c-654d-4b37-818b-a134d01ae19c-kube-api-access-vfvx2\") pod \"istiod-openshift-gateway-7b7995d5cd-82qfg\" (UID: \"b594797c-654d-4b37-818b-a134d01ae19c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg"
Sep 10 02:52:08.214479 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:08.214446    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b594797c-654d-4b37-818b-a134d01ae19c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-82qfg\" (UID: \"b594797c-654d-4b37-818b-a134d01ae19c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg"
Sep 10 02:52:08.214627 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:08.214499    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b594797c-654d-4b37-818b-a134d01ae19c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-82qfg\" (UID: \"b594797c-654d-4b37-818b-a134d01ae19c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg"
Sep 10 02:52:08.214627 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:08.214528    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b594797c-654d-4b37-818b-a134d01ae19c-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-82qfg\" (UID: \"b594797c-654d-4b37-818b-a134d01ae19c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg"
Sep 10 02:52:08.214750 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:08.214645    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b594797c-654d-4b37-818b-a134d01ae19c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-82qfg\" (UID: \"b594797c-654d-4b37-818b-a134d01ae19c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg"
Sep 10 02:52:08.214750 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:08.214682    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfvx2\" (UniqueName: \"kubernetes.io/projected/b594797c-654d-4b37-818b-a134d01ae19c-kube-api-access-vfvx2\") pod \"istiod-openshift-gateway-7b7995d5cd-82qfg\" (UID: \"b594797c-654d-4b37-818b-a134d01ae19c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg"
Sep 10 02:52:08.214750 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:08.214731    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b594797c-654d-4b37-818b-a134d01ae19c-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-82qfg\" (UID: \"b594797c-654d-4b37-818b-a134d01ae19c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg"
Sep 10 02:52:08.214906 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:08.214780    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b594797c-654d-4b37-818b-a134d01ae19c-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-82qfg\" (UID: \"b594797c-654d-4b37-818b-a134d01ae19c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg"
Sep 10 02:52:08.215285 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:08.215260    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b594797c-654d-4b37-818b-a134d01ae19c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-82qfg\" (UID: \"b594797c-654d-4b37-818b-a134d01ae19c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg"
Sep 10 02:52:08.217460 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:08.217415    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b594797c-654d-4b37-818b-a134d01ae19c-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-82qfg\" (UID: \"b594797c-654d-4b37-818b-a134d01ae19c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg"
Sep 10 02:52:08.217549 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:08.217466    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b594797c-654d-4b37-818b-a134d01ae19c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-82qfg\" (UID: \"b594797c-654d-4b37-818b-a134d01ae19c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg"
Sep 10 02:52:08.217682 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:08.217656    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b594797c-654d-4b37-818b-a134d01ae19c-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-82qfg\" (UID: \"b594797c-654d-4b37-818b-a134d01ae19c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg"
Sep 10 02:52:08.218585 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:08.218553    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b594797c-654d-4b37-818b-a134d01ae19c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-82qfg\" (UID: \"b594797c-654d-4b37-818b-a134d01ae19c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg"
Sep 10 02:52:08.222481 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:08.222454    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b594797c-654d-4b37-818b-a134d01ae19c-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-82qfg\" (UID: \"b594797c-654d-4b37-818b-a134d01ae19c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg"
Sep 10 02:52:08.222875 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:08.222851    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfvx2\" (UniqueName: \"kubernetes.io/projected/b594797c-654d-4b37-818b-a134d01ae19c-kube-api-access-vfvx2\") pod \"istiod-openshift-gateway-7b7995d5cd-82qfg\" (UID: \"b594797c-654d-4b37-818b-a134d01ae19c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg"
Sep 10 02:52:08.257691 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:08.257662    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg"
Sep 10 02:52:08.544774 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:08.544743    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg"]
Sep 10 02:52:08.547668 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:52:08.547508    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb594797c_654d_4b37_818b_a134d01ae19c.slice/crio-81e45baa16d893ae58037675ae7af6e57a401abe3af54f01da50d6a8329a4047 WatchSource:0}: Error finding container 81e45baa16d893ae58037675ae7af6e57a401abe3af54f01da50d6a8329a4047: Status 404 returned error can't find the container with id 81e45baa16d893ae58037675ae7af6e57a401abe3af54f01da50d6a8329a4047
Sep 10 02:52:08.854585 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:08.854452    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:52:09.451639 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:09.451605    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg" event={"ID":"b594797c-654d-4b37-818b-a134d01ae19c","Type":"ContainerStarted","Data":"81e45baa16d893ae58037675ae7af6e57a401abe3af54f01da50d6a8329a4047"}
Sep 10 02:52:10.458476 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:10.458439    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-759bbcf6c6-xgrqq" event={"ID":"e8968a45-059c-4148-b1fc-195e51b13330","Type":"ContainerStarted","Data":"99713b2f97dd02994b61da54a1bf4a331c8f654ee4ec06e827d7c792d19d1dd1"}
Sep 10 02:52:10.474415 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:10.474369    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-759bbcf6c6-xgrqq" podStartSLOduration=1.7595485100000001 podStartE2EDuration="4.474349683s" totalImagesPullingTime="2.714801173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:52:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:52:10.473241799 +0000 UTC m=+320.848327732" watchObservedRunningTime="2026-09-10 02:52:10.474349683 +0000 UTC m=+320.849435627"
Sep 10 02:52:11.744529 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:11.744495    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:52:11.744772 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:11.744562    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 02:52:12.257822 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:12.257782    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:52:12.328845 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:12.328811    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:52:12.470166 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:12.470129    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg" event={"ID":"b594797c-654d-4b37-818b-a134d01ae19c","Type":"ContainerStarted","Data":"0761cc4f66b0dd8ee94d35926a941e850d0904528185ef285f7157ef98df24b2"}
Sep 10 02:52:12.470366 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:12.470343    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg"
Sep 10 02:52:12.472340 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:12.472308    2558 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-82qfg container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 02:52:12.472469 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:12.472368    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg" podUID="b594797c-654d-4b37-818b-a134d01ae19c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 02:52:12.494654 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:12.494608    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg" podStartSLOduration=2.300499872 podStartE2EDuration="5.494596328s" totalImagesPullingTime="3.194096456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:52:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:52:12.493427104 +0000 UTC m=+322.868513055" watchObservedRunningTime="2026-09-10 02:52:12.494596328 +0000 UTC m=+322.869682248"
Sep 10 02:52:12.850319 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:12.850287    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:52:13.053853 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:13.053821    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:52:13.146764 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:13.146685    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:52:13.474490 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:13.474459    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg"
Sep 10 02:52:17.332760 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:17.332730    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mzhlk"]
Sep 10 02:52:17.337010 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:17.336987    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mzhlk"
Sep 10 02:52:17.339222 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:17.339149    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-4vt8f\""
Sep 10 02:52:17.339222 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:17.339195    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 10 02:52:17.345828 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:17.345797    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mzhlk"]
Sep 10 02:52:17.354257 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:17.354235    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:52:17.420864 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:17.420835    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:52:17.477041 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:17.477008    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7eb27485-1656-4e2d-b38f-e59551e49c8b-cert\") pod \"odh-model-controller-7998c67dc4-mzhlk\" (UID: \"7eb27485-1656-4e2d-b38f-e59551e49c8b\") " pod="opendatahub/odh-model-controller-7998c67dc4-mzhlk"
Sep 10 02:52:17.477217 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:17.477065    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqgxt\" (UniqueName: \"kubernetes.io/projected/7eb27485-1656-4e2d-b38f-e59551e49c8b-kube-api-access-xqgxt\") pod \"odh-model-controller-7998c67dc4-mzhlk\" (UID: \"7eb27485-1656-4e2d-b38f-e59551e49c8b\") " pod="opendatahub/odh-model-controller-7998c67dc4-mzhlk"
Sep 10 02:52:17.518571 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:17.518537    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:52:17.578561 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:17.578525    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqgxt\" (UniqueName: \"kubernetes.io/projected/7eb27485-1656-4e2d-b38f-e59551e49c8b-kube-api-access-xqgxt\") pod \"odh-model-controller-7998c67dc4-mzhlk\" (UID: \"7eb27485-1656-4e2d-b38f-e59551e49c8b\") " pod="opendatahub/odh-model-controller-7998c67dc4-mzhlk"
Sep 10 02:52:17.578742 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:17.578704    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7eb27485-1656-4e2d-b38f-e59551e49c8b-cert\") pod \"odh-model-controller-7998c67dc4-mzhlk\" (UID: \"7eb27485-1656-4e2d-b38f-e59551e49c8b\") " pod="opendatahub/odh-model-controller-7998c67dc4-mzhlk"
Sep 10 02:52:17.581210 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:17.581183    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7eb27485-1656-4e2d-b38f-e59551e49c8b-cert\") pod \"odh-model-controller-7998c67dc4-mzhlk\" (UID: \"7eb27485-1656-4e2d-b38f-e59551e49c8b\") " pod="opendatahub/odh-model-controller-7998c67dc4-mzhlk"
Sep 10 02:52:17.587005 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:17.586950    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqgxt\" (UniqueName: \"kubernetes.io/projected/7eb27485-1656-4e2d-b38f-e59551e49c8b-kube-api-access-xqgxt\") pod \"odh-model-controller-7998c67dc4-mzhlk\" (UID: \"7eb27485-1656-4e2d-b38f-e59551e49c8b\") " pod="opendatahub/odh-model-controller-7998c67dc4-mzhlk"
Sep 10 02:52:17.649272 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:17.649246    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mzhlk"
Sep 10 02:52:17.732748 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:17.732721    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-6dv7b"]
Sep 10 02:52:17.736021 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:17.735997    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-6dv7b"
Sep 10 02:52:17.739218 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:17.739188    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 10 02:52:17.752261 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:17.752241    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-6dv7b"]
Sep 10 02:52:17.758700 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:17.758676    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:52:17.854919 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:17.854863    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:52:17.882048 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:17.882019    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c8mq\" (UniqueName: \"kubernetes.io/projected/65045f32-f1e0-4756-b1dc-d36d99c20175-kube-api-access-2c8mq\") pod \"model-serving-api-54d89bd5fc-6dv7b\" (UID: \"65045f32-f1e0-4756-b1dc-d36d99c20175\") " pod="opendatahub/model-serving-api-54d89bd5fc-6dv7b"
Sep 10 02:52:17.882189 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:17.882070    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65045f32-f1e0-4756-b1dc-d36d99c20175-tls-certs\") pod \"model-serving-api-54d89bd5fc-6dv7b\" (UID: \"65045f32-f1e0-4756-b1dc-d36d99c20175\") " pod="opendatahub/model-serving-api-54d89bd5fc-6dv7b"
Sep 10 02:52:17.925250 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:17.925214    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mzhlk"]
Sep 10 02:52:17.930143 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:52:17.930092    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eb27485_1656_4e2d_b38f_e59551e49c8b.slice/crio-b321a978a3d73e14e29aa3e6a15a6336ad6f8e619b4de7b22639475f59f9c5f4 WatchSource:0}: Error finding container b321a978a3d73e14e29aa3e6a15a6336ad6f8e619b4de7b22639475f59f9c5f4: Status 404 returned error can't find the container with id b321a978a3d73e14e29aa3e6a15a6336ad6f8e619b4de7b22639475f59f9c5f4
Sep 10 02:52:17.983156 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:17.983117    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c8mq\" (UniqueName: \"kubernetes.io/projected/65045f32-f1e0-4756-b1dc-d36d99c20175-kube-api-access-2c8mq\") pod \"model-serving-api-54d89bd5fc-6dv7b\" (UID: \"65045f32-f1e0-4756-b1dc-d36d99c20175\") " pod="opendatahub/model-serving-api-54d89bd5fc-6dv7b"
Sep 10 02:52:17.983295 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:17.983172    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65045f32-f1e0-4756-b1dc-d36d99c20175-tls-certs\") pod \"model-serving-api-54d89bd5fc-6dv7b\" (UID: \"65045f32-f1e0-4756-b1dc-d36d99c20175\") " pod="opendatahub/model-serving-api-54d89bd5fc-6dv7b"
Sep 10 02:52:17.985462 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:17.985439    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65045f32-f1e0-4756-b1dc-d36d99c20175-tls-certs\") pod \"model-serving-api-54d89bd5fc-6dv7b\" (UID: \"65045f32-f1e0-4756-b1dc-d36d99c20175\") " pod="opendatahub/model-serving-api-54d89bd5fc-6dv7b"
Sep 10 02:52:17.989658 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:17.989638    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c8mq\" (UniqueName: \"kubernetes.io/projected/65045f32-f1e0-4756-b1dc-d36d99c20175-kube-api-access-2c8mq\") pod \"model-serving-api-54d89bd5fc-6dv7b\" (UID: \"65045f32-f1e0-4756-b1dc-d36d99c20175\") " pod="opendatahub/model-serving-api-54d89bd5fc-6dv7b"
Sep 10 02:52:18.048208 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:18.048182    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-6dv7b"
Sep 10 02:52:18.330872 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:18.330830    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-6dv7b"]
Sep 10 02:52:18.333466 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:52:18.333442    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65045f32_f1e0_4756_b1dc_d36d99c20175.slice/crio-b8bb9e49c3fff27f95774ba42e78ab8e649bfbb0c887830456a9154193007e79 WatchSource:0}: Error finding container b8bb9e49c3fff27f95774ba42e78ab8e649bfbb0c887830456a9154193007e79: Status 404 returned error can't find the container with id b8bb9e49c3fff27f95774ba42e78ab8e649bfbb0c887830456a9154193007e79
Sep 10 02:52:18.497540 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:18.497498    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-6dv7b" event={"ID":"65045f32-f1e0-4756-b1dc-d36d99c20175","Type":"ContainerStarted","Data":"b8bb9e49c3fff27f95774ba42e78ab8e649bfbb0c887830456a9154193007e79"}
Sep 10 02:52:18.498885 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:18.498854    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mzhlk" event={"ID":"7eb27485-1656-4e2d-b38f-e59551e49c8b","Type":"ContainerStarted","Data":"b321a978a3d73e14e29aa3e6a15a6336ad6f8e619b4de7b22639475f59f9c5f4"}
Sep 10 02:52:21.516844 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:21.516811    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-6dv7b" event={"ID":"65045f32-f1e0-4756-b1dc-d36d99c20175","Type":"ContainerStarted","Data":"c6560dc9fdfa17a5e6147904ba367de1beb7f43d0e50032ed7638daa7f367a91"}
Sep 10 02:52:21.517265 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:21.516909    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-6dv7b"
Sep 10 02:52:21.518401 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:21.518375    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mzhlk" event={"ID":"7eb27485-1656-4e2d-b38f-e59551e49c8b","Type":"ContainerStarted","Data":"956c48d4b6e28e6ae917616085da0314cfc916bfe8009dad3034156a55fd1e37"}
Sep 10 02:52:21.518532 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:21.518508    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-mzhlk"
Sep 10 02:52:21.532698 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:21.532655    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-6dv7b" podStartSLOduration=1.806151131 podStartE2EDuration="4.532643835s" totalImagesPullingTime="2.726492704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:52:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:52:21.531244208 +0000 UTC m=+331.906330143" watchObservedRunningTime="2026-09-10 02:52:21.532643835 +0000 UTC m=+331.907729756"
Sep 10 02:52:21.546076 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:21.546036    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-mzhlk" podStartSLOduration=1.419978245 podStartE2EDuration="4.54602377s" totalImagesPullingTime="3.126045525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:52:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:52:21.544963461 +0000 UTC m=+331.920049383" watchObservedRunningTime="2026-09-10 02:52:21.54602377 +0000 UTC m=+331.921109691"
Sep 10 02:52:26.991636 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:26.991598    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:52:27.026572 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:27.026542    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-mxtj6"]
Sep 10 02:52:27.032703 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:27.032683    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mxtj6"
Sep 10 02:52:27.034592 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:27.034568    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-v8f8w"]
Sep 10 02:52:27.034721 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:27.034703    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 10 02:52:27.034799 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:27.034782    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-mhnjc\""
Sep 10 02:52:27.036526 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:27.036508    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v8f8w"
Sep 10 02:52:27.038302 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:27.038281    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 10 02:52:27.038383 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:27.038370    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-xkmp6\""
Sep 10 02:52:27.041180 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:27.041157    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-mxtj6"]
Sep 10 02:52:27.048503 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:27.048482    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-v8f8w"]
Sep 10 02:52:27.057152 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:27.057119    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b5dd34b-715d-469c-8e64-6406b64cd509-cert\") pod \"kserve-controller-manager-d6f4b65b8-mxtj6\" (UID: \"4b5dd34b-715d-469c-8e64-6406b64cd509\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mxtj6"
Sep 10 02:52:27.057240 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:27.057199    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lftn\" (UniqueName: \"kubernetes.io/projected/6a49b00d-e027-421f-883e-3cacf773af95-kube-api-access-8lftn\") pod \"llmisvc-controller-manager-777c6bd7d5-v8f8w\" (UID: \"6a49b00d-e027-421f-883e-3cacf773af95\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v8f8w"
Sep 10 02:52:27.057340 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:27.057241    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z57r\" (UniqueName: \"kubernetes.io/projected/4b5dd34b-715d-469c-8e64-6406b64cd509-kube-api-access-4z57r\") pod \"kserve-controller-manager-d6f4b65b8-mxtj6\" (UID: \"4b5dd34b-715d-469c-8e64-6406b64cd509\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mxtj6"
Sep 10 02:52:27.057340 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:27.057269    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a49b00d-e027-421f-883e-3cacf773af95-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-v8f8w\" (UID: \"6a49b00d-e027-421f-883e-3cacf773af95\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v8f8w"
Sep 10 02:52:27.073729 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:27.073707    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:52:27.157778 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:27.157746    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b5dd34b-715d-469c-8e64-6406b64cd509-cert\") pod \"kserve-controller-manager-d6f4b65b8-mxtj6\" (UID: \"4b5dd34b-715d-469c-8e64-6406b64cd509\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mxtj6"
Sep 10 02:52:27.157905 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:27.157820    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lftn\" (UniqueName: \"kubernetes.io/projected/6a49b00d-e027-421f-883e-3cacf773af95-kube-api-access-8lftn\") pod \"llmisvc-controller-manager-777c6bd7d5-v8f8w\" (UID: \"6a49b00d-e027-421f-883e-3cacf773af95\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v8f8w"
Sep 10 02:52:27.157905 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:27.157848    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z57r\" (UniqueName: \"kubernetes.io/projected/4b5dd34b-715d-469c-8e64-6406b64cd509-kube-api-access-4z57r\") pod \"kserve-controller-manager-d6f4b65b8-mxtj6\" (UID: \"4b5dd34b-715d-469c-8e64-6406b64cd509\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mxtj6"
Sep 10 02:52:27.157905 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:27.157864    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a49b00d-e027-421f-883e-3cacf773af95-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-v8f8w\" (UID: \"6a49b00d-e027-421f-883e-3cacf773af95\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v8f8w"
Sep 10 02:52:27.157905 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:52:27.157898    2558 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 10 02:52:27.158094 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:52:27.157969    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b5dd34b-715d-469c-8e64-6406b64cd509-cert podName:4b5dd34b-715d-469c-8e64-6406b64cd509 nodeName:}" failed. No retries permitted until 2026-09-10 02:52:27.657951899 +0000 UTC m=+338.033037801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/4b5dd34b-715d-469c-8e64-6406b64cd509-cert") pod "kserve-controller-manager-d6f4b65b8-mxtj6" (UID: "4b5dd34b-715d-469c-8e64-6406b64cd509") : secret "kserve-webhook-server-cert" not found
Sep 10 02:52:27.158094 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:52:27.158000    2558 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 10 02:52:27.158094 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:52:27.158060    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a49b00d-e027-421f-883e-3cacf773af95-cert podName:6a49b00d-e027-421f-883e-3cacf773af95 nodeName:}" failed. No retries permitted until 2026-09-10 02:52:27.658042563 +0000 UTC m=+338.033128478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6a49b00d-e027-421f-883e-3cacf773af95-cert") pod "llmisvc-controller-manager-777c6bd7d5-v8f8w" (UID: "6a49b00d-e027-421f-883e-3cacf773af95") : secret "llmisvc-webhook-server-cert" not found
Sep 10 02:52:27.167894 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:27.167867    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lftn\" (UniqueName: \"kubernetes.io/projected/6a49b00d-e027-421f-883e-3cacf773af95-kube-api-access-8lftn\") pod \"llmisvc-controller-manager-777c6bd7d5-v8f8w\" (UID: \"6a49b00d-e027-421f-883e-3cacf773af95\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v8f8w"
Sep 10 02:52:27.170836 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:27.170813    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z57r\" (UniqueName: \"kubernetes.io/projected/4b5dd34b-715d-469c-8e64-6406b64cd509-kube-api-access-4z57r\") pod \"kserve-controller-manager-d6f4b65b8-mxtj6\" (UID: \"4b5dd34b-715d-469c-8e64-6406b64cd509\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mxtj6"
Sep 10 02:52:27.267690 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:27.267639    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:52:27.629429 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:27.629349    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:52:27.662743 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:27.662716    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a49b00d-e027-421f-883e-3cacf773af95-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-v8f8w\" (UID: \"6a49b00d-e027-421f-883e-3cacf773af95\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v8f8w"
Sep 10 02:52:27.662850 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:27.662761    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b5dd34b-715d-469c-8e64-6406b64cd509-cert\") pod \"kserve-controller-manager-d6f4b65b8-mxtj6\" (UID: \"4b5dd34b-715d-469c-8e64-6406b64cd509\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mxtj6"
Sep 10 02:52:27.665152 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:27.665125    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b5dd34b-715d-469c-8e64-6406b64cd509-cert\") pod \"kserve-controller-manager-d6f4b65b8-mxtj6\" (UID: \"4b5dd34b-715d-469c-8e64-6406b64cd509\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mxtj6"
Sep 10 02:52:27.665234 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:27.665188    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a49b00d-e027-421f-883e-3cacf773af95-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-v8f8w\" (UID: \"6a49b00d-e027-421f-883e-3cacf773af95\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v8f8w"
Sep 10 02:52:27.945563 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:27.945490    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mxtj6"
Sep 10 02:52:27.956183 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:27.956048    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v8f8w"
Sep 10 02:52:28.240594 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:28.240564    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-mxtj6"]
Sep 10 02:52:28.242165 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:28.242141    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-v8f8w"]
Sep 10 02:52:28.243015 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:52:28.242988    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6a49b00d_e027_421f_883e_3cacf773af95.slice/crio-2bdd4cda21c554cc1e0adc79026211c673653130f269863af7b47cf4d40a765d WatchSource:0}: Error finding container 2bdd4cda21c554cc1e0adc79026211c673653130f269863af7b47cf4d40a765d: Status 404 returned error can't find the container with id 2bdd4cda21c554cc1e0adc79026211c673653130f269863af7b47cf4d40a765d
Sep 10 02:52:28.243544 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:52:28.243519    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b5dd34b_715d_469c_8e64_6406b64cd509.slice/crio-4c3f07ee0a4a269cf26e4dab5744e04359bb7c4c15764125664dcd5fa2a407b3 WatchSource:0}: Error finding container 4c3f07ee0a4a269cf26e4dab5744e04359bb7c4c15764125664dcd5fa2a407b3: Status 404 returned error can't find the container with id 4c3f07ee0a4a269cf26e4dab5744e04359bb7c4c15764125664dcd5fa2a407b3
Sep 10 02:52:28.328612 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:28.328569    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:52:28.397360 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:28.397331    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:52:28.550731 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:28.550655    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v8f8w" event={"ID":"6a49b00d-e027-421f-883e-3cacf773af95","Type":"ContainerStarted","Data":"2bdd4cda21c554cc1e0adc79026211c673653130f269863af7b47cf4d40a765d"}
Sep 10 02:52:28.551957 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:28.551933    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mxtj6" event={"ID":"4b5dd34b-715d-469c-8e64-6406b64cd509","Type":"ContainerStarted","Data":"4c3f07ee0a4a269cf26e4dab5744e04359bb7c4c15764125664dcd5fa2a407b3"}
Sep 10 02:52:28.977831 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:28.977801    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg"]
Sep 10 02:52:28.978068 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:28.978040    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg" podUID="b594797c-654d-4b37-818b-a134d01ae19c" containerName="discovery" containerID="cri-o://0761cc4f66b0dd8ee94d35926a941e850d0904528185ef285f7157ef98df24b2" gracePeriod=30
Sep 10 02:52:29.268351 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:29.268310    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg"
Sep 10 02:52:29.375523 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:29.375491    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b594797c-654d-4b37-818b-a134d01ae19c-istio-token\") pod \"b594797c-654d-4b37-818b-a134d01ae19c\" (UID: \"b594797c-654d-4b37-818b-a134d01ae19c\") "
Sep 10 02:52:29.375703 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:29.375548    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b594797c-654d-4b37-818b-a134d01ae19c-cacerts\") pod \"b594797c-654d-4b37-818b-a134d01ae19c\" (UID: \"b594797c-654d-4b37-818b-a134d01ae19c\") "
Sep 10 02:52:29.375703 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:29.375578    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b594797c-654d-4b37-818b-a134d01ae19c-istio-csr-ca-configmap\") pod \"b594797c-654d-4b37-818b-a134d01ae19c\" (UID: \"b594797c-654d-4b37-818b-a134d01ae19c\") "
Sep 10 02:52:29.375703 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:29.375626    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b594797c-654d-4b37-818b-a134d01ae19c-local-certs\") pod \"b594797c-654d-4b37-818b-a134d01ae19c\" (UID: \"b594797c-654d-4b37-818b-a134d01ae19c\") "
Sep 10 02:52:29.375703 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:29.375678    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b594797c-654d-4b37-818b-a134d01ae19c-istio-kubeconfig\") pod \"b594797c-654d-4b37-818b-a134d01ae19c\" (UID: \"b594797c-654d-4b37-818b-a134d01ae19c\") "
Sep 10 02:52:29.375910 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:29.375706    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfvx2\" (UniqueName: \"kubernetes.io/projected/b594797c-654d-4b37-818b-a134d01ae19c-kube-api-access-vfvx2\") pod \"b594797c-654d-4b37-818b-a134d01ae19c\" (UID: \"b594797c-654d-4b37-818b-a134d01ae19c\") "
Sep 10 02:52:29.375910 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:29.375737    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b594797c-654d-4b37-818b-a134d01ae19c-istio-csr-dns-cert\") pod \"b594797c-654d-4b37-818b-a134d01ae19c\" (UID: \"b594797c-654d-4b37-818b-a134d01ae19c\") "
Sep 10 02:52:29.376490 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:29.376430    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b594797c-654d-4b37-818b-a134d01ae19c-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "b594797c-654d-4b37-818b-a134d01ae19c" (UID: "b594797c-654d-4b37-818b-a134d01ae19c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 02:52:29.379490 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:29.378953    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b594797c-654d-4b37-818b-a134d01ae19c-istio-token" (OuterVolumeSpecName: "istio-token") pod "b594797c-654d-4b37-818b-a134d01ae19c" (UID: "b594797c-654d-4b37-818b-a134d01ae19c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 02:52:29.379490 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:29.379179    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b594797c-654d-4b37-818b-a134d01ae19c-kube-api-access-vfvx2" (OuterVolumeSpecName: "kube-api-access-vfvx2") pod "b594797c-654d-4b37-818b-a134d01ae19c" (UID: "b594797c-654d-4b37-818b-a134d01ae19c"). InnerVolumeSpecName "kube-api-access-vfvx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 02:52:29.379490 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:29.379308    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b594797c-654d-4b37-818b-a134d01ae19c-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "b594797c-654d-4b37-818b-a134d01ae19c" (UID: "b594797c-654d-4b37-818b-a134d01ae19c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 02:52:29.379490 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:29.379384    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b594797c-654d-4b37-818b-a134d01ae19c-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "b594797c-654d-4b37-818b-a134d01ae19c" (UID: "b594797c-654d-4b37-818b-a134d01ae19c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 02:52:29.379786 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:29.379551    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b594797c-654d-4b37-818b-a134d01ae19c-local-certs" (OuterVolumeSpecName: "local-certs") pod "b594797c-654d-4b37-818b-a134d01ae19c" (UID: "b594797c-654d-4b37-818b-a134d01ae19c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 02:52:29.380125 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:29.380088    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b594797c-654d-4b37-818b-a134d01ae19c-cacerts" (OuterVolumeSpecName: "cacerts") pod "b594797c-654d-4b37-818b-a134d01ae19c" (UID: "b594797c-654d-4b37-818b-a134d01ae19c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 02:52:29.477071 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:29.477035    2558 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b594797c-654d-4b37-818b-a134d01ae19c-istio-csr-dns-cert\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:52:29.477071 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:29.477072    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b594797c-654d-4b37-818b-a134d01ae19c-istio-token\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:52:29.477287 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:29.477087    2558 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b594797c-654d-4b37-818b-a134d01ae19c-cacerts\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:52:29.477287 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:29.477128    2558 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b594797c-654d-4b37-818b-a134d01ae19c-istio-csr-ca-configmap\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:52:29.477287 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:29.477146    2558 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b594797c-654d-4b37-818b-a134d01ae19c-local-certs\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:52:29.477287 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:29.477158    2558 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b594797c-654d-4b37-818b-a134d01ae19c-istio-kubeconfig\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:52:29.477287 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:29.477170    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfvx2\" (UniqueName: \"kubernetes.io/projected/b594797c-654d-4b37-818b-a134d01ae19c-kube-api-access-vfvx2\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:52:29.558808 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:29.558727    2558 generic.go:356] "Generic (PLEG): container finished" podID="b594797c-654d-4b37-818b-a134d01ae19c" containerID="0761cc4f66b0dd8ee94d35926a941e850d0904528185ef285f7157ef98df24b2" exitCode=0
Sep 10 02:52:29.558808 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:29.558783    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg"
Sep 10 02:52:29.558808 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:29.558797    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg" event={"ID":"b594797c-654d-4b37-818b-a134d01ae19c","Type":"ContainerDied","Data":"0761cc4f66b0dd8ee94d35926a941e850d0904528185ef285f7157ef98df24b2"}
Sep 10 02:52:29.559049 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:29.558862    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg" event={"ID":"b594797c-654d-4b37-818b-a134d01ae19c","Type":"ContainerDied","Data":"81e45baa16d893ae58037675ae7af6e57a401abe3af54f01da50d6a8329a4047"}
Sep 10 02:52:29.559049 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:29.558907    2558 scope.go:117] "RemoveContainer" containerID="0761cc4f66b0dd8ee94d35926a941e850d0904528185ef285f7157ef98df24b2"
Sep 10 02:52:29.583864 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:29.583838    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg"]
Sep 10 02:52:29.590625 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:29.590437    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-82qfg"]
Sep 10 02:52:29.701141 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:29.701087    2558 scope.go:117] "RemoveContainer" containerID="0761cc4f66b0dd8ee94d35926a941e850d0904528185ef285f7157ef98df24b2"
Sep 10 02:52:29.701540 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:52:29.701517    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0761cc4f66b0dd8ee94d35926a941e850d0904528185ef285f7157ef98df24b2\": container with ID starting with 0761cc4f66b0dd8ee94d35926a941e850d0904528185ef285f7157ef98df24b2 not found: ID does not exist" containerID="0761cc4f66b0dd8ee94d35926a941e850d0904528185ef285f7157ef98df24b2"
Sep 10 02:52:29.701631 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:29.701549    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0761cc4f66b0dd8ee94d35926a941e850d0904528185ef285f7157ef98df24b2"} err="failed to get container status \"0761cc4f66b0dd8ee94d35926a941e850d0904528185ef285f7157ef98df24b2\": rpc error: code = NotFound desc = could not find container \"0761cc4f66b0dd8ee94d35926a941e850d0904528185ef285f7157ef98df24b2\": container with ID starting with 0761cc4f66b0dd8ee94d35926a941e850d0904528185ef285f7157ef98df24b2 not found: ID does not exist"
Sep 10 02:52:30.069056 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:30.069025    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b594797c-654d-4b37-818b-a134d01ae19c" path="/var/lib/kubelet/pods/b594797c-654d-4b37-818b-a134d01ae19c/volumes"
Sep 10 02:52:31.571574 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:31.571489    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v8f8w" event={"ID":"6a49b00d-e027-421f-883e-3cacf773af95","Type":"ContainerStarted","Data":"f46f16533ed4f4d8665f6c53f24585bfe91bf51db7ae701ad3ce7df4078fc9fc"}
Sep 10 02:52:31.572019 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:31.571598    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v8f8w"
Sep 10 02:52:31.573158 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:31.573131    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mxtj6" event={"ID":"4b5dd34b-715d-469c-8e64-6406b64cd509","Type":"ContainerStarted","Data":"b42914640df956acbca0ae948fbcc58ce9023640e6bdf381d6962e8818feb323"}
Sep 10 02:52:31.573287 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:31.573269    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mxtj6"
Sep 10 02:52:31.602448 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:31.602406    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v8f8w" podStartSLOduration=1.637864982 podStartE2EDuration="4.602393433s" totalImagesPullingTime="2.964528451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:52:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:52:31.602121691 +0000 UTC m=+341.977207610" watchObservedRunningTime="2026-09-10 02:52:31.602393433 +0000 UTC m=+341.977479354"
Sep 10 02:52:32.527643 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:32.527617    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-6dv7b"
Sep 10 02:52:32.544421 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:32.544374    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mxtj6" podStartSLOduration=2.583238389 podStartE2EDuration="5.544359225s" totalImagesPullingTime="2.961120836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:52:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:52:31.635920649 +0000 UTC m=+342.011006569" watchObservedRunningTime="2026-09-10 02:52:32.544359225 +0000 UTC m=+342.919445147"
Sep 10 02:52:32.555841 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:32.555818    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-mzhlk"
Sep 10 02:52:50.120678 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:50.120642    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:52:52.061871 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:52.061823    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:52:59.119552 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:52:59.119490    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:53:02.579225 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:02.579177    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-v8f8w"
Sep 10 02:53:02.582216 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:02.582193    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mxtj6"
Sep 10 02:53:18.541805 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:18.541765    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:53:18.634882 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:18.634855    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:53:22.068810 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.068775    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awg66s"]
Sep 10 02:53:22.069320 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.069060    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b594797c-654d-4b37-818b-a134d01ae19c" containerName="discovery"
Sep 10 02:53:22.069320 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.069071    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b594797c-654d-4b37-818b-a134d01ae19c" containerName="discovery"
Sep 10 02:53:22.069320 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.069153    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b594797c-654d-4b37-818b-a134d01ae19c" containerName="discovery"
Sep 10 02:53:22.071096 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.071065    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awg66s"
Sep 10 02:53:22.077974 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.077942    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 02:53:22.078157 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.077996    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xllqd\""
Sep 10 02:53:22.078506 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.078482    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 02:53:22.080752 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.080731    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awg66s"]
Sep 10 02:53:22.141292 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.141263    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3d59ec7-386e-4c95-98a0-917194f458b5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awg66s\" (UID: \"b3d59ec7-386e-4c95-98a0-917194f458b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awg66s"
Sep 10 02:53:22.141388 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.141300    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s2q9\" (UniqueName: \"kubernetes.io/projected/b3d59ec7-386e-4c95-98a0-917194f458b5-kube-api-access-2s2q9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awg66s\" (UID: \"b3d59ec7-386e-4c95-98a0-917194f458b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awg66s"
Sep 10 02:53:22.141388 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.141326    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3d59ec7-386e-4c95-98a0-917194f458b5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awg66s\" (UID: \"b3d59ec7-386e-4c95-98a0-917194f458b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awg66s"
Sep 10 02:53:22.166706 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.166677    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xzdnx"]
Sep 10 02:53:22.168878 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.168863    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xzdnx"
Sep 10 02:53:22.177980 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.177959    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xzdnx"]
Sep 10 02:53:22.242490 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.242467    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3d59ec7-386e-4c95-98a0-917194f458b5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awg66s\" (UID: \"b3d59ec7-386e-4c95-98a0-917194f458b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awg66s"
Sep 10 02:53:22.242578 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.242499    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s2q9\" (UniqueName: \"kubernetes.io/projected/b3d59ec7-386e-4c95-98a0-917194f458b5-kube-api-access-2s2q9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awg66s\" (UID: \"b3d59ec7-386e-4c95-98a0-917194f458b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awg66s"
Sep 10 02:53:22.242578 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.242524    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3d59ec7-386e-4c95-98a0-917194f458b5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awg66s\" (UID: \"b3d59ec7-386e-4c95-98a0-917194f458b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awg66s"
Sep 10 02:53:22.242578 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.242551    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98xls\" (UniqueName: \"kubernetes.io/projected/edf6a7a9-80ea-4c0f-805f-8b71cea4560c-kube-api-access-98xls\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xzdnx\" (UID: \"edf6a7a9-80ea-4c0f-805f-8b71cea4560c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xzdnx"
Sep 10 02:53:22.242702 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.242598    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edf6a7a9-80ea-4c0f-805f-8b71cea4560c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xzdnx\" (UID: \"edf6a7a9-80ea-4c0f-805f-8b71cea4560c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xzdnx"
Sep 10 02:53:22.242702 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.242681    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edf6a7a9-80ea-4c0f-805f-8b71cea4560c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xzdnx\" (UID: \"edf6a7a9-80ea-4c0f-805f-8b71cea4560c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xzdnx"
Sep 10 02:53:22.242906 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.242883    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3d59ec7-386e-4c95-98a0-917194f458b5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awg66s\" (UID: \"b3d59ec7-386e-4c95-98a0-917194f458b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awg66s"
Sep 10 02:53:22.242944 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.242900    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3d59ec7-386e-4c95-98a0-917194f458b5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awg66s\" (UID: \"b3d59ec7-386e-4c95-98a0-917194f458b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awg66s"
Sep 10 02:53:22.250565 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.250543    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s2q9\" (UniqueName: \"kubernetes.io/projected/b3d59ec7-386e-4c95-98a0-917194f458b5-kube-api-access-2s2q9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awg66s\" (UID: \"b3d59ec7-386e-4c95-98a0-917194f458b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awg66s"
Sep 10 02:53:22.264637 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.264612    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpt9j"]
Sep 10 02:53:22.266940 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.266922    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpt9j"
Sep 10 02:53:22.275047 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.275019    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpt9j"]
Sep 10 02:53:22.343182 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.343125    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bjj2\" (UniqueName: \"kubernetes.io/projected/3fc5a3d1-56f7-495a-9569-4bfab214f1d3-kube-api-access-4bjj2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpt9j\" (UID: \"3fc5a3d1-56f7-495a-9569-4bfab214f1d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpt9j"
Sep 10 02:53:22.343182 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.343170    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edf6a7a9-80ea-4c0f-805f-8b71cea4560c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xzdnx\" (UID: \"edf6a7a9-80ea-4c0f-805f-8b71cea4560c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xzdnx"
Sep 10 02:53:22.343337 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.343196    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edf6a7a9-80ea-4c0f-805f-8b71cea4560c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xzdnx\" (UID: \"edf6a7a9-80ea-4c0f-805f-8b71cea4560c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xzdnx"
Sep 10 02:53:22.343337 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.343239    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3fc5a3d1-56f7-495a-9569-4bfab214f1d3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpt9j\" (UID: \"3fc5a3d1-56f7-495a-9569-4bfab214f1d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpt9j"
Sep 10 02:53:22.343337 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.343272    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98xls\" (UniqueName: \"kubernetes.io/projected/edf6a7a9-80ea-4c0f-805f-8b71cea4560c-kube-api-access-98xls\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xzdnx\" (UID: \"edf6a7a9-80ea-4c0f-805f-8b71cea4560c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xzdnx"
Sep 10 02:53:22.343337 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.343297    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3fc5a3d1-56f7-495a-9569-4bfab214f1d3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpt9j\" (UID: \"3fc5a3d1-56f7-495a-9569-4bfab214f1d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpt9j"
Sep 10 02:53:22.343528 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.343510    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edf6a7a9-80ea-4c0f-805f-8b71cea4560c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xzdnx\" (UID: \"edf6a7a9-80ea-4c0f-805f-8b71cea4560c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xzdnx"
Sep 10 02:53:22.343567 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.343540    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edf6a7a9-80ea-4c0f-805f-8b71cea4560c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xzdnx\" (UID: \"edf6a7a9-80ea-4c0f-805f-8b71cea4560c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xzdnx"
Sep 10 02:53:22.351252 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.351226    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98xls\" (UniqueName: \"kubernetes.io/projected/edf6a7a9-80ea-4c0f-805f-8b71cea4560c-kube-api-access-98xls\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xzdnx\" (UID: \"edf6a7a9-80ea-4c0f-805f-8b71cea4560c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xzdnx"
Sep 10 02:53:22.368832 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.368808    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ggj7f"]
Sep 10 02:53:22.370856 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.370841    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ggj7f"
Sep 10 02:53:22.379531 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.379508    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ggj7f"]
Sep 10 02:53:22.387844 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.387829    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awg66s"
Sep 10 02:53:22.445958 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.444722    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3fc5a3d1-56f7-495a-9569-4bfab214f1d3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpt9j\" (UID: \"3fc5a3d1-56f7-495a-9569-4bfab214f1d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpt9j"
Sep 10 02:53:22.445958 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.444781    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7de98e0a-58a8-42c3-8341-023e6098be8e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ggj7f\" (UID: \"7de98e0a-58a8-42c3-8341-023e6098be8e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ggj7f"
Sep 10 02:53:22.445958 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.444817    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ktrt\" (UniqueName: \"kubernetes.io/projected/7de98e0a-58a8-42c3-8341-023e6098be8e-kube-api-access-8ktrt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ggj7f\" (UID: \"7de98e0a-58a8-42c3-8341-023e6098be8e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ggj7f"
Sep 10 02:53:22.445958 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.444858    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3fc5a3d1-56f7-495a-9569-4bfab214f1d3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpt9j\" (UID: \"3fc5a3d1-56f7-495a-9569-4bfab214f1d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpt9j"
Sep 10 02:53:22.445958 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.444886    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7de98e0a-58a8-42c3-8341-023e6098be8e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ggj7f\" (UID: \"7de98e0a-58a8-42c3-8341-023e6098be8e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ggj7f"
Sep 10 02:53:22.445958 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.444938    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bjj2\" (UniqueName: \"kubernetes.io/projected/3fc5a3d1-56f7-495a-9569-4bfab214f1d3-kube-api-access-4bjj2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpt9j\" (UID: \"3fc5a3d1-56f7-495a-9569-4bfab214f1d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpt9j"
Sep 10 02:53:22.445958 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.445419    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3fc5a3d1-56f7-495a-9569-4bfab214f1d3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpt9j\" (UID: \"3fc5a3d1-56f7-495a-9569-4bfab214f1d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpt9j"
Sep 10 02:53:22.445958 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.445704    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3fc5a3d1-56f7-495a-9569-4bfab214f1d3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpt9j\" (UID: \"3fc5a3d1-56f7-495a-9569-4bfab214f1d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpt9j"
Sep 10 02:53:22.453521 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.453503    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bjj2\" (UniqueName: \"kubernetes.io/projected/3fc5a3d1-56f7-495a-9569-4bfab214f1d3-kube-api-access-4bjj2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpt9j\" (UID: \"3fc5a3d1-56f7-495a-9569-4bfab214f1d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpt9j"
Sep 10 02:53:22.477424 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.477404    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xzdnx"
Sep 10 02:53:22.545892 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.545860    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7de98e0a-58a8-42c3-8341-023e6098be8e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ggj7f\" (UID: \"7de98e0a-58a8-42c3-8341-023e6098be8e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ggj7f"
Sep 10 02:53:22.546021 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.545906    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ktrt\" (UniqueName: \"kubernetes.io/projected/7de98e0a-58a8-42c3-8341-023e6098be8e-kube-api-access-8ktrt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ggj7f\" (UID: \"7de98e0a-58a8-42c3-8341-023e6098be8e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ggj7f"
Sep 10 02:53:22.546021 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.545946    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7de98e0a-58a8-42c3-8341-023e6098be8e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ggj7f\" (UID: \"7de98e0a-58a8-42c3-8341-023e6098be8e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ggj7f"
Sep 10 02:53:22.546343 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.546313    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7de98e0a-58a8-42c3-8341-023e6098be8e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ggj7f\" (UID: \"7de98e0a-58a8-42c3-8341-023e6098be8e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ggj7f"
Sep 10 02:53:22.546434 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.546353    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7de98e0a-58a8-42c3-8341-023e6098be8e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ggj7f\" (UID: \"7de98e0a-58a8-42c3-8341-023e6098be8e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ggj7f"
Sep 10 02:53:22.554425 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.554402    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ktrt\" (UniqueName: \"kubernetes.io/projected/7de98e0a-58a8-42c3-8341-023e6098be8e-kube-api-access-8ktrt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ggj7f\" (UID: \"7de98e0a-58a8-42c3-8341-023e6098be8e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ggj7f"
Sep 10 02:53:22.576215 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.576188    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpt9j"
Sep 10 02:53:22.659937 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.659910    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awg66s"]
Sep 10 02:53:22.662296 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:53:22.662266    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d59ec7_386e_4c95_98a0_917194f458b5.slice/crio-3204da913ed8c6038a6b772b5e04d7eb4951d784f63d9a00ae40c5ada0c53c2c WatchSource:0}: Error finding container 3204da913ed8c6038a6b772b5e04d7eb4951d784f63d9a00ae40c5ada0c53c2c: Status 404 returned error can't find the container with id 3204da913ed8c6038a6b772b5e04d7eb4951d784f63d9a00ae40c5ada0c53c2c
Sep 10 02:53:22.679370 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.679343    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ggj7f"
Sep 10 02:53:22.749711 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.749633    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xzdnx"]
Sep 10 02:53:22.802519 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.802490    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xzdnx" event={"ID":"edf6a7a9-80ea-4c0f-805f-8b71cea4560c","Type":"ContainerStarted","Data":"32281a8785b8eec881a044b4f2db5c97270a2f128c5b4d2385e055808d7f2630"}
Sep 10 02:53:22.803959 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.803938    2558 generic.go:356] "Generic (PLEG): container finished" podID="b3d59ec7-386e-4c95-98a0-917194f458b5" containerID="ae750687e325928ade4cfc065a03cce1cc07a3a56c30eb166e31f83809ca31a6" exitCode=0
Sep 10 02:53:22.804051 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.804015    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awg66s" event={"ID":"b3d59ec7-386e-4c95-98a0-917194f458b5","Type":"ContainerDied","Data":"ae750687e325928ade4cfc065a03cce1cc07a3a56c30eb166e31f83809ca31a6"}
Sep 10 02:53:22.804121 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.804064    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awg66s" event={"ID":"b3d59ec7-386e-4c95-98a0-917194f458b5","Type":"ContainerStarted","Data":"3204da913ed8c6038a6b772b5e04d7eb4951d784f63d9a00ae40c5ada0c53c2c"}
Sep 10 02:53:22.858790 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.858769    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpt9j"]
Sep 10 02:53:22.859840 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:53:22.859819    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc5a3d1_56f7_495a_9569_4bfab214f1d3.slice/crio-d7f8e3828aa543919a1374d2beddd496cc9d68feb07b204591cd777697d4f7be WatchSource:0}: Error finding container d7f8e3828aa543919a1374d2beddd496cc9d68feb07b204591cd777697d4f7be: Status 404 returned error can't find the container with id d7f8e3828aa543919a1374d2beddd496cc9d68feb07b204591cd777697d4f7be
Sep 10 02:53:22.952728 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:22.952708    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ggj7f"]
Sep 10 02:53:22.993830 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:53:22.993804    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de98e0a_58a8_42c3_8341_023e6098be8e.slice/crio-1464f524a5b7c5399c334a47142d86594afd1a26e6bbb41e4d5fa269c2ff55d8 WatchSource:0}: Error finding container 1464f524a5b7c5399c334a47142d86594afd1a26e6bbb41e4d5fa269c2ff55d8: Status 404 returned error can't find the container with id 1464f524a5b7c5399c334a47142d86594afd1a26e6bbb41e4d5fa269c2ff55d8
Sep 10 02:53:23.810808 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:23.810774    2558 generic.go:356] "Generic (PLEG): container finished" podID="edf6a7a9-80ea-4c0f-805f-8b71cea4560c" containerID="1dc98c8bd2a71e1fe17fd8ba0ed6e8606a7d57bab6ff3bee9989139f67da7a73" exitCode=0
Sep 10 02:53:23.811246 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:23.810832    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xzdnx" event={"ID":"edf6a7a9-80ea-4c0f-805f-8b71cea4560c","Type":"ContainerDied","Data":"1dc98c8bd2a71e1fe17fd8ba0ed6e8606a7d57bab6ff3bee9989139f67da7a73"}
Sep 10 02:53:23.813298 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:23.813261    2558 generic.go:356] "Generic (PLEG): container finished" podID="b3d59ec7-386e-4c95-98a0-917194f458b5" containerID="462a86cb6ace526a95cce3e403f7c2c57ffc70f198f895aa40b8ecafe8d10b0c" exitCode=0
Sep 10 02:53:23.813406 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:23.813343    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awg66s" event={"ID":"b3d59ec7-386e-4c95-98a0-917194f458b5","Type":"ContainerDied","Data":"462a86cb6ace526a95cce3e403f7c2c57ffc70f198f895aa40b8ecafe8d10b0c"}
Sep 10 02:53:23.815545 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:23.815524    2558 generic.go:356] "Generic (PLEG): container finished" podID="7de98e0a-58a8-42c3-8341-023e6098be8e" containerID="b1ffbf940392c7ddb5921f995022f9cc7630d338c51b85bed48e4dfc53ca0c82" exitCode=0
Sep 10 02:53:23.815624 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:23.815587    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ggj7f" event={"ID":"7de98e0a-58a8-42c3-8341-023e6098be8e","Type":"ContainerDied","Data":"b1ffbf940392c7ddb5921f995022f9cc7630d338c51b85bed48e4dfc53ca0c82"}
Sep 10 02:53:23.815671 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:23.815632    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ggj7f" event={"ID":"7de98e0a-58a8-42c3-8341-023e6098be8e","Type":"ContainerStarted","Data":"1464f524a5b7c5399c334a47142d86594afd1a26e6bbb41e4d5fa269c2ff55d8"}
Sep 10 02:53:23.817711 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:23.817668    2558 generic.go:356] "Generic (PLEG): container finished" podID="3fc5a3d1-56f7-495a-9569-4bfab214f1d3" containerID="26d142815a803fa9b2a783e9f3ee1f295f4cecc61297b7e2ac6ffdb66594e7ad" exitCode=0
Sep 10 02:53:23.817814 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:23.817723    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpt9j" event={"ID":"3fc5a3d1-56f7-495a-9569-4bfab214f1d3","Type":"ContainerDied","Data":"26d142815a803fa9b2a783e9f3ee1f295f4cecc61297b7e2ac6ffdb66594e7ad"}
Sep 10 02:53:23.817814 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:23.817800    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpt9j" event={"ID":"3fc5a3d1-56f7-495a-9569-4bfab214f1d3","Type":"ContainerStarted","Data":"d7f8e3828aa543919a1374d2beddd496cc9d68feb07b204591cd777697d4f7be"}
Sep 10 02:53:24.824211 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:24.824183    2558 generic.go:356] "Generic (PLEG): container finished" podID="edf6a7a9-80ea-4c0f-805f-8b71cea4560c" containerID="06a04427c0c229ef25ac94022f88917ae596fb2da5e0e37afdf57beea15a268b" exitCode=0
Sep 10 02:53:24.824575 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:24.824285    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xzdnx" event={"ID":"edf6a7a9-80ea-4c0f-805f-8b71cea4560c","Type":"ContainerDied","Data":"06a04427c0c229ef25ac94022f88917ae596fb2da5e0e37afdf57beea15a268b"}
Sep 10 02:53:24.826876 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:24.826849    2558 generic.go:356] "Generic (PLEG): container finished" podID="b3d59ec7-386e-4c95-98a0-917194f458b5" containerID="24fa145f417b174044c05c0d620f6d959022e625f1497302a7550e40fddcb6a5" exitCode=0
Sep 10 02:53:24.826952 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:24.826896    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awg66s" event={"ID":"b3d59ec7-386e-4c95-98a0-917194f458b5","Type":"ContainerDied","Data":"24fa145f417b174044c05c0d620f6d959022e625f1497302a7550e40fddcb6a5"}
Sep 10 02:53:25.833855 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:25.833823    2558 generic.go:356] "Generic (PLEG): container finished" podID="7de98e0a-58a8-42c3-8341-023e6098be8e" containerID="a18b8cf2e639305c9bc9d4d0c93275daf59fc8239bd17e77dec7370f3c33ab35" exitCode=0
Sep 10 02:53:25.834211 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:25.833902    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ggj7f" event={"ID":"7de98e0a-58a8-42c3-8341-023e6098be8e","Type":"ContainerDied","Data":"a18b8cf2e639305c9bc9d4d0c93275daf59fc8239bd17e77dec7370f3c33ab35"}
Sep 10 02:53:25.836587 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:25.836562    2558 generic.go:356] "Generic (PLEG): container finished" podID="edf6a7a9-80ea-4c0f-805f-8b71cea4560c" containerID="cfc2bd12e373de5978b4e1fc451e8e5643239289bf48ca9d022ed45c7842b402" exitCode=0
Sep 10 02:53:25.836685 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:25.836665    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xzdnx" event={"ID":"edf6a7a9-80ea-4c0f-805f-8b71cea4560c","Type":"ContainerDied","Data":"cfc2bd12e373de5978b4e1fc451e8e5643239289bf48ca9d022ed45c7842b402"}
Sep 10 02:53:25.962474 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:25.962451    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awg66s"
Sep 10 02:53:26.071082 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:26.071045    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2s2q9\" (UniqueName: \"kubernetes.io/projected/b3d59ec7-386e-4c95-98a0-917194f458b5-kube-api-access-2s2q9\") pod \"b3d59ec7-386e-4c95-98a0-917194f458b5\" (UID: \"b3d59ec7-386e-4c95-98a0-917194f458b5\") "
Sep 10 02:53:26.071236 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:26.071138    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3d59ec7-386e-4c95-98a0-917194f458b5-bundle\") pod \"b3d59ec7-386e-4c95-98a0-917194f458b5\" (UID: \"b3d59ec7-386e-4c95-98a0-917194f458b5\") "
Sep 10 02:53:26.071308 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:26.071236    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3d59ec7-386e-4c95-98a0-917194f458b5-util\") pod \"b3d59ec7-386e-4c95-98a0-917194f458b5\" (UID: \"b3d59ec7-386e-4c95-98a0-917194f458b5\") "
Sep 10 02:53:26.071796 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:26.071767    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3d59ec7-386e-4c95-98a0-917194f458b5-bundle" (OuterVolumeSpecName: "bundle") pod "b3d59ec7-386e-4c95-98a0-917194f458b5" (UID: "b3d59ec7-386e-4c95-98a0-917194f458b5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 02:53:26.072989 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:26.072964    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3d59ec7-386e-4c95-98a0-917194f458b5-kube-api-access-2s2q9" (OuterVolumeSpecName: "kube-api-access-2s2q9") pod "b3d59ec7-386e-4c95-98a0-917194f458b5" (UID: "b3d59ec7-386e-4c95-98a0-917194f458b5"). InnerVolumeSpecName "kube-api-access-2s2q9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 02:53:26.076523 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:26.076504    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3d59ec7-386e-4c95-98a0-917194f458b5-util" (OuterVolumeSpecName: "util") pod "b3d59ec7-386e-4c95-98a0-917194f458b5" (UID: "b3d59ec7-386e-4c95-98a0-917194f458b5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 02:53:26.154944 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:26.154898    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:53:26.171956 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:26.171932    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3d59ec7-386e-4c95-98a0-917194f458b5-util\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:53:26.171956 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:26.171954    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s2q9\" (UniqueName: \"kubernetes.io/projected/b3d59ec7-386e-4c95-98a0-917194f458b5-kube-api-access-2s2q9\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:53:26.172068 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:26.171964    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3d59ec7-386e-4c95-98a0-917194f458b5-bundle\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:53:26.210358 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:26.210334    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:53:26.842873 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:26.842850    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awg66s"
Sep 10 02:53:26.843277 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:26.842842    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awg66s" event={"ID":"b3d59ec7-386e-4c95-98a0-917194f458b5","Type":"ContainerDied","Data":"3204da913ed8c6038a6b772b5e04d7eb4951d784f63d9a00ae40c5ada0c53c2c"}
Sep 10 02:53:26.843277 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:26.842983    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3204da913ed8c6038a6b772b5e04d7eb4951d784f63d9a00ae40c5ada0c53c2c"
Sep 10 02:53:26.845467 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:26.845440    2558 generic.go:356] "Generic (PLEG): container finished" podID="7de98e0a-58a8-42c3-8341-023e6098be8e" containerID="8ef12d358fe78dc8b8620581fb5ec867bbddc1159e257d85cf94da2018c15507" exitCode=0
Sep 10 02:53:26.845572 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:26.845524    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ggj7f" event={"ID":"7de98e0a-58a8-42c3-8341-023e6098be8e","Type":"ContainerDied","Data":"8ef12d358fe78dc8b8620581fb5ec867bbddc1159e257d85cf94da2018c15507"}
Sep 10 02:53:26.979053 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:26.979032    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xzdnx"
Sep 10 02:53:27.078188 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:27.078163    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edf6a7a9-80ea-4c0f-805f-8b71cea4560c-util\") pod \"edf6a7a9-80ea-4c0f-805f-8b71cea4560c\" (UID: \"edf6a7a9-80ea-4c0f-805f-8b71cea4560c\") "
Sep 10 02:53:27.078297 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:27.078218    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98xls\" (UniqueName: \"kubernetes.io/projected/edf6a7a9-80ea-4c0f-805f-8b71cea4560c-kube-api-access-98xls\") pod \"edf6a7a9-80ea-4c0f-805f-8b71cea4560c\" (UID: \"edf6a7a9-80ea-4c0f-805f-8b71cea4560c\") "
Sep 10 02:53:27.078297 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:27.078243    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edf6a7a9-80ea-4c0f-805f-8b71cea4560c-bundle\") pod \"edf6a7a9-80ea-4c0f-805f-8b71cea4560c\" (UID: \"edf6a7a9-80ea-4c0f-805f-8b71cea4560c\") "
Sep 10 02:53:27.078756 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:27.078728    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf6a7a9-80ea-4c0f-805f-8b71cea4560c-bundle" (OuterVolumeSpecName: "bundle") pod "edf6a7a9-80ea-4c0f-805f-8b71cea4560c" (UID: "edf6a7a9-80ea-4c0f-805f-8b71cea4560c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 02:53:27.080281 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:27.080255    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edf6a7a9-80ea-4c0f-805f-8b71cea4560c-kube-api-access-98xls" (OuterVolumeSpecName: "kube-api-access-98xls") pod "edf6a7a9-80ea-4c0f-805f-8b71cea4560c" (UID: "edf6a7a9-80ea-4c0f-805f-8b71cea4560c"). InnerVolumeSpecName "kube-api-access-98xls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 02:53:27.083493 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:27.083460    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf6a7a9-80ea-4c0f-805f-8b71cea4560c-util" (OuterVolumeSpecName: "util") pod "edf6a7a9-80ea-4c0f-805f-8b71cea4560c" (UID: "edf6a7a9-80ea-4c0f-805f-8b71cea4560c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 02:53:27.179220 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:27.179157    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edf6a7a9-80ea-4c0f-805f-8b71cea4560c-util\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:53:27.179220 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:27.179182    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98xls\" (UniqueName: \"kubernetes.io/projected/edf6a7a9-80ea-4c0f-805f-8b71cea4560c-kube-api-access-98xls\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:53:27.179220 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:27.179199    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edf6a7a9-80ea-4c0f-805f-8b71cea4560c-bundle\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:53:27.854127 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:27.854073    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xzdnx"
Sep 10 02:53:27.854127 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:27.854085    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xzdnx" event={"ID":"edf6a7a9-80ea-4c0f-805f-8b71cea4560c","Type":"ContainerDied","Data":"32281a8785b8eec881a044b4f2db5c97270a2f128c5b4d2385e055808d7f2630"}
Sep 10 02:53:27.854571 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:27.854161    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32281a8785b8eec881a044b4f2db5c97270a2f128c5b4d2385e055808d7f2630"
Sep 10 02:53:27.978549 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:27.978525    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ggj7f"
Sep 10 02:53:28.085398 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:28.085374    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7de98e0a-58a8-42c3-8341-023e6098be8e-util\") pod \"7de98e0a-58a8-42c3-8341-023e6098be8e\" (UID: \"7de98e0a-58a8-42c3-8341-023e6098be8e\") "
Sep 10 02:53:28.085507 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:28.085443    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ktrt\" (UniqueName: \"kubernetes.io/projected/7de98e0a-58a8-42c3-8341-023e6098be8e-kube-api-access-8ktrt\") pod \"7de98e0a-58a8-42c3-8341-023e6098be8e\" (UID: \"7de98e0a-58a8-42c3-8341-023e6098be8e\") "
Sep 10 02:53:28.085507 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:28.085494    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7de98e0a-58a8-42c3-8341-023e6098be8e-bundle\") pod \"7de98e0a-58a8-42c3-8341-023e6098be8e\" (UID: \"7de98e0a-58a8-42c3-8341-023e6098be8e\") "
Sep 10 02:53:28.085996 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:28.085941    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de98e0a-58a8-42c3-8341-023e6098be8e-bundle" (OuterVolumeSpecName: "bundle") pod "7de98e0a-58a8-42c3-8341-023e6098be8e" (UID: "7de98e0a-58a8-42c3-8341-023e6098be8e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 02:53:28.087537 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:28.087511    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7de98e0a-58a8-42c3-8341-023e6098be8e-kube-api-access-8ktrt" (OuterVolumeSpecName: "kube-api-access-8ktrt") pod "7de98e0a-58a8-42c3-8341-023e6098be8e" (UID: "7de98e0a-58a8-42c3-8341-023e6098be8e"). InnerVolumeSpecName "kube-api-access-8ktrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 02:53:28.090516 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:28.090483    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de98e0a-58a8-42c3-8341-023e6098be8e-util" (OuterVolumeSpecName: "util") pod "7de98e0a-58a8-42c3-8341-023e6098be8e" (UID: "7de98e0a-58a8-42c3-8341-023e6098be8e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 02:53:28.185941 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:28.185887    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ktrt\" (UniqueName: \"kubernetes.io/projected/7de98e0a-58a8-42c3-8341-023e6098be8e-kube-api-access-8ktrt\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:53:28.185941 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:28.185908    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7de98e0a-58a8-42c3-8341-023e6098be8e-bundle\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:53:28.185941 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:28.185917    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7de98e0a-58a8-42c3-8341-023e6098be8e-util\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:53:28.860248 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:28.860217    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ggj7f"
Sep 10 02:53:28.860733 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:28.860212    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ggj7f" event={"ID":"7de98e0a-58a8-42c3-8341-023e6098be8e","Type":"ContainerDied","Data":"1464f524a5b7c5399c334a47142d86594afd1a26e6bbb41e4d5fa269c2ff55d8"}
Sep 10 02:53:28.860733 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:28.860358    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1464f524a5b7c5399c334a47142d86594afd1a26e6bbb41e4d5fa269c2ff55d8"
Sep 10 02:53:30.871459 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:30.871424    2558 generic.go:356] "Generic (PLEG): container finished" podID="3fc5a3d1-56f7-495a-9569-4bfab214f1d3" containerID="9e158e87b2c9be0bdf74a5489082944fbc5588651b4b5d4708c30bb73ac790b6" exitCode=0
Sep 10 02:53:30.871889 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:30.871524    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpt9j" event={"ID":"3fc5a3d1-56f7-495a-9569-4bfab214f1d3","Type":"ContainerDied","Data":"9e158e87b2c9be0bdf74a5489082944fbc5588651b4b5d4708c30bb73ac790b6"}
Sep 10 02:53:31.878824 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:31.878792    2558 generic.go:356] "Generic (PLEG): container finished" podID="3fc5a3d1-56f7-495a-9569-4bfab214f1d3" containerID="7ed0a887de85f5ebc4ca1919e9c17815431139501a7ff5813a16a415a2ccb64f" exitCode=0
Sep 10 02:53:31.879225 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:31.878886    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpt9j" event={"ID":"3fc5a3d1-56f7-495a-9569-4bfab214f1d3","Type":"ContainerDied","Data":"7ed0a887de85f5ebc4ca1919e9c17815431139501a7ff5813a16a415a2ccb64f"}
Sep 10 02:53:33.008853 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:33.008830    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpt9j"
Sep 10 02:53:33.122810 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:33.122784    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bjj2\" (UniqueName: \"kubernetes.io/projected/3fc5a3d1-56f7-495a-9569-4bfab214f1d3-kube-api-access-4bjj2\") pod \"3fc5a3d1-56f7-495a-9569-4bfab214f1d3\" (UID: \"3fc5a3d1-56f7-495a-9569-4bfab214f1d3\") "
Sep 10 02:53:33.122930 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:33.122820    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3fc5a3d1-56f7-495a-9569-4bfab214f1d3-bundle\") pod \"3fc5a3d1-56f7-495a-9569-4bfab214f1d3\" (UID: \"3fc5a3d1-56f7-495a-9569-4bfab214f1d3\") "
Sep 10 02:53:33.122930 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:33.122846    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3fc5a3d1-56f7-495a-9569-4bfab214f1d3-util\") pod \"3fc5a3d1-56f7-495a-9569-4bfab214f1d3\" (UID: \"3fc5a3d1-56f7-495a-9569-4bfab214f1d3\") "
Sep 10 02:53:33.123351 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:33.123325    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc5a3d1-56f7-495a-9569-4bfab214f1d3-bundle" (OuterVolumeSpecName: "bundle") pod "3fc5a3d1-56f7-495a-9569-4bfab214f1d3" (UID: "3fc5a3d1-56f7-495a-9569-4bfab214f1d3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 02:53:33.124871 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:33.124848    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc5a3d1-56f7-495a-9569-4bfab214f1d3-kube-api-access-4bjj2" (OuterVolumeSpecName: "kube-api-access-4bjj2") pod "3fc5a3d1-56f7-495a-9569-4bfab214f1d3" (UID: "3fc5a3d1-56f7-495a-9569-4bfab214f1d3"). InnerVolumeSpecName "kube-api-access-4bjj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 02:53:33.127316 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:33.127295    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc5a3d1-56f7-495a-9569-4bfab214f1d3-util" (OuterVolumeSpecName: "util") pod "3fc5a3d1-56f7-495a-9569-4bfab214f1d3" (UID: "3fc5a3d1-56f7-495a-9569-4bfab214f1d3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 02:53:33.223757 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:33.223736    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3fc5a3d1-56f7-495a-9569-4bfab214f1d3-bundle\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:53:33.223757 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:33.223756    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3fc5a3d1-56f7-495a-9569-4bfab214f1d3-util\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:53:33.223858 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:33.223766    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bjj2\" (UniqueName: \"kubernetes.io/projected/3fc5a3d1-56f7-495a-9569-4bfab214f1d3-kube-api-access-4bjj2\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:53:33.891611 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:33.891575    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpt9j"
Sep 10 02:53:33.891611 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:33.891590    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpt9j" event={"ID":"3fc5a3d1-56f7-495a-9569-4bfab214f1d3","Type":"ContainerDied","Data":"d7f8e3828aa543919a1374d2beddd496cc9d68feb07b204591cd777697d4f7be"}
Sep 10 02:53:33.891828 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:33.891650    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7f8e3828aa543919a1374d2beddd496cc9d68feb07b204591cd777697d4f7be"
Sep 10 02:53:37.282228 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:37.282195    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:53:37.424403 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:37.424377    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:53:37.515728 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:37.515698    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:53:37.618339 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:37.618274    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:53:37.716581 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:37.716554    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:53:39.304865 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.304830    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2m2kv"]
Sep 10 02:53:39.305266 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.305121    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fc5a3d1-56f7-495a-9569-4bfab214f1d3" containerName="util"
Sep 10 02:53:39.305266 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.305134    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fc5a3d1-56f7-495a-9569-4bfab214f1d3" containerName="util"
Sep 10 02:53:39.305266 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.305141    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3d59ec7-386e-4c95-98a0-917194f458b5" containerName="pull"
Sep 10 02:53:39.305266 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.305147    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3d59ec7-386e-4c95-98a0-917194f458b5" containerName="pull"
Sep 10 02:53:39.305266 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.305154    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edf6a7a9-80ea-4c0f-805f-8b71cea4560c" containerName="pull"
Sep 10 02:53:39.305266 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.305160    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="edf6a7a9-80ea-4c0f-805f-8b71cea4560c" containerName="pull"
Sep 10 02:53:39.305266 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.305169    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edf6a7a9-80ea-4c0f-805f-8b71cea4560c" containerName="extract"
Sep 10 02:53:39.305266 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.305174    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="edf6a7a9-80ea-4c0f-805f-8b71cea4560c" containerName="extract"
Sep 10 02:53:39.305266 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.305183    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edf6a7a9-80ea-4c0f-805f-8b71cea4560c" containerName="util"
Sep 10 02:53:39.305266 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.305188    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="edf6a7a9-80ea-4c0f-805f-8b71cea4560c" containerName="util"
Sep 10 02:53:39.305266 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.305194    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7de98e0a-58a8-42c3-8341-023e6098be8e" containerName="util"
Sep 10 02:53:39.305266 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.305200    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7de98e0a-58a8-42c3-8341-023e6098be8e" containerName="util"
Sep 10 02:53:39.305266 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.305205    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3d59ec7-386e-4c95-98a0-917194f458b5" containerName="extract"
Sep 10 02:53:39.305266 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.305210    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3d59ec7-386e-4c95-98a0-917194f458b5" containerName="extract"
Sep 10 02:53:39.305266 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.305216    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7de98e0a-58a8-42c3-8341-023e6098be8e" containerName="extract"
Sep 10 02:53:39.305266 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.305221    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7de98e0a-58a8-42c3-8341-023e6098be8e" containerName="extract"
Sep 10 02:53:39.305266 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.305225    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3d59ec7-386e-4c95-98a0-917194f458b5" containerName="util"
Sep 10 02:53:39.305266 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.305230    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3d59ec7-386e-4c95-98a0-917194f458b5" containerName="util"
Sep 10 02:53:39.305266 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.305238    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fc5a3d1-56f7-495a-9569-4bfab214f1d3" containerName="extract"
Sep 10 02:53:39.305266 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.305243    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fc5a3d1-56f7-495a-9569-4bfab214f1d3" containerName="extract"
Sep 10 02:53:39.305266 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.305251    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fc5a3d1-56f7-495a-9569-4bfab214f1d3" containerName="pull"
Sep 10 02:53:39.305266 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.305256    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fc5a3d1-56f7-495a-9569-4bfab214f1d3" containerName="pull"
Sep 10 02:53:39.305266 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.305269    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7de98e0a-58a8-42c3-8341-023e6098be8e" containerName="pull"
Sep 10 02:53:39.305266 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.305274    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7de98e0a-58a8-42c3-8341-023e6098be8e" containerName="pull"
Sep 10 02:53:39.305949 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.305322    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7de98e0a-58a8-42c3-8341-023e6098be8e" containerName="extract"
Sep 10 02:53:39.305949 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.305332    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3d59ec7-386e-4c95-98a0-917194f458b5" containerName="extract"
Sep 10 02:53:39.305949 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.305339    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="edf6a7a9-80ea-4c0f-805f-8b71cea4560c" containerName="extract"
Sep 10 02:53:39.305949 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.305344    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fc5a3d1-56f7-495a-9569-4bfab214f1d3" containerName="extract"
Sep 10 02:53:39.307223 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.307201    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2m2kv"
Sep 10 02:53:39.309568 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.309525    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 02:53:39.309568 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.309542    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 10 02:53:39.309568 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.309564    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 02:53:39.309809 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.309652    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-64c8m\""
Sep 10 02:53:39.319369 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.319344    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2m2kv"]
Sep 10 02:53:39.460655 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.460629    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvfcs\" (UniqueName: \"kubernetes.io/projected/55057b44-ed7c-40b2-8db8-3e7d2e26993d-kube-api-access-cvfcs\") pod \"dns-operator-controller-manager-7b84cd7c9c-2m2kv\" (UID: \"55057b44-ed7c-40b2-8db8-3e7d2e26993d\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2m2kv"
Sep 10 02:53:39.561798 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.561741    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvfcs\" (UniqueName: \"kubernetes.io/projected/55057b44-ed7c-40b2-8db8-3e7d2e26993d-kube-api-access-cvfcs\") pod \"dns-operator-controller-manager-7b84cd7c9c-2m2kv\" (UID: \"55057b44-ed7c-40b2-8db8-3e7d2e26993d\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2m2kv"
Sep 10 02:53:39.570838 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.570816    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvfcs\" (UniqueName: \"kubernetes.io/projected/55057b44-ed7c-40b2-8db8-3e7d2e26993d-kube-api-access-cvfcs\") pod \"dns-operator-controller-manager-7b84cd7c9c-2m2kv\" (UID: \"55057b44-ed7c-40b2-8db8-3e7d2e26993d\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2m2kv"
Sep 10 02:53:39.617682 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.617658    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2m2kv"
Sep 10 02:53:39.888666 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.888641    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2m2kv"]
Sep 10 02:53:39.891020 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:53:39.890990    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55057b44_ed7c_40b2_8db8_3e7d2e26993d.slice/crio-47476f3382653fb30c9dc20626fe02140fcc3c48a08f5056721c99963f342ce7 WatchSource:0}: Error finding container 47476f3382653fb30c9dc20626fe02140fcc3c48a08f5056721c99963f342ce7: Status 404 returned error can't find the container with id 47476f3382653fb30c9dc20626fe02140fcc3c48a08f5056721c99963f342ce7
Sep 10 02:53:39.920137 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:39.920096    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2m2kv" event={"ID":"55057b44-ed7c-40b2-8db8-3e7d2e26993d","Type":"ContainerStarted","Data":"47476f3382653fb30c9dc20626fe02140fcc3c48a08f5056721c99963f342ce7"}
Sep 10 02:53:40.061963 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:40.061929    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:53:42.184113 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:42.184062    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5tcnr"]
Sep 10 02:53:42.191356 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:42.191330    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5tcnr"
Sep 10 02:53:42.194463 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:42.194430    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-jtznx\""
Sep 10 02:53:42.198032 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:42.198008    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5tcnr"]
Sep 10 02:53:42.280623 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:42.280584    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgkzs\" (UniqueName: \"kubernetes.io/projected/8f3d3edb-f9f2-4e33-9fde-98e84ed8b3fc-kube-api-access-mgkzs\") pod \"limitador-operator-controller-manager-74495c69f8-5tcnr\" (UID: \"8f3d3edb-f9f2-4e33-9fde-98e84ed8b3fc\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5tcnr"
Sep 10 02:53:42.382016 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:42.381974    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgkzs\" (UniqueName: \"kubernetes.io/projected/8f3d3edb-f9f2-4e33-9fde-98e84ed8b3fc-kube-api-access-mgkzs\") pod \"limitador-operator-controller-manager-74495c69f8-5tcnr\" (UID: \"8f3d3edb-f9f2-4e33-9fde-98e84ed8b3fc\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5tcnr"
Sep 10 02:53:42.396818 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:42.396778    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgkzs\" (UniqueName: \"kubernetes.io/projected/8f3d3edb-f9f2-4e33-9fde-98e84ed8b3fc-kube-api-access-mgkzs\") pod \"limitador-operator-controller-manager-74495c69f8-5tcnr\" (UID: \"8f3d3edb-f9f2-4e33-9fde-98e84ed8b3fc\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5tcnr"
Sep 10 02:53:42.503529 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:42.503496    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5tcnr"
Sep 10 02:53:42.823918 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:42.823855    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5tcnr"]
Sep 10 02:53:42.826659 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:53:42.826619    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f3d3edb_f9f2_4e33_9fde_98e84ed8b3fc.slice/crio-0555cd9e282450c4dc740639ea0ae947cf22f793421381598199012697570142 WatchSource:0}: Error finding container 0555cd9e282450c4dc740639ea0ae947cf22f793421381598199012697570142: Status 404 returned error can't find the container with id 0555cd9e282450c4dc740639ea0ae947cf22f793421381598199012697570142
Sep 10 02:53:42.867585 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:42.867554    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:53:42.937078 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:42.937051    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5tcnr" event={"ID":"8f3d3edb-f9f2-4e33-9fde-98e84ed8b3fc","Type":"ContainerStarted","Data":"0555cd9e282450c4dc740639ea0ae947cf22f793421381598199012697570142"}
Sep 10 02:53:42.938601 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:42.938578    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2m2kv" event={"ID":"55057b44-ed7c-40b2-8db8-3e7d2e26993d","Type":"ContainerStarted","Data":"d23bfc0bc85c054dada53aa6cd5f3b5ebd0d030111340fbbd125c242cccd5984"}
Sep 10 02:53:42.938726 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:42.938708    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2m2kv"
Sep 10 02:53:42.967823 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:42.967780    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2m2kv" podStartSLOduration=1.259273265 podStartE2EDuration="3.967765655s" totalImagesPullingTime="2.70849239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:53:42.966734106 +0000 UTC m=+413.341820024" watchObservedRunningTime="2026-09-10 02:53:42.967765655 +0000 UTC m=+413.342851575"
Sep 10 02:53:43.310388 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:43.310362    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:53:43.464174 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:43.464140    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:53:44.283365 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:44.283329    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:53:44.951536 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:44.951493    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5tcnr" event={"ID":"8f3d3edb-f9f2-4e33-9fde-98e84ed8b3fc","Type":"ContainerStarted","Data":"d4c893df9abaef260338d35e52a2ecf0410e9f5e4bac4efab13c9182efe5e6ea"}
Sep 10 02:53:44.951982 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:44.951582    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5tcnr"
Sep 10 02:53:44.968964 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:44.968910    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5tcnr" podStartSLOduration=1.537974752 podStartE2EDuration="2.968896335s" totalImagesPullingTime="1.430921583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:53:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:53:44.966898201 +0000 UTC m=+415.341984122" watchObservedRunningTime="2026-09-10 02:53:44.968896335 +0000 UTC m=+415.343982256"
Sep 10 02:53:45.316955 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:45.316879    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:53:46.702503 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:46.702467    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:53:46.799631 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:46.799595    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:53:48.566295 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:48.566262    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9dx99"]
Sep 10 02:53:48.568582 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:48.568559    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9dx99"
Sep 10 02:53:48.572117 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:48.572083    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-sp59z\""
Sep 10 02:53:48.594924 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:48.594901    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9dx99"]
Sep 10 02:53:48.728260 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:48.728233    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjlz4\" (UniqueName: \"kubernetes.io/projected/da5f1e33-ba7e-424e-a870-f800d1cd5d31-kube-api-access-bjlz4\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9dx99\" (UID: \"da5f1e33-ba7e-424e-a870-f800d1cd5d31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9dx99"
Sep 10 02:53:48.728340 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:48.728286    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/da5f1e33-ba7e-424e-a870-f800d1cd5d31-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9dx99\" (UID: \"da5f1e33-ba7e-424e-a870-f800d1cd5d31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9dx99"
Sep 10 02:53:48.828974 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:48.828903    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/da5f1e33-ba7e-424e-a870-f800d1cd5d31-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9dx99\" (UID: \"da5f1e33-ba7e-424e-a870-f800d1cd5d31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9dx99"
Sep 10 02:53:48.828974 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:48.828956    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjlz4\" (UniqueName: \"kubernetes.io/projected/da5f1e33-ba7e-424e-a870-f800d1cd5d31-kube-api-access-bjlz4\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9dx99\" (UID: \"da5f1e33-ba7e-424e-a870-f800d1cd5d31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9dx99"
Sep 10 02:53:48.829304 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:48.829279    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/da5f1e33-ba7e-424e-a870-f800d1cd5d31-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9dx99\" (UID: \"da5f1e33-ba7e-424e-a870-f800d1cd5d31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9dx99"
Sep 10 02:53:48.836829 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:48.836808    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjlz4\" (UniqueName: \"kubernetes.io/projected/da5f1e33-ba7e-424e-a870-f800d1cd5d31-kube-api-access-bjlz4\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9dx99\" (UID: \"da5f1e33-ba7e-424e-a870-f800d1cd5d31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9dx99"
Sep 10 02:53:48.852139 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:48.852080    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:53:48.878595 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:48.878574    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9dx99"
Sep 10 02:53:49.084874 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:49.084812    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:53:49.154998 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:49.154975    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9dx99"]
Sep 10 02:53:49.157080 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:53:49.157052    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda5f1e33_ba7e_424e_a870_f800d1cd5d31.slice/crio-c5f207b9fb304ce9953b7c5aff9b878c87ff27ee0599f63e93f89715c2b7df8d WatchSource:0}: Error finding container c5f207b9fb304ce9953b7c5aff9b878c87ff27ee0599f63e93f89715c2b7df8d: Status 404 returned error can't find the container with id c5f207b9fb304ce9953b7c5aff9b878c87ff27ee0599f63e93f89715c2b7df8d
Sep 10 02:53:49.466541 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:49.466516    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:53:49.594608 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:49.594584    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:53:49.979397 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:49.979361    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9dx99" event={"ID":"da5f1e33-ba7e-424e-a870-f800d1cd5d31","Type":"ContainerStarted","Data":"c5f207b9fb304ce9953b7c5aff9b878c87ff27ee0599f63e93f89715c2b7df8d"}
Sep 10 02:53:51.704936 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:51.704899    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-sv8sd"]
Sep 10 02:53:51.707520 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:51.707501    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-sv8sd"
Sep 10 02:53:51.709992 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:51.709969    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-j68j5\""
Sep 10 02:53:51.721553 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:51.721528    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-sv8sd"]
Sep 10 02:53:51.850802 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:51.850765    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-955kw\" (UniqueName: \"kubernetes.io/projected/9f5cecb4-ee84-45f3-a3e9-5ef4367c02b6-kube-api-access-955kw\") pod \"authorino-operator-76b478bd78-sv8sd\" (UID: \"9f5cecb4-ee84-45f3-a3e9-5ef4367c02b6\") " pod="kuadrant-system/authorino-operator-76b478bd78-sv8sd"
Sep 10 02:53:51.951411 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:51.951381    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-955kw\" (UniqueName: \"kubernetes.io/projected/9f5cecb4-ee84-45f3-a3e9-5ef4367c02b6-kube-api-access-955kw\") pod \"authorino-operator-76b478bd78-sv8sd\" (UID: \"9f5cecb4-ee84-45f3-a3e9-5ef4367c02b6\") " pod="kuadrant-system/authorino-operator-76b478bd78-sv8sd"
Sep 10 02:53:51.965216 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:51.965159    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-955kw\" (UniqueName: \"kubernetes.io/projected/9f5cecb4-ee84-45f3-a3e9-5ef4367c02b6-kube-api-access-955kw\") pod \"authorino-operator-76b478bd78-sv8sd\" (UID: \"9f5cecb4-ee84-45f3-a3e9-5ef4367c02b6\") " pod="kuadrant-system/authorino-operator-76b478bd78-sv8sd"
Sep 10 02:53:52.018798 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:52.018774    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-sv8sd"
Sep 10 02:53:52.882012 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:52.881980    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:53:52.982082 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:52.982058    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:53:52.997087 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:52.997065    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9dx99" event={"ID":"da5f1e33-ba7e-424e-a870-f800d1cd5d31","Type":"ContainerStarted","Data":"b305d7bbd07ecf4de49a06b46bd0b5f9cb5a73ffb42fb0a6a9fe742c6352762c"}
Sep 10 02:53:52.997345 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:52.997325    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9dx99"
Sep 10 02:53:53.023624 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:53.023575    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9dx99" podStartSLOduration=1.260667162 podStartE2EDuration="5.023560308s" totalImagesPullingTime="3.762893146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:53:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:53:53.020420884 +0000 UTC m=+423.395506806" watchObservedRunningTime="2026-09-10 02:53:53.023560308 +0000 UTC m=+423.398646229"
Sep 10 02:53:53.142764 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:53.142743    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-sv8sd"]
Sep 10 02:53:53.144339 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:53:53.144318    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f5cecb4_ee84_45f3_a3e9_5ef4367c02b6.slice/crio-c7657be7f8ba0a444b20ebde832671e073007ee16517aa6e2e331e13d4ce8266 WatchSource:0}: Error finding container c7657be7f8ba0a444b20ebde832671e073007ee16517aa6e2e331e13d4ce8266: Status 404 returned error can't find the container with id c7657be7f8ba0a444b20ebde832671e073007ee16517aa6e2e331e13d4ce8266
Sep 10 02:53:53.946624 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:53.946584    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2m2kv"
Sep 10 02:53:54.005328 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:54.005296    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-sv8sd" event={"ID":"9f5cecb4-ee84-45f3-a3e9-5ef4367c02b6","Type":"ContainerStarted","Data":"c7657be7f8ba0a444b20ebde832671e073007ee16517aa6e2e331e13d4ce8266"}
Sep 10 02:53:55.012239 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:55.012157    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-sv8sd" event={"ID":"9f5cecb4-ee84-45f3-a3e9-5ef4367c02b6","Type":"ContainerStarted","Data":"3ab16bd9c98cfcf3d452c9d7aa6f28faa0ae51cda4cb3a075346393c3ebde137"}
Sep 10 02:53:55.012686 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:55.012313    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-sv8sd"
Sep 10 02:53:55.029121 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:55.029058    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-sv8sd" podStartSLOduration=2.427099472 podStartE2EDuration="4.029042519s" totalImagesPullingTime="1.601943047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:53:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:53:55.027909928 +0000 UTC m=+425.402995850" watchObservedRunningTime="2026-09-10 02:53:55.029042519 +0000 UTC m=+425.404128440"
Sep 10 02:53:55.707428 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:55.707396    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-52nqn"]
Sep 10 02:53:55.709860 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:55.709839    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-52nqn"
Sep 10 02:53:55.709860 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:55.709855    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:53:55.711912 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:55.711890    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 10 02:53:55.712003 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:55.711897    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 10 02:53:55.712175 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:55.712161    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-pdldg\""
Sep 10 02:53:55.719132 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:55.719097    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-52nqn"]
Sep 10 02:53:55.881649 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:55.881624    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c41ef279-4fec-4d1b-a20d-17144bbf0b09-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-52nqn\" (UID: \"c41ef279-4fec-4d1b-a20d-17144bbf0b09\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-52nqn"
Sep 10 02:53:55.881747 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:55.881662    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnfpw\" (UniqueName: \"kubernetes.io/projected/c41ef279-4fec-4d1b-a20d-17144bbf0b09-kube-api-access-bnfpw\") pod \"kuadrant-console-plugin-fbdd4f445-52nqn\" (UID: \"c41ef279-4fec-4d1b-a20d-17144bbf0b09\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-52nqn"
Sep 10 02:53:55.881747 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:55.881732    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c41ef279-4fec-4d1b-a20d-17144bbf0b09-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-52nqn\" (UID: \"c41ef279-4fec-4d1b-a20d-17144bbf0b09\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-52nqn"
Sep 10 02:53:55.958721 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:55.958673    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5tcnr"
Sep 10 02:53:55.982832 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:55.982809    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c41ef279-4fec-4d1b-a20d-17144bbf0b09-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-52nqn\" (UID: \"c41ef279-4fec-4d1b-a20d-17144bbf0b09\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-52nqn"
Sep 10 02:53:55.982941 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:55.982844    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnfpw\" (UniqueName: \"kubernetes.io/projected/c41ef279-4fec-4d1b-a20d-17144bbf0b09-kube-api-access-bnfpw\") pod \"kuadrant-console-plugin-fbdd4f445-52nqn\" (UID: \"c41ef279-4fec-4d1b-a20d-17144bbf0b09\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-52nqn"
Sep 10 02:53:55.982941 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:55.982890    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c41ef279-4fec-4d1b-a20d-17144bbf0b09-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-52nqn\" (UID: \"c41ef279-4fec-4d1b-a20d-17144bbf0b09\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-52nqn"
Sep 10 02:53:55.983625 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:55.983602    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c41ef279-4fec-4d1b-a20d-17144bbf0b09-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-52nqn\" (UID: \"c41ef279-4fec-4d1b-a20d-17144bbf0b09\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-52nqn"
Sep 10 02:53:55.985678 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:55.985655    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c41ef279-4fec-4d1b-a20d-17144bbf0b09-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-52nqn\" (UID: \"c41ef279-4fec-4d1b-a20d-17144bbf0b09\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-52nqn"
Sep 10 02:53:55.990781 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:55.990756    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnfpw\" (UniqueName: \"kubernetes.io/projected/c41ef279-4fec-4d1b-a20d-17144bbf0b09-kube-api-access-bnfpw\") pod \"kuadrant-console-plugin-fbdd4f445-52nqn\" (UID: \"c41ef279-4fec-4d1b-a20d-17144bbf0b09\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-52nqn"
Sep 10 02:53:56.018744 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:56.018729    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-52nqn"
Sep 10 02:53:56.032003 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:56.031978    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:53:56.130920 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:56.130888    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:53:56.287879 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:56.287839    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-52nqn"]
Sep 10 02:53:56.290455 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:53:56.290430    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc41ef279_4fec_4d1b_a20d_17144bbf0b09.slice/crio-337d77fd40370a9135d2a211acdc7ba74674e66d686abafe42025942afe22cd0 WatchSource:0}: Error finding container 337d77fd40370a9135d2a211acdc7ba74674e66d686abafe42025942afe22cd0: Status 404 returned error can't find the container with id 337d77fd40370a9135d2a211acdc7ba74674e66d686abafe42025942afe22cd0
Sep 10 02:53:57.025854 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:53:57.025820    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-52nqn" event={"ID":"c41ef279-4fec-4d1b-a20d-17144bbf0b09","Type":"ContainerStarted","Data":"337d77fd40370a9135d2a211acdc7ba74674e66d686abafe42025942afe22cd0"}
Sep 10 02:54:00.428148 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:00.428099    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:54:00.528044 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:00.528012    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:54:00.642446 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:00.642407    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:54:02.056545 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:02.056509    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-52nqn" event={"ID":"c41ef279-4fec-4d1b-a20d-17144bbf0b09","Type":"ContainerStarted","Data":"37e8fa65d92b426d702bfb7dbddf578181efa92759da40bc74d9a47be6b4da45"}
Sep 10 02:54:02.072072 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:02.072017    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-52nqn" podStartSLOduration=2.304731402 podStartE2EDuration="7.071998767s" totalImagesPullingTime="4.767267365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:54:02.070397792 +0000 UTC m=+432.445483712" watchObservedRunningTime="2026-09-10 02:54:02.071998767 +0000 UTC m=+432.447084760"
Sep 10 02:54:04.007259 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:04.007229    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9dx99"
Sep 10 02:54:05.624706 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:05.624673    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q5f75"]
Sep 10 02:54:05.629350 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:05.629333    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q5f75"
Sep 10 02:54:05.638285 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:05.638259    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q5f75"]
Sep 10 02:54:05.654800 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:05.654775    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkq9h\" (UniqueName: \"kubernetes.io/projected/14f8cb48-f4b2-4f5c-b32d-c25f0b14c152-kube-api-access-nkq9h\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-q5f75\" (UID: \"14f8cb48-f4b2-4f5c-b32d-c25f0b14c152\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q5f75"
Sep 10 02:54:05.654917 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:05.654809    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/14f8cb48-f4b2-4f5c-b32d-c25f0b14c152-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-q5f75\" (UID: \"14f8cb48-f4b2-4f5c-b32d-c25f0b14c152\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q5f75"
Sep 10 02:54:05.755995 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:05.755970    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkq9h\" (UniqueName: \"kubernetes.io/projected/14f8cb48-f4b2-4f5c-b32d-c25f0b14c152-kube-api-access-nkq9h\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-q5f75\" (UID: \"14f8cb48-f4b2-4f5c-b32d-c25f0b14c152\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q5f75"
Sep 10 02:54:05.756156 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:05.756010    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/14f8cb48-f4b2-4f5c-b32d-c25f0b14c152-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-q5f75\" (UID: \"14f8cb48-f4b2-4f5c-b32d-c25f0b14c152\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q5f75"
Sep 10 02:54:05.756429 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:05.756405    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/14f8cb48-f4b2-4f5c-b32d-c25f0b14c152-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-q5f75\" (UID: \"14f8cb48-f4b2-4f5c-b32d-c25f0b14c152\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q5f75"
Sep 10 02:54:05.763889 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:05.763868    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkq9h\" (UniqueName: \"kubernetes.io/projected/14f8cb48-f4b2-4f5c-b32d-c25f0b14c152-kube-api-access-nkq9h\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-q5f75\" (UID: \"14f8cb48-f4b2-4f5c-b32d-c25f0b14c152\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q5f75"
Sep 10 02:54:05.940202 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:05.940128    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q5f75"
Sep 10 02:54:06.021865 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.021840    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-sv8sd"
Sep 10 02:54:06.141534 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.141505    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9dx99"]
Sep 10 02:54:06.141719 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.141697    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9dx99" podUID="da5f1e33-ba7e-424e-a870-f800d1cd5d31" containerName="manager" containerID="cri-o://b305d7bbd07ecf4de49a06b46bd0b5f9cb5a73ffb42fb0a6a9fe742c6352762c" gracePeriod=2
Sep 10 02:54:06.161610 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.161582    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9dx99"]
Sep 10 02:54:06.174323 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:54:06.174284    2558 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"da5f1e33-ba7e-424e-a870-f800d1cd5d31\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-9dx99\": pods \"kuadrant-operator-controller-manager-76f8f686bf-9dx99\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9dx99"
Sep 10 02:54:06.175327 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.175302    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q5f75"]
Sep 10 02:54:06.188364 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.188339    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q5f75"]
Sep 10 02:54:06.200794 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.200774    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5tcnr"]
Sep 10 02:54:06.200994 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.200970    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5tcnr" podUID="8f3d3edb-f9f2-4e33-9fde-98e84ed8b3fc" containerName="manager" containerID="cri-o://d4c893df9abaef260338d35e52a2ecf0410e9f5e4bac4efab13c9182efe5e6ea" gracePeriod=2
Sep 10 02:54:06.209047 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.209026    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-27644"]
Sep 10 02:54:06.209348 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.209334    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da5f1e33-ba7e-424e-a870-f800d1cd5d31" containerName="manager"
Sep 10 02:54:06.209398 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.209350    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="da5f1e33-ba7e-424e-a870-f800d1cd5d31" containerName="manager"
Sep 10 02:54:06.209439 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.209411    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="da5f1e33-ba7e-424e-a870-f800d1cd5d31" containerName="manager"
Sep 10 02:54:06.214246 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:54:06.214213    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-9dx99\" is forbidden: User \"system:node:ip-10-0-128-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-101.ec2.internal' and this object" podUID="da5f1e33-ba7e-424e-a870-f800d1cd5d31" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9dx99"
Sep 10 02:54:06.214541 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.214521    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-27644"
Sep 10 02:54:06.216711 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:54:06.216676    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5tcnr\" is forbidden: User \"system:node:ip-10-0-128-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-101.ec2.internal' and this object" podUID="8f3d3edb-f9f2-4e33-9fde-98e84ed8b3fc" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5tcnr"
Sep 10 02:54:06.217367 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.217344    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5tcnr"]
Sep 10 02:54:06.223794 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:54:06.223767    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-9dx99\" is forbidden: User \"system:node:ip-10-0-128-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-101.ec2.internal' and this object" podUID="da5f1e33-ba7e-424e-a870-f800d1cd5d31" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9dx99"
Sep 10 02:54:06.228990 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.228972    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-27644"]
Sep 10 02:54:06.239380 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.239356    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-27xfc"]
Sep 10 02:54:06.239677 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.239661    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f3d3edb-f9f2-4e33-9fde-98e84ed8b3fc" containerName="manager"
Sep 10 02:54:06.239677 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.239677    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f3d3edb-f9f2-4e33-9fde-98e84ed8b3fc" containerName="manager"
Sep 10 02:54:06.239790 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.239733    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f3d3edb-f9f2-4e33-9fde-98e84ed8b3fc" containerName="manager"
Sep 10 02:54:06.242810 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.242777    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-27xfc"
Sep 10 02:54:06.262486 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.262207    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-27xfc"]
Sep 10 02:54:06.295715 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:54:06.293034    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5tcnr\" is forbidden: User \"system:node:ip-10-0-128-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-101.ec2.internal' and this object" podUID="8f3d3edb-f9f2-4e33-9fde-98e84ed8b3fc" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5tcnr"
Sep 10 02:54:06.295715 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:54:06.295155    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-9dx99\" is forbidden: User \"system:node:ip-10-0-128-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-101.ec2.internal' and this object" podUID="da5f1e33-ba7e-424e-a870-f800d1cd5d31" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9dx99"
Sep 10 02:54:06.360589 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.360549    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/16c6a4d7-f087-4b7a-bb70-e3813dac85de-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-27644\" (UID: \"16c6a4d7-f087-4b7a-bb70-e3813dac85de\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-27644"
Sep 10 02:54:06.360762 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.360601    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-828tx\" (UniqueName: \"kubernetes.io/projected/16c6a4d7-f087-4b7a-bb70-e3813dac85de-kube-api-access-828tx\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-27644\" (UID: \"16c6a4d7-f087-4b7a-bb70-e3813dac85de\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-27644"
Sep 10 02:54:06.360762 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.360720    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7h4k\" (UniqueName: \"kubernetes.io/projected/d75ae269-1b0b-485b-97db-2d005740e830-kube-api-access-r7h4k\") pod \"limitador-operator-controller-manager-74495c69f8-27xfc\" (UID: \"d75ae269-1b0b-485b-97db-2d005740e830\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-27xfc"
Sep 10 02:54:06.361163 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:54:06.361127    2558 log.go:32] "RunPodSandbox from runtime service failed" err=<
Sep 10 02:54:06.361163 ip-10-0-128-101 kubenswrapper[2558]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-q5f75_kuadrant-system_14f8cb48-f4b2-4f5c-b32d-c25f0b14c152_0(d83425574212c28eaa723c21c3ac1f4cc682057688cb85a841711a82c7c83d1f): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-q5f75 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"d83425574212c28eaa723c21c3ac1f4cc682057688cb85a841711a82c7c83d1f" Netns:"/var/run/netns/2dd4ebec-5d12-4183-9efb-e3bacc32e47e" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-q5f75;K8S_POD_INFRA_CONTAINER_ID=d83425574212c28eaa723c21c3ac1f4cc682057688cb85a841711a82c7c83d1f;K8S_POD_UID=14f8cb48-f4b2-4f5c-b32d-c25f0b14c152" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q5f75] networking: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q5f75/14f8cb48-f4b2-4f5c-b32d-c25f0b14c152:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q5f75 d83425574212c28eaa723c21c3ac1f4cc682057688cb85a841711a82c7c83d1f network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:86:00:21 [10.134.0.33/23]
Sep 10 02:54:06.361163 ip-10-0-128-101 kubenswrapper[2558]:         '
Sep 10 02:54:06.361163 ip-10-0-128-101 kubenswrapper[2558]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 10 02:54:06.361163 ip-10-0-128-101 kubenswrapper[2558]:  >
Sep 10 02:54:06.361456 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:54:06.361179    2558 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Sep 10 02:54:06.361456 ip-10-0-128-101 kubenswrapper[2558]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-q5f75_kuadrant-system_14f8cb48-f4b2-4f5c-b32d-c25f0b14c152_0(d83425574212c28eaa723c21c3ac1f4cc682057688cb85a841711a82c7c83d1f): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-q5f75 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"d83425574212c28eaa723c21c3ac1f4cc682057688cb85a841711a82c7c83d1f" Netns:"/var/run/netns/2dd4ebec-5d12-4183-9efb-e3bacc32e47e" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-q5f75;K8S_POD_INFRA_CONTAINER_ID=d83425574212c28eaa723c21c3ac1f4cc682057688cb85a841711a82c7c83d1f;K8S_POD_UID=14f8cb48-f4b2-4f5c-b32d-c25f0b14c152" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q5f75] networking: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q5f75/14f8cb48-f4b2-4f5c-b32d-c25f0b14c152:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q5f75 d83425574212c28eaa723c21c3ac1f4cc682057688cb85a841711a82c7c83d1f network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:86:00:21 [10.134.0.33/23]
Sep 10 02:54:06.361456 ip-10-0-128-101 kubenswrapper[2558]:         '
Sep 10 02:54:06.361456 ip-10-0-128-101 kubenswrapper[2558]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 10 02:54:06.361456 ip-10-0-128-101 kubenswrapper[2558]:  > pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q5f75"
Sep 10 02:54:06.437696 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.437672    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9dx99"
Sep 10 02:54:06.439695 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:54:06.439668    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-9dx99\" is forbidden: User \"system:node:ip-10-0-128-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-101.ec2.internal' and this object" podUID="da5f1e33-ba7e-424e-a870-f800d1cd5d31" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9dx99"
Sep 10 02:54:06.441549 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:54:06.441521    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5tcnr\" is forbidden: User \"system:node:ip-10-0-128-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-101.ec2.internal' and this object" podUID="8f3d3edb-f9f2-4e33-9fde-98e84ed8b3fc" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5tcnr"
Sep 10 02:54:06.444406 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.444388    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5tcnr"
Sep 10 02:54:06.446171 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:54:06.446149    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5tcnr\" is forbidden: User \"system:node:ip-10-0-128-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-101.ec2.internal' and this object" podUID="8f3d3edb-f9f2-4e33-9fde-98e84ed8b3fc" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5tcnr"
Sep 10 02:54:06.447669 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:54:06.447648    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-9dx99\" is forbidden: User \"system:node:ip-10-0-128-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-101.ec2.internal' and this object" podUID="da5f1e33-ba7e-424e-a870-f800d1cd5d31" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9dx99"
Sep 10 02:54:06.461037 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.460992    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/da5f1e33-ba7e-424e-a870-f800d1cd5d31-extensions-socket-volume\") pod \"da5f1e33-ba7e-424e-a870-f800d1cd5d31\" (UID: \"da5f1e33-ba7e-424e-a870-f800d1cd5d31\") "
Sep 10 02:54:06.461037 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.461032    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjlz4\" (UniqueName: \"kubernetes.io/projected/da5f1e33-ba7e-424e-a870-f800d1cd5d31-kube-api-access-bjlz4\") pod \"da5f1e33-ba7e-424e-a870-f800d1cd5d31\" (UID: \"da5f1e33-ba7e-424e-a870-f800d1cd5d31\") "
Sep 10 02:54:06.461225 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.461057    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgkzs\" (UniqueName: \"kubernetes.io/projected/8f3d3edb-f9f2-4e33-9fde-98e84ed8b3fc-kube-api-access-mgkzs\") pod \"8f3d3edb-f9f2-4e33-9fde-98e84ed8b3fc\" (UID: \"8f3d3edb-f9f2-4e33-9fde-98e84ed8b3fc\") "
Sep 10 02:54:06.461225 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.461205    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7h4k\" (UniqueName: \"kubernetes.io/projected/d75ae269-1b0b-485b-97db-2d005740e830-kube-api-access-r7h4k\") pod \"limitador-operator-controller-manager-74495c69f8-27xfc\" (UID: \"d75ae269-1b0b-485b-97db-2d005740e830\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-27xfc"
Sep 10 02:54:06.461329 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.461245    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/16c6a4d7-f087-4b7a-bb70-e3813dac85de-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-27644\" (UID: \"16c6a4d7-f087-4b7a-bb70-e3813dac85de\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-27644"
Sep 10 02:54:06.461329 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.461267    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-828tx\" (UniqueName: \"kubernetes.io/projected/16c6a4d7-f087-4b7a-bb70-e3813dac85de-kube-api-access-828tx\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-27644\" (UID: \"16c6a4d7-f087-4b7a-bb70-e3813dac85de\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-27644"
Sep 10 02:54:06.461571 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.461545    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da5f1e33-ba7e-424e-a870-f800d1cd5d31-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "da5f1e33-ba7e-424e-a870-f800d1cd5d31" (UID: "da5f1e33-ba7e-424e-a870-f800d1cd5d31"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 02:54:06.461695 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.461674    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/16c6a4d7-f087-4b7a-bb70-e3813dac85de-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-27644\" (UID: \"16c6a4d7-f087-4b7a-bb70-e3813dac85de\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-27644"
Sep 10 02:54:06.463198 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.463175    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da5f1e33-ba7e-424e-a870-f800d1cd5d31-kube-api-access-bjlz4" (OuterVolumeSpecName: "kube-api-access-bjlz4") pod "da5f1e33-ba7e-424e-a870-f800d1cd5d31" (UID: "da5f1e33-ba7e-424e-a870-f800d1cd5d31"). InnerVolumeSpecName "kube-api-access-bjlz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 02:54:06.463271 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.463208    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f3d3edb-f9f2-4e33-9fde-98e84ed8b3fc-kube-api-access-mgkzs" (OuterVolumeSpecName: "kube-api-access-mgkzs") pod "8f3d3edb-f9f2-4e33-9fde-98e84ed8b3fc" (UID: "8f3d3edb-f9f2-4e33-9fde-98e84ed8b3fc"). InnerVolumeSpecName "kube-api-access-mgkzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 02:54:06.480014 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.479994    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-828tx\" (UniqueName: \"kubernetes.io/projected/16c6a4d7-f087-4b7a-bb70-e3813dac85de-kube-api-access-828tx\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-27644\" (UID: \"16c6a4d7-f087-4b7a-bb70-e3813dac85de\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-27644"
Sep 10 02:54:06.480936 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.480917    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7h4k\" (UniqueName: \"kubernetes.io/projected/d75ae269-1b0b-485b-97db-2d005740e830-kube-api-access-r7h4k\") pod \"limitador-operator-controller-manager-74495c69f8-27xfc\" (UID: \"d75ae269-1b0b-485b-97db-2d005740e830\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-27xfc"
Sep 10 02:54:06.561969 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.561935    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjlz4\" (UniqueName: \"kubernetes.io/projected/da5f1e33-ba7e-424e-a870-f800d1cd5d31-kube-api-access-bjlz4\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:54:06.561969 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.561964    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgkzs\" (UniqueName: \"kubernetes.io/projected/8f3d3edb-f9f2-4e33-9fde-98e84ed8b3fc-kube-api-access-mgkzs\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:54:06.561969 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.561975    2558 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/da5f1e33-ba7e-424e-a870-f800d1cd5d31-extensions-socket-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:54:06.571928 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.571899    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-27644"
Sep 10 02:54:06.578268 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.578242    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-27xfc"
Sep 10 02:54:06.853314 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.853288    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-27644"]
Sep 10 02:54:06.854806 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:54:06.854775    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd75ae269_1b0b_485b_97db_2d005740e830.slice/crio-1b8a4d0dd9241ac86c6200abc3489a42e9f7c32ee42973f976d1b0269acf0761 WatchSource:0}: Error finding container 1b8a4d0dd9241ac86c6200abc3489a42e9f7c32ee42973f976d1b0269acf0761: Status 404 returned error can't find the container with id 1b8a4d0dd9241ac86c6200abc3489a42e9f7c32ee42973f976d1b0269acf0761
Sep 10 02:54:06.855595 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:06.855520    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-27xfc"]
Sep 10 02:54:06.855804 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:54:06.855785    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16c6a4d7_f087_4b7a_bb70_e3813dac85de.slice/crio-2bf34fdc8efc142a78a64c5a1276b9c26cd5600796d9487b1bf38da93b4b1a4d WatchSource:0}: Error finding container 2bf34fdc8efc142a78a64c5a1276b9c26cd5600796d9487b1bf38da93b4b1a4d: Status 404 returned error can't find the container with id 2bf34fdc8efc142a78a64c5a1276b9c26cd5600796d9487b1bf38da93b4b1a4d
Sep 10 02:54:07.085799 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:07.085708    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-27xfc" event={"ID":"d75ae269-1b0b-485b-97db-2d005740e830","Type":"ContainerStarted","Data":"957c909ee57d09c8ef969fc8e175ce35eef15cbdba769ef8bc38fcc7f4cce9aa"}
Sep 10 02:54:07.085799 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:07.085767    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-27xfc" event={"ID":"d75ae269-1b0b-485b-97db-2d005740e830","Type":"ContainerStarted","Data":"1b8a4d0dd9241ac86c6200abc3489a42e9f7c32ee42973f976d1b0269acf0761"}
Sep 10 02:54:07.085991 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:07.085824    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-27xfc"
Sep 10 02:54:07.087419 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:07.087398    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-27644" event={"ID":"16c6a4d7-f087-4b7a-bb70-e3813dac85de","Type":"ContainerStarted","Data":"63226e4f8b3dc45a573bdbc8e8550605332541a2d2db902b3023b5ba5da86956"}
Sep 10 02:54:07.087535 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:07.087432    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-27644" event={"ID":"16c6a4d7-f087-4b7a-bb70-e3813dac85de","Type":"ContainerStarted","Data":"2bf34fdc8efc142a78a64c5a1276b9c26cd5600796d9487b1bf38da93b4b1a4d"}
Sep 10 02:54:07.087535 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:07.087460    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-27644"
Sep 10 02:54:07.087535 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:54:07.087504    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-9dx99\" is forbidden: User \"system:node:ip-10-0-128-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-101.ec2.internal' and this object" podUID="da5f1e33-ba7e-424e-a870-f800d1cd5d31" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9dx99"
Sep 10 02:54:07.088702 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:07.088679    2558 generic.go:356] "Generic (PLEG): container finished" podID="8f3d3edb-f9f2-4e33-9fde-98e84ed8b3fc" containerID="d4c893df9abaef260338d35e52a2ecf0410e9f5e4bac4efab13c9182efe5e6ea" exitCode=0
Sep 10 02:54:07.088788 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:07.088727    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5tcnr"
Sep 10 02:54:07.088788 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:07.088753    2558 scope.go:117] "RemoveContainer" containerID="d4c893df9abaef260338d35e52a2ecf0410e9f5e4bac4efab13c9182efe5e6ea"
Sep 10 02:54:07.090361 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:07.090332    2558 generic.go:356] "Generic (PLEG): container finished" podID="da5f1e33-ba7e-424e-a870-f800d1cd5d31" containerID="b305d7bbd07ecf4de49a06b46bd0b5f9cb5a73ffb42fb0a6a9fe742c6352762c" exitCode=0
Sep 10 02:54:07.090477 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:07.090376    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9dx99"
Sep 10 02:54:07.090589 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:07.090573    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q5f75"
Sep 10 02:54:07.098741 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:07.098724    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q5f75"
Sep 10 02:54:07.099375 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:07.099358    2558 scope.go:117] "RemoveContainer" containerID="d4c893df9abaef260338d35e52a2ecf0410e9f5e4bac4efab13c9182efe5e6ea"
Sep 10 02:54:07.099608 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:54:07.099592    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4c893df9abaef260338d35e52a2ecf0410e9f5e4bac4efab13c9182efe5e6ea\": container with ID starting with d4c893df9abaef260338d35e52a2ecf0410e9f5e4bac4efab13c9182efe5e6ea not found: ID does not exist" containerID="d4c893df9abaef260338d35e52a2ecf0410e9f5e4bac4efab13c9182efe5e6ea"
Sep 10 02:54:07.099680 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:07.099613    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4c893df9abaef260338d35e52a2ecf0410e9f5e4bac4efab13c9182efe5e6ea"} err="failed to get container status \"d4c893df9abaef260338d35e52a2ecf0410e9f5e4bac4efab13c9182efe5e6ea\": rpc error: code = NotFound desc = could not find container \"d4c893df9abaef260338d35e52a2ecf0410e9f5e4bac4efab13c9182efe5e6ea\": container with ID starting with d4c893df9abaef260338d35e52a2ecf0410e9f5e4bac4efab13c9182efe5e6ea not found: ID does not exist"
Sep 10 02:54:07.099680 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:07.099630    2558 scope.go:117] "RemoveContainer" containerID="b305d7bbd07ecf4de49a06b46bd0b5f9cb5a73ffb42fb0a6a9fe742c6352762c"
Sep 10 02:54:07.104207 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:07.104168    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-27xfc" podStartSLOduration=1.104143581 podStartE2EDuration="1.104143581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:54:07.102474443 +0000 UTC m=+437.477560365" watchObservedRunningTime="2026-09-10 02:54:07.104143581 +0000 UTC m=+437.479229503"
Sep 10 02:54:07.104299 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:54:07.104199    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5tcnr\" is forbidden: User \"system:node:ip-10-0-128-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-101.ec2.internal' and this object" podUID="8f3d3edb-f9f2-4e33-9fde-98e84ed8b3fc" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5tcnr"
Sep 10 02:54:07.107589 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:07.107570    2558 scope.go:117] "RemoveContainer" containerID="b305d7bbd07ecf4de49a06b46bd0b5f9cb5a73ffb42fb0a6a9fe742c6352762c"
Sep 10 02:54:07.107849 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:54:07.107829    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b305d7bbd07ecf4de49a06b46bd0b5f9cb5a73ffb42fb0a6a9fe742c6352762c\": container with ID starting with b305d7bbd07ecf4de49a06b46bd0b5f9cb5a73ffb42fb0a6a9fe742c6352762c not found: ID does not exist" containerID="b305d7bbd07ecf4de49a06b46bd0b5f9cb5a73ffb42fb0a6a9fe742c6352762c"
Sep 10 02:54:07.107944 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:07.107853    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b305d7bbd07ecf4de49a06b46bd0b5f9cb5a73ffb42fb0a6a9fe742c6352762c"} err="failed to get container status \"b305d7bbd07ecf4de49a06b46bd0b5f9cb5a73ffb42fb0a6a9fe742c6352762c\": rpc error: code = NotFound desc = could not find container \"b305d7bbd07ecf4de49a06b46bd0b5f9cb5a73ffb42fb0a6a9fe742c6352762c\": container with ID starting with b305d7bbd07ecf4de49a06b46bd0b5f9cb5a73ffb42fb0a6a9fe742c6352762c not found: ID does not exist"
Sep 10 02:54:07.123348 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:07.123312    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-27644" podStartSLOduration=1.123299465 podStartE2EDuration="1.123299465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:54:07.122039191 +0000 UTC m=+437.497125112" watchObservedRunningTime="2026-09-10 02:54:07.123299465 +0000 UTC m=+437.498385385"
Sep 10 02:54:07.123764 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:54:07.123742    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-q5f75\" is forbidden: User \"system:node:ip-10-0-128-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-101.ec2.internal' and this object" podUID="14f8cb48-f4b2-4f5c-b32d-c25f0b14c152" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q5f75"
Sep 10 02:54:07.125374 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:54:07.125352    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5tcnr\" is forbidden: User \"system:node:ip-10-0-128-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-101.ec2.internal' and this object" podUID="8f3d3edb-f9f2-4e33-9fde-98e84ed8b3fc" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5tcnr"
Sep 10 02:54:07.127435 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:54:07.127418    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-9dx99\" is forbidden: User \"system:node:ip-10-0-128-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-101.ec2.internal' and this object" podUID="da5f1e33-ba7e-424e-a870-f800d1cd5d31" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9dx99"
Sep 10 02:54:07.266254 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:07.266229    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkq9h\" (UniqueName: \"kubernetes.io/projected/14f8cb48-f4b2-4f5c-b32d-c25f0b14c152-kube-api-access-nkq9h\") pod \"14f8cb48-f4b2-4f5c-b32d-c25f0b14c152\" (UID: \"14f8cb48-f4b2-4f5c-b32d-c25f0b14c152\") "
Sep 10 02:54:07.266384 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:07.266271    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/14f8cb48-f4b2-4f5c-b32d-c25f0b14c152-extensions-socket-volume\") pod \"14f8cb48-f4b2-4f5c-b32d-c25f0b14c152\" (UID: \"14f8cb48-f4b2-4f5c-b32d-c25f0b14c152\") "
Sep 10 02:54:07.266554 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:07.266530    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f8cb48-f4b2-4f5c-b32d-c25f0b14c152-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "14f8cb48-f4b2-4f5c-b32d-c25f0b14c152" (UID: "14f8cb48-f4b2-4f5c-b32d-c25f0b14c152"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 02:54:07.268256 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:07.268235    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14f8cb48-f4b2-4f5c-b32d-c25f0b14c152-kube-api-access-nkq9h" (OuterVolumeSpecName: "kube-api-access-nkq9h") pod "14f8cb48-f4b2-4f5c-b32d-c25f0b14c152" (UID: "14f8cb48-f4b2-4f5c-b32d-c25f0b14c152"). InnerVolumeSpecName "kube-api-access-nkq9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 02:54:07.366902 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:07.366835    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkq9h\" (UniqueName: \"kubernetes.io/projected/14f8cb48-f4b2-4f5c-b32d-c25f0b14c152-kube-api-access-nkq9h\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:54:07.366902 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:07.366857    2558 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/14f8cb48-f4b2-4f5c-b32d-c25f0b14c152-extensions-socket-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:54:08.068848 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:08.068765    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14f8cb48-f4b2-4f5c-b32d-c25f0b14c152" path="/var/lib/kubelet/pods/14f8cb48-f4b2-4f5c-b32d-c25f0b14c152/volumes"
Sep 10 02:54:08.069320 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:08.069078    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f3d3edb-f9f2-4e33-9fde-98e84ed8b3fc" path="/var/lib/kubelet/pods/8f3d3edb-f9f2-4e33-9fde-98e84ed8b3fc/volumes"
Sep 10 02:54:08.069557 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:08.069535    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da5f1e33-ba7e-424e-a870-f800d1cd5d31" path="/var/lib/kubelet/pods/da5f1e33-ba7e-424e-a870-f800d1cd5d31/volumes"
Sep 10 02:54:08.096668 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:08.096645    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q5f75"
Sep 10 02:54:08.100657 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:54:08.100632    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-q5f75\" is forbidden: User \"system:node:ip-10-0-128-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-101.ec2.internal' and this object" podUID="14f8cb48-f4b2-4f5c-b32d-c25f0b14c152" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q5f75"
Sep 10 02:54:09.266555 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:09.266520    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:54:09.387435 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:09.387404    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:54:10.067308 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:54:10.067265    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-q5f75\" is forbidden: User \"system:node:ip-10-0-128-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-101.ec2.internal' and this object" podUID="14f8cb48-f4b2-4f5c-b32d-c25f0b14c152" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q5f75"
Sep 10 02:54:11.796813 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:11.796779    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:54:11.898341 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:11.898307    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:54:11.998424 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:11.998391    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:54:15.066363 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:15.066331    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:54:15.166473 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:15.166445    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:54:18.098746 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:18.098712    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-27644"
Sep 10 02:54:18.099148 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:18.098779    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-27xfc"
Sep 10 02:54:18.411081 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:18.411053    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgc8x"]
Sep 10 02:54:18.415259 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:18.415240    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgc8x"
Sep 10 02:54:18.431790 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:18.431766    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgc8x"]
Sep 10 02:54:18.445176 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:18.445155    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nflss\" (UniqueName: \"kubernetes.io/projected/6b1e6ef4-b258-4a0f-aa44-d8c049d8909e-kube-api-access-nflss\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mgc8x\" (UID: \"6b1e6ef4-b258-4a0f-aa44-d8c049d8909e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgc8x"
Sep 10 02:54:18.445258 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:18.445188    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b1e6ef4-b258-4a0f-aa44-d8c049d8909e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mgc8x\" (UID: \"6b1e6ef4-b258-4a0f-aa44-d8c049d8909e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgc8x"
Sep 10 02:54:18.545574 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:18.545545    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nflss\" (UniqueName: \"kubernetes.io/projected/6b1e6ef4-b258-4a0f-aa44-d8c049d8909e-kube-api-access-nflss\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mgc8x\" (UID: \"6b1e6ef4-b258-4a0f-aa44-d8c049d8909e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgc8x"
Sep 10 02:54:18.545671 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:18.545584    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b1e6ef4-b258-4a0f-aa44-d8c049d8909e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mgc8x\" (UID: \"6b1e6ef4-b258-4a0f-aa44-d8c049d8909e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgc8x"
Sep 10 02:54:18.545956 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:18.545935    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b1e6ef4-b258-4a0f-aa44-d8c049d8909e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mgc8x\" (UID: \"6b1e6ef4-b258-4a0f-aa44-d8c049d8909e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgc8x"
Sep 10 02:54:18.557068 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:18.557044    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nflss\" (UniqueName: \"kubernetes.io/projected/6b1e6ef4-b258-4a0f-aa44-d8c049d8909e-kube-api-access-nflss\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mgc8x\" (UID: \"6b1e6ef4-b258-4a0f-aa44-d8c049d8909e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgc8x"
Sep 10 02:54:18.725041 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:18.725020    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgc8x"
Sep 10 02:54:18.996240 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:18.996219    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgc8x"]
Sep 10 02:54:18.998528 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:54:18.998493    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b1e6ef4_b258_4a0f_aa44_d8c049d8909e.slice/crio-b50996cf4e676e299df257dfb0a267b68de8999e5aca1da435a3ccde22297ce5 WatchSource:0}: Error finding container b50996cf4e676e299df257dfb0a267b68de8999e5aca1da435a3ccde22297ce5: Status 404 returned error can't find the container with id b50996cf4e676e299df257dfb0a267b68de8999e5aca1da435a3ccde22297ce5
Sep 10 02:54:19.152870 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:19.152842    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgc8x" event={"ID":"6b1e6ef4-b258-4a0f-aa44-d8c049d8909e","Type":"ContainerStarted","Data":"ff2fafeed5751d3ed7fe416f5048d3e4e107d89353826e8b8afd52b05675cde2"}
Sep 10 02:54:19.153174 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:19.152892    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgc8x" event={"ID":"6b1e6ef4-b258-4a0f-aa44-d8c049d8909e","Type":"ContainerStarted","Data":"b50996cf4e676e299df257dfb0a267b68de8999e5aca1da435a3ccde22297ce5"}
Sep 10 02:54:19.153174 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:19.152969    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgc8x"
Sep 10 02:54:19.173265 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:19.173207    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgc8x" podStartSLOduration=1.173194159 podStartE2EDuration="1.173194159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:54:19.172430943 +0000 UTC m=+449.547516864" watchObservedRunningTime="2026-09-10 02:54:19.173194159 +0000 UTC m=+449.548280079"
Sep 10 02:54:19.508488 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:19.508458    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:54:19.609727 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:19.609677    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:54:19.709004 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:19.708965    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:54:19.808974 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:19.808904    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:54:20.672484 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:20.672450    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:54:20.774079 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:20.774025    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:54:22.613115 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:22.613066    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:54:28.290661 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:28.290623    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:54:28.389901 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:28.389873    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:54:30.159676 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:30.159646    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgc8x"
Sep 10 02:54:30.232709 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:30.232681    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-27644"]
Sep 10 02:54:30.232921 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:30.232898    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-27644" podUID="16c6a4d7-f087-4b7a-bb70-e3813dac85de" containerName="manager" containerID="cri-o://63226e4f8b3dc45a573bdbc8e8550605332541a2d2db902b3023b5ba5da86956" gracePeriod=10
Sep 10 02:54:30.468483 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:30.468456    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-27644"
Sep 10 02:54:30.527216 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:30.527186    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/16c6a4d7-f087-4b7a-bb70-e3813dac85de-extensions-socket-volume\") pod \"16c6a4d7-f087-4b7a-bb70-e3813dac85de\" (UID: \"16c6a4d7-f087-4b7a-bb70-e3813dac85de\") "
Sep 10 02:54:30.527323 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:30.527221    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-828tx\" (UniqueName: \"kubernetes.io/projected/16c6a4d7-f087-4b7a-bb70-e3813dac85de-kube-api-access-828tx\") pod \"16c6a4d7-f087-4b7a-bb70-e3813dac85de\" (UID: \"16c6a4d7-f087-4b7a-bb70-e3813dac85de\") "
Sep 10 02:54:30.527540 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:30.527516    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c6a4d7-f087-4b7a-bb70-e3813dac85de-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "16c6a4d7-f087-4b7a-bb70-e3813dac85de" (UID: "16c6a4d7-f087-4b7a-bb70-e3813dac85de"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 02:54:30.529116 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:30.529082    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16c6a4d7-f087-4b7a-bb70-e3813dac85de-kube-api-access-828tx" (OuterVolumeSpecName: "kube-api-access-828tx") pod "16c6a4d7-f087-4b7a-bb70-e3813dac85de" (UID: "16c6a4d7-f087-4b7a-bb70-e3813dac85de"). InnerVolumeSpecName "kube-api-access-828tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 02:54:30.628590 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:30.628565    2558 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/16c6a4d7-f087-4b7a-bb70-e3813dac85de-extensions-socket-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:54:30.628590 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:30.628588    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-828tx\" (UniqueName: \"kubernetes.io/projected/16c6a4d7-f087-4b7a-bb70-e3813dac85de-kube-api-access-828tx\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:54:31.219846 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:31.219810    2558 generic.go:356] "Generic (PLEG): container finished" podID="16c6a4d7-f087-4b7a-bb70-e3813dac85de" containerID="63226e4f8b3dc45a573bdbc8e8550605332541a2d2db902b3023b5ba5da86956" exitCode=0
Sep 10 02:54:31.220279 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:31.219863    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-27644"
Sep 10 02:54:31.220279 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:31.219905    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-27644" event={"ID":"16c6a4d7-f087-4b7a-bb70-e3813dac85de","Type":"ContainerDied","Data":"63226e4f8b3dc45a573bdbc8e8550605332541a2d2db902b3023b5ba5da86956"}
Sep 10 02:54:31.220279 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:31.219971    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-27644" event={"ID":"16c6a4d7-f087-4b7a-bb70-e3813dac85de","Type":"ContainerDied","Data":"2bf34fdc8efc142a78a64c5a1276b9c26cd5600796d9487b1bf38da93b4b1a4d"}
Sep 10 02:54:31.220279 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:31.219996    2558 scope.go:117] "RemoveContainer" containerID="63226e4f8b3dc45a573bdbc8e8550605332541a2d2db902b3023b5ba5da86956"
Sep 10 02:54:31.230306 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:31.230284    2558 scope.go:117] "RemoveContainer" containerID="63226e4f8b3dc45a573bdbc8e8550605332541a2d2db902b3023b5ba5da86956"
Sep 10 02:54:31.230614 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:54:31.230594    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63226e4f8b3dc45a573bdbc8e8550605332541a2d2db902b3023b5ba5da86956\": container with ID starting with 63226e4f8b3dc45a573bdbc8e8550605332541a2d2db902b3023b5ba5da86956 not found: ID does not exist" containerID="63226e4f8b3dc45a573bdbc8e8550605332541a2d2db902b3023b5ba5da86956"
Sep 10 02:54:31.230658 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:31.230624    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63226e4f8b3dc45a573bdbc8e8550605332541a2d2db902b3023b5ba5da86956"} err="failed to get container status \"63226e4f8b3dc45a573bdbc8e8550605332541a2d2db902b3023b5ba5da86956\": rpc error: code = NotFound desc = could not find container \"63226e4f8b3dc45a573bdbc8e8550605332541a2d2db902b3023b5ba5da86956\": container with ID starting with 63226e4f8b3dc45a573bdbc8e8550605332541a2d2db902b3023b5ba5da86956 not found: ID does not exist"
Sep 10 02:54:31.246076 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:31.246049    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-27644"]
Sep 10 02:54:31.249452 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:31.249429    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-27644"]
Sep 10 02:54:32.069493 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:32.069461    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16c6a4d7-f087-4b7a-bb70-e3813dac85de" path="/var/lib/kubelet/pods/16c6a4d7-f087-4b7a-bb70-e3813dac85de/volumes"
Sep 10 02:54:34.577975 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:34.577944    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:54:34.935291 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:34.935216    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:54:50.711065 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:50.711034    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-84qc9"]
Sep 10 02:54:50.711458 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:50.711355    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16c6a4d7-f087-4b7a-bb70-e3813dac85de" containerName="manager"
Sep 10 02:54:50.711458 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:50.711369    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="16c6a4d7-f087-4b7a-bb70-e3813dac85de" containerName="manager"
Sep 10 02:54:50.711458 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:50.711418    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="16c6a4d7-f087-4b7a-bb70-e3813dac85de" containerName="manager"
Sep 10 02:54:50.713933 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:50.713915    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-84qc9"
Sep 10 02:54:50.716135 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:50.716113    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 10 02:54:50.717456 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:50.717407    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:54:50.725810 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:50.725785    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-84qc9"]
Sep 10 02:54:50.765212 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:50.765183    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bdd46bc7-4845-46f4-bd55-00131d9b7bd4-config-file\") pod \"limitador-limitador-8588749977-84qc9\" (UID: \"bdd46bc7-4845-46f4-bd55-00131d9b7bd4\") " pod="kuadrant-system/limitador-limitador-8588749977-84qc9"
Sep 10 02:54:50.765340 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:50.765217    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z99cc\" (UniqueName: \"kubernetes.io/projected/bdd46bc7-4845-46f4-bd55-00131d9b7bd4-kube-api-access-z99cc\") pod \"limitador-limitador-8588749977-84qc9\" (UID: \"bdd46bc7-4845-46f4-bd55-00131d9b7bd4\") " pod="kuadrant-system/limitador-limitador-8588749977-84qc9"
Sep 10 02:54:50.808917 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:50.808889    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-84qc9"]
Sep 10 02:54:50.817294 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:50.817266    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:54:50.865703 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:50.865679    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bdd46bc7-4845-46f4-bd55-00131d9b7bd4-config-file\") pod \"limitador-limitador-8588749977-84qc9\" (UID: \"bdd46bc7-4845-46f4-bd55-00131d9b7bd4\") " pod="kuadrant-system/limitador-limitador-8588749977-84qc9"
Sep 10 02:54:50.865811 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:50.865709    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z99cc\" (UniqueName: \"kubernetes.io/projected/bdd46bc7-4845-46f4-bd55-00131d9b7bd4-kube-api-access-z99cc\") pod \"limitador-limitador-8588749977-84qc9\" (UID: \"bdd46bc7-4845-46f4-bd55-00131d9b7bd4\") " pod="kuadrant-system/limitador-limitador-8588749977-84qc9"
Sep 10 02:54:50.866334 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:50.866317    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bdd46bc7-4845-46f4-bd55-00131d9b7bd4-config-file\") pod \"limitador-limitador-8588749977-84qc9\" (UID: \"bdd46bc7-4845-46f4-bd55-00131d9b7bd4\") " pod="kuadrant-system/limitador-limitador-8588749977-84qc9"
Sep 10 02:54:50.872462 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:50.872440    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:54:50.873419 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:50.873401    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z99cc\" (UniqueName: \"kubernetes.io/projected/bdd46bc7-4845-46f4-bd55-00131d9b7bd4-kube-api-access-z99cc\") pod \"limitador-limitador-8588749977-84qc9\" (UID: \"bdd46bc7-4845-46f4-bd55-00131d9b7bd4\") " pod="kuadrant-system/limitador-limitador-8588749977-84qc9"
Sep 10 02:54:51.026899 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:51.026840    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-84qc9"
Sep 10 02:54:51.299748 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:51.299724    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-84qc9"]
Sep 10 02:54:51.301719 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:54:51.301691    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdd46bc7_4845_46f4_bd55_00131d9b7bd4.slice/crio-47ef86c5d11e2e42ca83d6d96d376160b2384b4d5fad635c4e1399d3389d43fe WatchSource:0}: Error finding container 47ef86c5d11e2e42ca83d6d96d376160b2384b4d5fad635c4e1399d3389d43fe: Status 404 returned error can't find the container with id 47ef86c5d11e2e42ca83d6d96d376160b2384b4d5fad635c4e1399d3389d43fe
Sep 10 02:54:51.329339 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:51.329310    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-84qc9" event={"ID":"bdd46bc7-4845-46f4-bd55-00131d9b7bd4","Type":"ContainerStarted","Data":"47ef86c5d11e2e42ca83d6d96d376160b2384b4d5fad635c4e1399d3389d43fe"}
Sep 10 02:54:51.556407 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:51.556329    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:54:51.644701 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:51.644673    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:54:51.648823 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:51.648803    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-fvwhm"]
Sep 10 02:54:51.651929 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:51.651911    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-fvwhm"
Sep 10 02:54:51.654058 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:51.654041    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5stxp\""
Sep 10 02:54:51.657614 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:51.657592    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-fvwhm"]
Sep 10 02:54:51.774009 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:51.773986    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjwh6\" (UniqueName: \"kubernetes.io/projected/80cafcc1-65ea-4e0b-9394-e5dd56d9ea8b-kube-api-access-wjwh6\") pod \"authorino-59c974bd48-fvwhm\" (UID: \"80cafcc1-65ea-4e0b-9394-e5dd56d9ea8b\") " pod="kuadrant-system/authorino-59c974bd48-fvwhm"
Sep 10 02:54:51.874753 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:51.874684    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjwh6\" (UniqueName: \"kubernetes.io/projected/80cafcc1-65ea-4e0b-9394-e5dd56d9ea8b-kube-api-access-wjwh6\") pod \"authorino-59c974bd48-fvwhm\" (UID: \"80cafcc1-65ea-4e0b-9394-e5dd56d9ea8b\") " pod="kuadrant-system/authorino-59c974bd48-fvwhm"
Sep 10 02:54:51.883054 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:51.883031    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjwh6\" (UniqueName: \"kubernetes.io/projected/80cafcc1-65ea-4e0b-9394-e5dd56d9ea8b-kube-api-access-wjwh6\") pod \"authorino-59c974bd48-fvwhm\" (UID: \"80cafcc1-65ea-4e0b-9394-e5dd56d9ea8b\") " pod="kuadrant-system/authorino-59c974bd48-fvwhm"
Sep 10 02:54:51.962753 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:51.962733    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-fvwhm"
Sep 10 02:54:52.232046 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:52.232017    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-fvwhm"]
Sep 10 02:54:52.234653 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:54:52.234619    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80cafcc1_65ea_4e0b_9394_e5dd56d9ea8b.slice/crio-45cb69f87eb92a04b6da8c806c8679d8cca43163fcd91d68b6a08bacab915cd1 WatchSource:0}: Error finding container 45cb69f87eb92a04b6da8c806c8679d8cca43163fcd91d68b6a08bacab915cd1: Status 404 returned error can't find the container with id 45cb69f87eb92a04b6da8c806c8679d8cca43163fcd91d68b6a08bacab915cd1
Sep 10 02:54:52.335952 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:52.335922    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-fvwhm" event={"ID":"80cafcc1-65ea-4e0b-9394-e5dd56d9ea8b","Type":"ContainerStarted","Data":"45cb69f87eb92a04b6da8c806c8679d8cca43163fcd91d68b6a08bacab915cd1"}
Sep 10 02:54:53.344313 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:53.344279    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-84qc9" event={"ID":"bdd46bc7-4845-46f4-bd55-00131d9b7bd4","Type":"ContainerStarted","Data":"c8ae90a502a999de083fdd138af1de3da17727e9f2cf85cc0586eebd8ab937c3"}
Sep 10 02:54:53.344751 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:53.344420    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-84qc9"
Sep 10 02:54:53.360631 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:53.360558    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-84qc9" podStartSLOduration=2.00350622 podStartE2EDuration="3.360544775s" totalImagesPullingTime="1.357038555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:54:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:54:53.3589412 +0000 UTC m=+483.734027120" watchObservedRunningTime="2026-09-10 02:54:53.360544775 +0000 UTC m=+483.735630696"
Sep 10 02:54:54.351445 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:54.351402    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-fvwhm" event={"ID":"80cafcc1-65ea-4e0b-9394-e5dd56d9ea8b","Type":"ContainerStarted","Data":"9e81745071ac9f871009cb1df0cd467f3ef7f71ed9cb9c33b1e9c3ea6da50adf"}
Sep 10 02:54:54.368329 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:54:54.368282    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-fvwhm" podStartSLOduration=1.698238117 podStartE2EDuration="3.368264159s" totalImagesPullingTime="1.670026042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:54:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:54:54.367067784 +0000 UTC m=+484.742153705" watchObservedRunningTime="2026-09-10 02:54:54.368264159 +0000 UTC m=+484.743350079"
Sep 10 02:55:00.242283 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:00.242246    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:55:00.302336 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:00.302292    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:55:04.264123 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:04.264063    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:55:04.352418 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:04.352392    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-84qc9"
Sep 10 02:55:04.608166 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:04.608093    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-84qc9"]
Sep 10 02:55:04.608295 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:04.608280    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-84qc9" podUID="bdd46bc7-4845-46f4-bd55-00131d9b7bd4" containerName="limitador" containerID="cri-o://c8ae90a502a999de083fdd138af1de3da17727e9f2cf85cc0586eebd8ab937c3" gracePeriod=30
Sep 10 02:55:05.409067 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:05.409034    2558 generic.go:356] "Generic (PLEG): container finished" podID="bdd46bc7-4845-46f4-bd55-00131d9b7bd4" containerID="c8ae90a502a999de083fdd138af1de3da17727e9f2cf85cc0586eebd8ab937c3" exitCode=0
Sep 10 02:55:05.409438 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:05.409087    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-84qc9" event={"ID":"bdd46bc7-4845-46f4-bd55-00131d9b7bd4","Type":"ContainerDied","Data":"c8ae90a502a999de083fdd138af1de3da17727e9f2cf85cc0586eebd8ab937c3"}
Sep 10 02:55:05.547883 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:05.547862    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-84qc9"
Sep 10 02:55:05.675359 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:05.675295    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z99cc\" (UniqueName: \"kubernetes.io/projected/bdd46bc7-4845-46f4-bd55-00131d9b7bd4-kube-api-access-z99cc\") pod \"bdd46bc7-4845-46f4-bd55-00131d9b7bd4\" (UID: \"bdd46bc7-4845-46f4-bd55-00131d9b7bd4\") "
Sep 10 02:55:05.675486 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:05.675379    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bdd46bc7-4845-46f4-bd55-00131d9b7bd4-config-file\") pod \"bdd46bc7-4845-46f4-bd55-00131d9b7bd4\" (UID: \"bdd46bc7-4845-46f4-bd55-00131d9b7bd4\") "
Sep 10 02:55:05.675705 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:05.675676    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdd46bc7-4845-46f4-bd55-00131d9b7bd4-config-file" (OuterVolumeSpecName: "config-file") pod "bdd46bc7-4845-46f4-bd55-00131d9b7bd4" (UID: "bdd46bc7-4845-46f4-bd55-00131d9b7bd4"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 02:55:05.677311 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:05.677287    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd46bc7-4845-46f4-bd55-00131d9b7bd4-kube-api-access-z99cc" (OuterVolumeSpecName: "kube-api-access-z99cc") pod "bdd46bc7-4845-46f4-bd55-00131d9b7bd4" (UID: "bdd46bc7-4845-46f4-bd55-00131d9b7bd4"). InnerVolumeSpecName "kube-api-access-z99cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 02:55:05.776746 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:05.776722    2558 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bdd46bc7-4845-46f4-bd55-00131d9b7bd4-config-file\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:55:05.776746 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:05.776743    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z99cc\" (UniqueName: \"kubernetes.io/projected/bdd46bc7-4845-46f4-bd55-00131d9b7bd4-kube-api-access-z99cc\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:55:05.958333 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:05.958309    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:55:06.056703 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:06.056678    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:55:06.415900 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:06.415832    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-84qc9"
Sep 10 02:55:06.415900 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:06.415871    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-84qc9" event={"ID":"bdd46bc7-4845-46f4-bd55-00131d9b7bd4","Type":"ContainerDied","Data":"47ef86c5d11e2e42ca83d6d96d376160b2384b4d5fad635c4e1399d3389d43fe"}
Sep 10 02:55:06.416364 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:06.415951    2558 scope.go:117] "RemoveContainer" containerID="c8ae90a502a999de083fdd138af1de3da17727e9f2cf85cc0586eebd8ab937c3"
Sep 10 02:55:06.441075 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:06.441048    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-84qc9"]
Sep 10 02:55:06.457315 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:06.457294    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-84qc9"]
Sep 10 02:55:06.575194 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:06.575168    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-xfr2r"]
Sep 10 02:55:06.575462 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:06.575451    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdd46bc7-4845-46f4-bd55-00131d9b7bd4" containerName="limitador"
Sep 10 02:55:06.575506 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:06.575464    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdd46bc7-4845-46f4-bd55-00131d9b7bd4" containerName="limitador"
Sep 10 02:55:06.575543 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:06.575526    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdd46bc7-4845-46f4-bd55-00131d9b7bd4" containerName="limitador"
Sep 10 02:55:06.577834 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:06.577820    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xfr2r"
Sep 10 02:55:06.580191 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:06.580174    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-d8qgq\""
Sep 10 02:55:06.580373 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:06.580359    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 02:55:06.580474 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:06.580456    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 10 02:55:06.582419 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:06.582400    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 02:55:06.591371 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:06.591349    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-xfr2r"]
Sep 10 02:55:06.606008 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:06.605988    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:55:06.683136 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:06.683071    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/65b485c5-f43c-4925-9621-ab193d525ba3-data\") pod \"postgres-69fb4594cf-xfr2r\" (UID: \"65b485c5-f43c-4925-9621-ab193d525ba3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xfr2r"
Sep 10 02:55:06.683136 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:06.683120    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9j99\" (UniqueName: \"kubernetes.io/projected/65b485c5-f43c-4925-9621-ab193d525ba3-kube-api-access-p9j99\") pod \"postgres-69fb4594cf-xfr2r\" (UID: \"65b485c5-f43c-4925-9621-ab193d525ba3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xfr2r"
Sep 10 02:55:06.783838 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:06.783818    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/65b485c5-f43c-4925-9621-ab193d525ba3-data\") pod \"postgres-69fb4594cf-xfr2r\" (UID: \"65b485c5-f43c-4925-9621-ab193d525ba3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xfr2r"
Sep 10 02:55:06.783934 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:06.783852    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9j99\" (UniqueName: \"kubernetes.io/projected/65b485c5-f43c-4925-9621-ab193d525ba3-kube-api-access-p9j99\") pod \"postgres-69fb4594cf-xfr2r\" (UID: \"65b485c5-f43c-4925-9621-ab193d525ba3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xfr2r"
Sep 10 02:55:06.784212 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:06.784194    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/65b485c5-f43c-4925-9621-ab193d525ba3-data\") pod \"postgres-69fb4594cf-xfr2r\" (UID: \"65b485c5-f43c-4925-9621-ab193d525ba3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xfr2r"
Sep 10 02:55:06.793541 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:06.793514    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9j99\" (UniqueName: \"kubernetes.io/projected/65b485c5-f43c-4925-9621-ab193d525ba3-kube-api-access-p9j99\") pod \"postgres-69fb4594cf-xfr2r\" (UID: \"65b485c5-f43c-4925-9621-ab193d525ba3\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xfr2r"
Sep 10 02:55:06.887156 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:06.887139    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xfr2r"
Sep 10 02:55:07.153571 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:07.153545    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-xfr2r"]
Sep 10 02:55:07.155415 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:55:07.155378    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b485c5_f43c_4925_9621_ab193d525ba3.slice/crio-8e5ea91194d47027b230bb942daf0918d0f7e1f4aa3d949a2687704614819d44 WatchSource:0}: Error finding container 8e5ea91194d47027b230bb942daf0918d0f7e1f4aa3d949a2687704614819d44: Status 404 returned error can't find the container with id 8e5ea91194d47027b230bb942daf0918d0f7e1f4aa3d949a2687704614819d44
Sep 10 02:55:07.427913 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:07.427885    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xfr2r" event={"ID":"65b485c5-f43c-4925-9621-ab193d525ba3","Type":"ContainerStarted","Data":"8e5ea91194d47027b230bb942daf0918d0f7e1f4aa3d949a2687704614819d44"}
Sep 10 02:55:08.069562 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:08.069527    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdd46bc7-4845-46f4-bd55-00131d9b7bd4" path="/var/lib/kubelet/pods/bdd46bc7-4845-46f4-bd55-00131d9b7bd4/volumes"
Sep 10 02:55:09.441830 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:09.441746    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xfr2r" event={"ID":"65b485c5-f43c-4925-9621-ab193d525ba3","Type":"ContainerStarted","Data":"f5bc006650eb02888704961b4fd58066d1c4e40ec0237e1da536890025e02b8e"}
Sep 10 02:55:09.442197 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:09.441906    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xfr2r"
Sep 10 02:55:09.458017 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:09.457971    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xfr2r" podStartSLOduration=1.422060924 podStartE2EDuration="3.457957408s" totalImagesPullingTime="2.035896484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:55:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:55:09.457265518 +0000 UTC m=+499.832351439" watchObservedRunningTime="2026-09-10 02:55:09.457957408 +0000 UTC m=+499.833043374"
Sep 10 02:55:15.476349 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:15.476319    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xfr2r"
Sep 10 02:55:15.791117 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:15.791001    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:55:15.998564 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:15.998528    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:55:17.721892 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:17.721844    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-fvwhm"]
Sep 10 02:55:17.722401 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:17.722070    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-fvwhm" podUID="80cafcc1-65ea-4e0b-9394-e5dd56d9ea8b" containerName="authorino" containerID="cri-o://9e81745071ac9f871009cb1df0cd467f3ef7f71ed9cb9c33b1e9c3ea6da50adf" gracePeriod=30
Sep 10 02:55:17.976545 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:17.976487    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-fvwhm"
Sep 10 02:55:18.075089 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:18.075060    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjwh6\" (UniqueName: \"kubernetes.io/projected/80cafcc1-65ea-4e0b-9394-e5dd56d9ea8b-kube-api-access-wjwh6\") pod \"80cafcc1-65ea-4e0b-9394-e5dd56d9ea8b\" (UID: \"80cafcc1-65ea-4e0b-9394-e5dd56d9ea8b\") "
Sep 10 02:55:18.076893 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:18.076872    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80cafcc1-65ea-4e0b-9394-e5dd56d9ea8b-kube-api-access-wjwh6" (OuterVolumeSpecName: "kube-api-access-wjwh6") pod "80cafcc1-65ea-4e0b-9394-e5dd56d9ea8b" (UID: "80cafcc1-65ea-4e0b-9394-e5dd56d9ea8b"). InnerVolumeSpecName "kube-api-access-wjwh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 02:55:18.175895 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:18.175869    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjwh6\" (UniqueName: \"kubernetes.io/projected/80cafcc1-65ea-4e0b-9394-e5dd56d9ea8b-kube-api-access-wjwh6\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:55:18.489795 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:18.489760    2558 generic.go:356] "Generic (PLEG): container finished" podID="80cafcc1-65ea-4e0b-9394-e5dd56d9ea8b" containerID="9e81745071ac9f871009cb1df0cd467f3ef7f71ed9cb9c33b1e9c3ea6da50adf" exitCode=0
Sep 10 02:55:18.489951 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:18.489804    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-fvwhm"
Sep 10 02:55:18.489951 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:18.489852    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-fvwhm" event={"ID":"80cafcc1-65ea-4e0b-9394-e5dd56d9ea8b","Type":"ContainerDied","Data":"9e81745071ac9f871009cb1df0cd467f3ef7f71ed9cb9c33b1e9c3ea6da50adf"}
Sep 10 02:55:18.489951 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:18.489902    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-fvwhm" event={"ID":"80cafcc1-65ea-4e0b-9394-e5dd56d9ea8b","Type":"ContainerDied","Data":"45cb69f87eb92a04b6da8c806c8679d8cca43163fcd91d68b6a08bacab915cd1"}
Sep 10 02:55:18.489951 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:18.489925    2558 scope.go:117] "RemoveContainer" containerID="9e81745071ac9f871009cb1df0cd467f3ef7f71ed9cb9c33b1e9c3ea6da50adf"
Sep 10 02:55:18.498735 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:18.498713    2558 scope.go:117] "RemoveContainer" containerID="9e81745071ac9f871009cb1df0cd467f3ef7f71ed9cb9c33b1e9c3ea6da50adf"
Sep 10 02:55:18.499001 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:55:18.498979    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e81745071ac9f871009cb1df0cd467f3ef7f71ed9cb9c33b1e9c3ea6da50adf\": container with ID starting with 9e81745071ac9f871009cb1df0cd467f3ef7f71ed9cb9c33b1e9c3ea6da50adf not found: ID does not exist" containerID="9e81745071ac9f871009cb1df0cd467f3ef7f71ed9cb9c33b1e9c3ea6da50adf"
Sep 10 02:55:18.499081 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:18.499014    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e81745071ac9f871009cb1df0cd467f3ef7f71ed9cb9c33b1e9c3ea6da50adf"} err="failed to get container status \"9e81745071ac9f871009cb1df0cd467f3ef7f71ed9cb9c33b1e9c3ea6da50adf\": rpc error: code = NotFound desc = could not find container \"9e81745071ac9f871009cb1df0cd467f3ef7f71ed9cb9c33b1e9c3ea6da50adf\": container with ID starting with 9e81745071ac9f871009cb1df0cd467f3ef7f71ed9cb9c33b1e9c3ea6da50adf not found: ID does not exist"
Sep 10 02:55:18.511881 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:18.511859    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-fvwhm"]
Sep 10 02:55:18.517399 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:18.517378    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-fvwhm"]
Sep 10 02:55:20.068979 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:20.068941    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80cafcc1-65ea-4e0b-9394-e5dd56d9ea8b" path="/var/lib/kubelet/pods/80cafcc1-65ea-4e0b-9394-e5dd56d9ea8b/volumes"
Sep 10 02:55:22.746460 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:22.746425    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:55:22.846626 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:22.846587    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:55:22.898919 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:22.898890    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:55:23.063200 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:23.063139    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:55:23.117935 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:23.117912    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:55:26.365906 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:26.365862    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:55:26.466556 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:26.466523    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:55:27.393144 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:27.393094    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:55:57.966127 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:57.966065    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:55:58.066185 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:58.066157    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:55:59.378667 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:59.378630    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:55:59.479716 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:59.479688    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:55:59.578808 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:59.578781    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:55:59.863351 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:59.863317    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:55:59.962997 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:59.962965    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:55:59.983276 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:59.983250    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-twvbt"]
Sep 10 02:55:59.983562 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:59.983550    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80cafcc1-65ea-4e0b-9394-e5dd56d9ea8b" containerName="authorino"
Sep 10 02:55:59.983562 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:59.983563    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="80cafcc1-65ea-4e0b-9394-e5dd56d9ea8b" containerName="authorino"
Sep 10 02:55:59.983649 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:59.983619    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="80cafcc1-65ea-4e0b-9394-e5dd56d9ea8b" containerName="authorino"
Sep 10 02:55:59.987836 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:59.987816    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-twvbt"
Sep 10 02:55:59.990128 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:59.990083    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 10 02:55:59.996352 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:55:59.996332    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-twvbt"]
Sep 10 02:56:00.050086 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:00.050062    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-tv2l8"]
Sep 10 02:56:00.053080 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:00.053065    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tv2l8"
Sep 10 02:56:00.054878 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:00.054860    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-fcm5z\""
Sep 10 02:56:00.066619 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:00.066600    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh9sv\" (UniqueName: \"kubernetes.io/projected/7d5508ae-a8b6-4aa6-a63f-112e565afc69-kube-api-access-mh9sv\") pod \"payload-processing-5b54b7fbb7-twvbt\" (UID: \"7d5508ae-a8b6-4aa6-a63f-112e565afc69\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-twvbt"
Sep 10 02:56:00.066737 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:00.066637    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e7d2343-e490-443b-9e0f-21c7b051a445-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-tv2l8\" (UID: \"8e7d2343-e490-443b-9e0f-21c7b051a445\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tv2l8"
Sep 10 02:56:00.066737 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:00.066682    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6gz2\" (UniqueName: \"kubernetes.io/projected/8e7d2343-e490-443b-9e0f-21c7b051a445-kube-api-access-f6gz2\") pod \"payload-pre-processing-59fbc47bf8-tv2l8\" (UID: \"8e7d2343-e490-443b-9e0f-21c7b051a445\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tv2l8"
Sep 10 02:56:00.066835 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:00.066734    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5508ae-a8b6-4aa6-a63f-112e565afc69-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-twvbt\" (UID: \"7d5508ae-a8b6-4aa6-a63f-112e565afc69\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-twvbt"
Sep 10 02:56:00.069297 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:00.069278    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-tv2l8"]
Sep 10 02:56:00.167112 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:00.167023    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh9sv\" (UniqueName: \"kubernetes.io/projected/7d5508ae-a8b6-4aa6-a63f-112e565afc69-kube-api-access-mh9sv\") pod \"payload-processing-5b54b7fbb7-twvbt\" (UID: \"7d5508ae-a8b6-4aa6-a63f-112e565afc69\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-twvbt"
Sep 10 02:56:00.167112 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:00.167069    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e7d2343-e490-443b-9e0f-21c7b051a445-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-tv2l8\" (UID: \"8e7d2343-e490-443b-9e0f-21c7b051a445\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tv2l8"
Sep 10 02:56:00.167287 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:00.167190    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6gz2\" (UniqueName: \"kubernetes.io/projected/8e7d2343-e490-443b-9e0f-21c7b051a445-kube-api-access-f6gz2\") pod \"payload-pre-processing-59fbc47bf8-tv2l8\" (UID: \"8e7d2343-e490-443b-9e0f-21c7b051a445\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tv2l8"
Sep 10 02:56:00.167287 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:00.167263    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5508ae-a8b6-4aa6-a63f-112e565afc69-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-twvbt\" (UID: \"7d5508ae-a8b6-4aa6-a63f-112e565afc69\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-twvbt"
Sep 10 02:56:00.167806 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:00.167784    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e7d2343-e490-443b-9e0f-21c7b051a445-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-tv2l8\" (UID: \"8e7d2343-e490-443b-9e0f-21c7b051a445\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tv2l8"
Sep 10 02:56:00.167916 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:00.167900    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5508ae-a8b6-4aa6-a63f-112e565afc69-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-twvbt\" (UID: \"7d5508ae-a8b6-4aa6-a63f-112e565afc69\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-twvbt"
Sep 10 02:56:00.174551 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:00.174528    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh9sv\" (UniqueName: \"kubernetes.io/projected/7d5508ae-a8b6-4aa6-a63f-112e565afc69-kube-api-access-mh9sv\") pod \"payload-processing-5b54b7fbb7-twvbt\" (UID: \"7d5508ae-a8b6-4aa6-a63f-112e565afc69\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-twvbt"
Sep 10 02:56:00.174666 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:00.174532    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6gz2\" (UniqueName: \"kubernetes.io/projected/8e7d2343-e490-443b-9e0f-21c7b051a445-kube-api-access-f6gz2\") pod \"payload-pre-processing-59fbc47bf8-tv2l8\" (UID: \"8e7d2343-e490-443b-9e0f-21c7b051a445\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tv2l8"
Sep 10 02:56:00.298690 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:00.298666    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-twvbt"
Sep 10 02:56:00.363410 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:00.363384    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tv2l8"
Sep 10 02:56:00.569630 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:00.569588    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-twvbt"]
Sep 10 02:56:00.575890 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:56:00.575864    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5508ae_a8b6_4aa6_a63f_112e565afc69.slice/crio-d97d19b6fb340467d8e69fab36c10c5e60b8466f40e6c83d29e00cf41723ee86 WatchSource:0}: Error finding container d97d19b6fb340467d8e69fab36c10c5e60b8466f40e6c83d29e00cf41723ee86: Status 404 returned error can't find the container with id d97d19b6fb340467d8e69fab36c10c5e60b8466f40e6c83d29e00cf41723ee86
Sep 10 02:56:00.637559 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:00.637539    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-tv2l8"]
Sep 10 02:56:00.639259 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:56:00.639236    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e7d2343_e490_443b_9e0f_21c7b051a445.slice/crio-6777d76e8287163b6cbc94d0c4b614ffe1b9455edf640ec5d47d25586337e702 WatchSource:0}: Error finding container 6777d76e8287163b6cbc94d0c4b614ffe1b9455edf640ec5d47d25586337e702: Status 404 returned error can't find the container with id 6777d76e8287163b6cbc94d0c4b614ffe1b9455edf640ec5d47d25586337e702
Sep 10 02:56:00.708364 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:00.708333    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-twvbt" event={"ID":"7d5508ae-a8b6-4aa6-a63f-112e565afc69","Type":"ContainerStarted","Data":"d97d19b6fb340467d8e69fab36c10c5e60b8466f40e6c83d29e00cf41723ee86"}
Sep 10 02:56:00.709482 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:00.709464    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tv2l8" event={"ID":"8e7d2343-e490-443b-9e0f-21c7b051a445","Type":"ContainerStarted","Data":"6777d76e8287163b6cbc94d0c4b614ffe1b9455edf640ec5d47d25586337e702"}
Sep 10 02:56:03.728300 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:03.728266    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tv2l8" event={"ID":"8e7d2343-e490-443b-9e0f-21c7b051a445","Type":"ContainerStarted","Data":"69b1b334ce72b5b5beb8f42e3fcd9bea85b1c984757b458b262e6236d86b5821"}
Sep 10 02:56:03.728691 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:03.728364    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tv2l8"
Sep 10 02:56:03.729818 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:03.729795    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-twvbt" event={"ID":"7d5508ae-a8b6-4aa6-a63f-112e565afc69","Type":"ContainerStarted","Data":"9ef44e8be2711a554bc7906cc2c196bb7b511c7fea2c762adf1ef8ec37f79f6f"}
Sep 10 02:56:03.729946 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:03.729931    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-twvbt"
Sep 10 02:56:03.745382 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:03.745341    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tv2l8" podStartSLOduration=1.242876674 podStartE2EDuration="3.745328706s" totalImagesPullingTime="2.502452032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:56:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:56:03.743902817 +0000 UTC m=+554.118988738" watchObservedRunningTime="2026-09-10 02:56:03.745328706 +0000 UTC m=+554.120414627"
Sep 10 02:56:03.759144 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:03.759096    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-twvbt" podStartSLOduration=2.199079144 podStartE2EDuration="4.759083919s" totalImagesPullingTime="2.560004775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:55:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:56:03.758460194 +0000 UTC m=+554.133546114" watchObservedRunningTime="2026-09-10 02:56:03.759083919 +0000 UTC m=+554.134169839"
Sep 10 02:56:09.697761 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:09.697642    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:56:09.794553 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:09.794519    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:56:14.736990 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:14.736963    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tv2l8"
Sep 10 02:56:14.737489 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:14.737053    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-twvbt"
Sep 10 02:56:30.605622 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:30.605580    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:56:36.145860 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:36.145831    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:56:47.303199 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:47.303160    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8"]
Sep 10 02:56:47.306907 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:47.306885    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8"
Sep 10 02:56:47.309680 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:47.309659    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 02:56:47.309780 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:47.309658    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dlt7k\""
Sep 10 02:56:47.309780 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:47.309659    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 10 02:56:47.309780 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:47.309675    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 02:56:47.319985 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:47.319964    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8"]
Sep 10 02:56:47.346404 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:47.346380    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:56:47.401910 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:47.401880    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/413b0e1a-6827-4487-9a96-cdc23dbbfb24-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8\" (UID: \"413b0e1a-6827-4487-9a96-cdc23dbbfb24\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8"
Sep 10 02:56:47.402028 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:47.401930    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/413b0e1a-6827-4487-9a96-cdc23dbbfb24-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8\" (UID: \"413b0e1a-6827-4487-9a96-cdc23dbbfb24\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8"
Sep 10 02:56:47.402028 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:47.401964    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/413b0e1a-6827-4487-9a96-cdc23dbbfb24-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8\" (UID: \"413b0e1a-6827-4487-9a96-cdc23dbbfb24\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8"
Sep 10 02:56:47.402028 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:47.402005    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/413b0e1a-6827-4487-9a96-cdc23dbbfb24-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8\" (UID: \"413b0e1a-6827-4487-9a96-cdc23dbbfb24\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8"
Sep 10 02:56:47.402213 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:47.402034    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/413b0e1a-6827-4487-9a96-cdc23dbbfb24-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8\" (UID: \"413b0e1a-6827-4487-9a96-cdc23dbbfb24\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8"
Sep 10 02:56:47.402213 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:47.402057    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk2bn\" (UniqueName: \"kubernetes.io/projected/413b0e1a-6827-4487-9a96-cdc23dbbfb24-kube-api-access-gk2bn\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8\" (UID: \"413b0e1a-6827-4487-9a96-cdc23dbbfb24\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8"
Sep 10 02:56:47.503450 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:47.503423    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/413b0e1a-6827-4487-9a96-cdc23dbbfb24-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8\" (UID: \"413b0e1a-6827-4487-9a96-cdc23dbbfb24\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8"
Sep 10 02:56:47.503563 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:47.503460    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/413b0e1a-6827-4487-9a96-cdc23dbbfb24-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8\" (UID: \"413b0e1a-6827-4487-9a96-cdc23dbbfb24\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8"
Sep 10 02:56:47.503563 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:47.503486    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk2bn\" (UniqueName: \"kubernetes.io/projected/413b0e1a-6827-4487-9a96-cdc23dbbfb24-kube-api-access-gk2bn\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8\" (UID: \"413b0e1a-6827-4487-9a96-cdc23dbbfb24\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8"
Sep 10 02:56:47.503563 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:47.503545    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/413b0e1a-6827-4487-9a96-cdc23dbbfb24-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8\" (UID: \"413b0e1a-6827-4487-9a96-cdc23dbbfb24\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8"
Sep 10 02:56:47.503728 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:47.503592    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/413b0e1a-6827-4487-9a96-cdc23dbbfb24-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8\" (UID: \"413b0e1a-6827-4487-9a96-cdc23dbbfb24\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8"
Sep 10 02:56:47.503728 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:47.503623    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/413b0e1a-6827-4487-9a96-cdc23dbbfb24-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8\" (UID: \"413b0e1a-6827-4487-9a96-cdc23dbbfb24\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8"
Sep 10 02:56:47.503815 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:47.503787    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/413b0e1a-6827-4487-9a96-cdc23dbbfb24-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8\" (UID: \"413b0e1a-6827-4487-9a96-cdc23dbbfb24\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8"
Sep 10 02:56:47.503922 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:47.503901    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/413b0e1a-6827-4487-9a96-cdc23dbbfb24-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8\" (UID: \"413b0e1a-6827-4487-9a96-cdc23dbbfb24\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8"
Sep 10 02:56:47.503983 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:47.503947    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/413b0e1a-6827-4487-9a96-cdc23dbbfb24-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8\" (UID: \"413b0e1a-6827-4487-9a96-cdc23dbbfb24\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8"
Sep 10 02:56:47.505703 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:47.505678    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/413b0e1a-6827-4487-9a96-cdc23dbbfb24-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8\" (UID: \"413b0e1a-6827-4487-9a96-cdc23dbbfb24\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8"
Sep 10 02:56:47.505983 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:47.505961    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/413b0e1a-6827-4487-9a96-cdc23dbbfb24-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8\" (UID: \"413b0e1a-6827-4487-9a96-cdc23dbbfb24\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8"
Sep 10 02:56:47.511751 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:47.511730    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk2bn\" (UniqueName: \"kubernetes.io/projected/413b0e1a-6827-4487-9a96-cdc23dbbfb24-kube-api-access-gk2bn\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8\" (UID: \"413b0e1a-6827-4487-9a96-cdc23dbbfb24\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8"
Sep 10 02:56:47.618584 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:47.618520    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8"
Sep 10 02:56:47.888605 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:47.888575    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8"]
Sep 10 02:56:47.893330 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:56:47.893303    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod413b0e1a_6827_4487_9a96_cdc23dbbfb24.slice/crio-1215d0135da0e27db88466081d34b69d3cf65fa37fcb0428bbabe81fffeba3d8 WatchSource:0}: Error finding container 1215d0135da0e27db88466081d34b69d3cf65fa37fcb0428bbabe81fffeba3d8: Status 404 returned error can't find the container with id 1215d0135da0e27db88466081d34b69d3cf65fa37fcb0428bbabe81fffeba3d8
Sep 10 02:56:47.956854 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:47.956822    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8" event={"ID":"413b0e1a-6827-4487-9a96-cdc23dbbfb24","Type":"ContainerStarted","Data":"1215d0135da0e27db88466081d34b69d3cf65fa37fcb0428bbabe81fffeba3d8"}
Sep 10 02:56:48.966421 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:48.966392    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8" event={"ID":"413b0e1a-6827-4487-9a96-cdc23dbbfb24","Type":"ContainerStarted","Data":"62aa33f1f0e7931d02835beffd459ef8e83b958f05c49807002bd8917059deb4"}
Sep 10 02:56:48.982705 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:48.982653    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8" podStartSLOduration=1.008794435 podStartE2EDuration="1.982634458s" totalImagesPullingTime="973.840023ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:56:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:56:48.982068915 +0000 UTC m=+599.357154847" watchObservedRunningTime="2026-09-10 02:56:48.982634458 +0000 UTC m=+599.357720379"
Sep 10 02:56:49.294202 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:49.294171    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:56:52.213184 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:52.213150    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-rptch"]
Sep 10 02:56:52.216369 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:52.216348    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rptch"
Sep 10 02:56:52.218418 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:52.218397    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 10 02:56:52.224184 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:52.224013    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-rptch"]
Sep 10 02:56:52.244544 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:52.244523    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:56:52.339578 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:52.339551    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/566cc6ae-630a-4ae3-abf9-8fd43be79e8e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rptch\" (UID: \"566cc6ae-630a-4ae3-abf9-8fd43be79e8e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rptch"
Sep 10 02:56:52.339700 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:52.339597    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74w99\" (UniqueName: \"kubernetes.io/projected/566cc6ae-630a-4ae3-abf9-8fd43be79e8e-kube-api-access-74w99\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rptch\" (UID: \"566cc6ae-630a-4ae3-abf9-8fd43be79e8e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rptch"
Sep 10 02:56:52.339700 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:52.339617    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/566cc6ae-630a-4ae3-abf9-8fd43be79e8e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rptch\" (UID: \"566cc6ae-630a-4ae3-abf9-8fd43be79e8e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rptch"
Sep 10 02:56:52.339810 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:52.339721    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/566cc6ae-630a-4ae3-abf9-8fd43be79e8e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rptch\" (UID: \"566cc6ae-630a-4ae3-abf9-8fd43be79e8e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rptch"
Sep 10 02:56:52.339810 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:52.339769    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/566cc6ae-630a-4ae3-abf9-8fd43be79e8e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rptch\" (UID: \"566cc6ae-630a-4ae3-abf9-8fd43be79e8e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rptch"
Sep 10 02:56:52.339891 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:52.339832    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/566cc6ae-630a-4ae3-abf9-8fd43be79e8e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rptch\" (UID: \"566cc6ae-630a-4ae3-abf9-8fd43be79e8e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rptch"
Sep 10 02:56:52.446878 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:52.440924    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/566cc6ae-630a-4ae3-abf9-8fd43be79e8e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rptch\" (UID: \"566cc6ae-630a-4ae3-abf9-8fd43be79e8e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rptch"
Sep 10 02:56:52.446878 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:52.441011    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74w99\" (UniqueName: \"kubernetes.io/projected/566cc6ae-630a-4ae3-abf9-8fd43be79e8e-kube-api-access-74w99\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rptch\" (UID: \"566cc6ae-630a-4ae3-abf9-8fd43be79e8e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rptch"
Sep 10 02:56:52.446878 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:52.441049    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/566cc6ae-630a-4ae3-abf9-8fd43be79e8e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rptch\" (UID: \"566cc6ae-630a-4ae3-abf9-8fd43be79e8e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rptch"
Sep 10 02:56:52.446878 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:52.441130    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/566cc6ae-630a-4ae3-abf9-8fd43be79e8e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rptch\" (UID: \"566cc6ae-630a-4ae3-abf9-8fd43be79e8e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rptch"
Sep 10 02:56:52.446878 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:52.441174    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/566cc6ae-630a-4ae3-abf9-8fd43be79e8e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rptch\" (UID: \"566cc6ae-630a-4ae3-abf9-8fd43be79e8e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rptch"
Sep 10 02:56:52.446878 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:52.441215    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/566cc6ae-630a-4ae3-abf9-8fd43be79e8e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rptch\" (UID: \"566cc6ae-630a-4ae3-abf9-8fd43be79e8e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rptch"
Sep 10 02:56:52.446878 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:52.443939    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/566cc6ae-630a-4ae3-abf9-8fd43be79e8e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rptch\" (UID: \"566cc6ae-630a-4ae3-abf9-8fd43be79e8e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rptch"
Sep 10 02:56:52.446878 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:52.444229    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/566cc6ae-630a-4ae3-abf9-8fd43be79e8e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rptch\" (UID: \"566cc6ae-630a-4ae3-abf9-8fd43be79e8e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rptch"
Sep 10 02:56:52.447339 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:52.447043    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/566cc6ae-630a-4ae3-abf9-8fd43be79e8e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rptch\" (UID: \"566cc6ae-630a-4ae3-abf9-8fd43be79e8e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rptch"
Sep 10 02:56:52.447339 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:52.447126    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/566cc6ae-630a-4ae3-abf9-8fd43be79e8e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rptch\" (UID: \"566cc6ae-630a-4ae3-abf9-8fd43be79e8e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rptch"
Sep 10 02:56:52.447339 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:52.447143    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/566cc6ae-630a-4ae3-abf9-8fd43be79e8e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rptch\" (UID: \"566cc6ae-630a-4ae3-abf9-8fd43be79e8e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rptch"
Sep 10 02:56:52.451696 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:52.451673    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74w99\" (UniqueName: \"kubernetes.io/projected/566cc6ae-630a-4ae3-abf9-8fd43be79e8e-kube-api-access-74w99\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rptch\" (UID: \"566cc6ae-630a-4ae3-abf9-8fd43be79e8e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rptch"
Sep 10 02:56:52.528521 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:52.528465    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rptch"
Sep 10 02:56:52.803056 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:52.803022    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-rptch"]
Sep 10 02:56:52.804694 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:56:52.804665    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod566cc6ae_630a_4ae3_abf9_8fd43be79e8e.slice/crio-960e747cd8f0c91d88f8a99c0103946f66a164c183c55f54296f1e9024c8c526 WatchSource:0}: Error finding container 960e747cd8f0c91d88f8a99c0103946f66a164c183c55f54296f1e9024c8c526: Status 404 returned error can't find the container with id 960e747cd8f0c91d88f8a99c0103946f66a164c183c55f54296f1e9024c8c526
Sep 10 02:56:52.806066 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:52.806047    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 02:56:52.987320 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:52.987295    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rptch" event={"ID":"566cc6ae-630a-4ae3-abf9-8fd43be79e8e","Type":"ContainerStarted","Data":"960e747cd8f0c91d88f8a99c0103946f66a164c183c55f54296f1e9024c8c526"}
Sep 10 02:56:53.994369 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:53.994333    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rptch" event={"ID":"566cc6ae-630a-4ae3-abf9-8fd43be79e8e","Type":"ContainerStarted","Data":"f14a3f4af94f18c92fa7a63e34db928ec575127503aff84c6b316074d8533d52"}
Sep 10 02:56:54.014136 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:54.014073    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rptch" podStartSLOduration=1.853599747 podStartE2EDuration="2.014058965s" totalImagesPullingTime="160.459218ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:56:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:56:54.011776091 +0000 UTC m=+604.386862012" watchObservedRunningTime="2026-09-10 02:56:54.014058965 +0000 UTC m=+604.389144886"
Sep 10 02:56:57.619556 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:57.619522    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8"
Sep 10 02:56:57.620065 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:57.619595    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8"
Sep 10 02:56:57.632096 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:57.632071    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8"
Sep 10 02:56:58.024695 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:56:58.024668    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8"
Sep 10 02:57:02.529396 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:57:02.529353    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rptch"
Sep 10 02:57:02.529825 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:57:02.529421    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rptch"
Sep 10 02:57:02.542196 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:57:02.542159    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rptch"
Sep 10 02:57:02.806678 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:57:02.806569    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:57:03.052719 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:57:03.052691    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rptch"
Sep 10 02:57:09.844945 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:57:09.844907    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:57:43.072451 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:57:43.072403    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:57:48.837962 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:57:48.837922    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:58:33.358878 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:33.358841    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5c96959ff4-44rtg"]
Sep 10 02:58:33.362793 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:33.362767    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c96959ff4-44rtg"
Sep 10 02:58:33.365630 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:33.365605    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5stxp\""
Sep 10 02:58:33.365630 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:33.365626    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 02:58:33.369833 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:33.369809    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c96959ff4-44rtg"]
Sep 10 02:58:33.449029 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:33.448999    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc976\" (UniqueName: \"kubernetes.io/projected/75ef62e9-cb32-463f-af86-685cc0890b21-kube-api-access-wc976\") pod \"authorino-5c96959ff4-44rtg\" (UID: \"75ef62e9-cb32-463f-af86-685cc0890b21\") " pod="kuadrant-system/authorino-5c96959ff4-44rtg"
Sep 10 02:58:33.449149 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:33.449052    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/75ef62e9-cb32-463f-af86-685cc0890b21-tls-cert\") pod \"authorino-5c96959ff4-44rtg\" (UID: \"75ef62e9-cb32-463f-af86-685cc0890b21\") " pod="kuadrant-system/authorino-5c96959ff4-44rtg"
Sep 10 02:58:33.550201 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:33.550177    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc976\" (UniqueName: \"kubernetes.io/projected/75ef62e9-cb32-463f-af86-685cc0890b21-kube-api-access-wc976\") pod \"authorino-5c96959ff4-44rtg\" (UID: \"75ef62e9-cb32-463f-af86-685cc0890b21\") " pod="kuadrant-system/authorino-5c96959ff4-44rtg"
Sep 10 02:58:33.550305 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:33.550214    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/75ef62e9-cb32-463f-af86-685cc0890b21-tls-cert\") pod \"authorino-5c96959ff4-44rtg\" (UID: \"75ef62e9-cb32-463f-af86-685cc0890b21\") " pod="kuadrant-system/authorino-5c96959ff4-44rtg"
Sep 10 02:58:33.552533 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:33.552513    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/75ef62e9-cb32-463f-af86-685cc0890b21-tls-cert\") pod \"authorino-5c96959ff4-44rtg\" (UID: \"75ef62e9-cb32-463f-af86-685cc0890b21\") " pod="kuadrant-system/authorino-5c96959ff4-44rtg"
Sep 10 02:58:33.559700 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:33.559677    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc976\" (UniqueName: \"kubernetes.io/projected/75ef62e9-cb32-463f-af86-685cc0890b21-kube-api-access-wc976\") pod \"authorino-5c96959ff4-44rtg\" (UID: \"75ef62e9-cb32-463f-af86-685cc0890b21\") " pod="kuadrant-system/authorino-5c96959ff4-44rtg"
Sep 10 02:58:33.673802 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:33.673750    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c96959ff4-44rtg"
Sep 10 02:58:33.945052 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:33.945023    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c96959ff4-44rtg"]
Sep 10 02:58:33.947265 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:58:33.947234    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75ef62e9_cb32_463f_af86_685cc0890b21.slice/crio-4f6b9fcf9f64060505d4416e4671486a703ca13c5c0bcddc455358921c2be848 WatchSource:0}: Error finding container 4f6b9fcf9f64060505d4416e4671486a703ca13c5c0bcddc455358921c2be848: Status 404 returned error can't find the container with id 4f6b9fcf9f64060505d4416e4671486a703ca13c5c0bcddc455358921c2be848
Sep 10 02:58:34.497832 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:34.497798    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c96959ff4-44rtg" event={"ID":"75ef62e9-cb32-463f-af86-685cc0890b21","Type":"ContainerStarted","Data":"4f6b9fcf9f64060505d4416e4671486a703ca13c5c0bcddc455358921c2be848"}
Sep 10 02:58:35.504691 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:35.504648    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c96959ff4-44rtg" event={"ID":"75ef62e9-cb32-463f-af86-685cc0890b21","Type":"ContainerStarted","Data":"0c5c3a44b2236c959eed020be57ca721be3beb60fcc7a5f93ad6a07adfaffb6e"}
Sep 10 02:58:35.520239 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:35.520187    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5c96959ff4-44rtg" podStartSLOduration=1.5228463140000001 podStartE2EDuration="2.520171736s" totalImagesPullingTime="997.325422ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:58:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:58:35.519012092 +0000 UTC m=+705.894098015" watchObservedRunningTime="2026-09-10 02:58:35.520171736 +0000 UTC m=+705.895257656"
Sep 10 02:58:36.127831 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:36.127784    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:58:36.226356 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:36.226326    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:58:37.318935 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:37.318898    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:58:37.417411 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:37.417366    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:58:53.424175 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:53.424089    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:58:53.523347 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:53.523317    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:58:53.622526 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:53.622498    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:58:53.788450 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:53.788411    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:58:53.866170 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:53.866142    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:58:53.947623 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:53.947572    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:58:54.206782 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:54.206747    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:58:54.369649 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:54.369618    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:58:54.664671 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:54.664584    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:58:55.004057 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:55.004022    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:58:55.125538 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:55.125500    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:58:55.505574 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:55.505537    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:58:56.067667 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:56.067639    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:58:56.955744 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:56.955706    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:58:57.064687 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:57.064637    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:58:57.151960 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:57.151916    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:58:58.151652 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:58.151626    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:58:58.251941 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:58.251914    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:58:58.698949 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:58.698916    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:58:58.798038 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:58.798002    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:58:58.843712 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:58.843682    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-097e16-65ddd76cdc-nrwtx"]
Sep 10 02:58:58.847021 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:58.847005    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-097e16-65ddd76cdc-nrwtx"
Sep 10 02:58:58.849071 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:58.849052    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-097e16\""
Sep 10 02:58:58.849214 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:58.849183    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-097e16-dockercfg-5knx4\""
Sep 10 02:58:58.868297 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:58.868272    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-097e16-65ddd76cdc-nrwtx"]
Sep 10 02:58:58.873068 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:58.873042    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:58:58.880284 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:58.880265    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-097e16-546949c98-csmxk"]
Sep 10 02:58:58.883510 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:58.883495    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-097e16-546949c98-csmxk"
Sep 10 02:58:58.893919 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:58.893899    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-097e16-546949c98-csmxk"]
Sep 10 02:58:58.947363 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:58.947340    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1792e8c-6841-4193-9ca0-53ecbe580290-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-097e16-65ddd76cdc-nrwtx\" (UID: \"a1792e8c-6841-4193-9ca0-53ecbe580290\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-097e16-65ddd76cdc-nrwtx"
Sep 10 02:58:58.947457 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:58.947400    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9pn5\" (UniqueName: \"kubernetes.io/projected/a1792e8c-6841-4193-9ca0-53ecbe580290-kube-api-access-k9pn5\") pod \"payload-processing-e2e-worker-w0-097e16-65ddd76cdc-nrwtx\" (UID: \"a1792e8c-6841-4193-9ca0-53ecbe580290\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-097e16-65ddd76cdc-nrwtx"
Sep 10 02:58:59.048375 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:59.048319    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1792e8c-6841-4193-9ca0-53ecbe580290-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-097e16-65ddd76cdc-nrwtx\" (UID: \"a1792e8c-6841-4193-9ca0-53ecbe580290\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-097e16-65ddd76cdc-nrwtx"
Sep 10 02:58:59.048375 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:59.048365    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47788\" (UniqueName: \"kubernetes.io/projected/a4255849-3d3c-4e26-b766-6d1474682843-kube-api-access-47788\") pod \"payload-pre-processing-e2e-worker-w0-097e16-546949c98-csmxk\" (UID: \"a4255849-3d3c-4e26-b766-6d1474682843\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-097e16-546949c98-csmxk"
Sep 10 02:58:59.048522 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:59.048390    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4255849-3d3c-4e26-b766-6d1474682843-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-097e16-546949c98-csmxk\" (UID: \"a4255849-3d3c-4e26-b766-6d1474682843\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-097e16-546949c98-csmxk"
Sep 10 02:58:59.048522 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:59.048415    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9pn5\" (UniqueName: \"kubernetes.io/projected/a1792e8c-6841-4193-9ca0-53ecbe580290-kube-api-access-k9pn5\") pod \"payload-processing-e2e-worker-w0-097e16-65ddd76cdc-nrwtx\" (UID: \"a1792e8c-6841-4193-9ca0-53ecbe580290\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-097e16-65ddd76cdc-nrwtx"
Sep 10 02:58:59.049038 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:59.049018    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1792e8c-6841-4193-9ca0-53ecbe580290-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-097e16-65ddd76cdc-nrwtx\" (UID: \"a1792e8c-6841-4193-9ca0-53ecbe580290\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-097e16-65ddd76cdc-nrwtx"
Sep 10 02:58:59.056764 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:59.056738    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9pn5\" (UniqueName: \"kubernetes.io/projected/a1792e8c-6841-4193-9ca0-53ecbe580290-kube-api-access-k9pn5\") pod \"payload-processing-e2e-worker-w0-097e16-65ddd76cdc-nrwtx\" (UID: \"a1792e8c-6841-4193-9ca0-53ecbe580290\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-097e16-65ddd76cdc-nrwtx"
Sep 10 02:58:59.149170 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:59.149141    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47788\" (UniqueName: \"kubernetes.io/projected/a4255849-3d3c-4e26-b766-6d1474682843-kube-api-access-47788\") pod \"payload-pre-processing-e2e-worker-w0-097e16-546949c98-csmxk\" (UID: \"a4255849-3d3c-4e26-b766-6d1474682843\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-097e16-546949c98-csmxk"
Sep 10 02:58:59.149271 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:59.149177    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4255849-3d3c-4e26-b766-6d1474682843-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-097e16-546949c98-csmxk\" (UID: \"a4255849-3d3c-4e26-b766-6d1474682843\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-097e16-546949c98-csmxk"
Sep 10 02:58:59.149767 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:59.149748    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4255849-3d3c-4e26-b766-6d1474682843-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-097e16-546949c98-csmxk\" (UID: \"a4255849-3d3c-4e26-b766-6d1474682843\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-097e16-546949c98-csmxk"
Sep 10 02:58:59.167587 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:59.167558    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47788\" (UniqueName: \"kubernetes.io/projected/a4255849-3d3c-4e26-b766-6d1474682843-kube-api-access-47788\") pod \"payload-pre-processing-e2e-worker-w0-097e16-546949c98-csmxk\" (UID: \"a4255849-3d3c-4e26-b766-6d1474682843\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-097e16-546949c98-csmxk"
Sep 10 02:58:59.169190 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:59.169172    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-097e16-65ddd76cdc-nrwtx"
Sep 10 02:58:59.193462 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:59.193440    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-097e16-546949c98-csmxk"
Sep 10 02:58:59.321770 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:59.321704    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:58:59.388492 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:59.388461    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:58:59.444325 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:59.444299    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-097e16-65ddd76cdc-nrwtx"]
Sep 10 02:58:59.445436 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:58:59.445409    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1792e8c_6841_4193_9ca0_53ecbe580290.slice/crio-f54fd4f4ae90757abd1e4a009b70707a38ed07e5ed5cfd5e388155e4661df592 WatchSource:0}: Error finding container f54fd4f4ae90757abd1e4a009b70707a38ed07e5ed5cfd5e388155e4661df592: Status 404 returned error can't find the container with id f54fd4f4ae90757abd1e4a009b70707a38ed07e5ed5cfd5e388155e4661df592
Sep 10 02:58:59.479022 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:59.479002    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-097e16-546949c98-csmxk"]
Sep 10 02:58:59.491132 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:58:59.491092    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4255849_3d3c_4e26_b766_6d1474682843.slice/crio-9fe20a5ef60edab0343d45df28b18c493dd3f40cd90d9f3b0ccc7f57ad1d002e WatchSource:0}: Error finding container 9fe20a5ef60edab0343d45df28b18c493dd3f40cd90d9f3b0ccc7f57ad1d002e: Status 404 returned error can't find the container with id 9fe20a5ef60edab0343d45df28b18c493dd3f40cd90d9f3b0ccc7f57ad1d002e
Sep 10 02:58:59.632716 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:59.632629    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-097e16-546949c98-csmxk" event={"ID":"a4255849-3d3c-4e26-b766-6d1474682843","Type":"ContainerStarted","Data":"5d8296c90ea9f4938e466d1b282417091299b9e72ea088066757fc3a30725875"}
Sep 10 02:58:59.632716 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:59.632693    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-097e16-546949c98-csmxk" event={"ID":"a4255849-3d3c-4e26-b766-6d1474682843","Type":"ContainerStarted","Data":"9fe20a5ef60edab0343d45df28b18c493dd3f40cd90d9f3b0ccc7f57ad1d002e"}
Sep 10 02:58:59.632910 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:59.632727    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-097e16-546949c98-csmxk"
Sep 10 02:58:59.634332 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:59.634303    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-097e16-65ddd76cdc-nrwtx" event={"ID":"a1792e8c-6841-4193-9ca0-53ecbe580290","Type":"ContainerStarted","Data":"3b9ab0504c679f5b32edb381d00600aed5c563bbbba9b14279710ba4e49e8997"}
Sep 10 02:58:59.634662 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:59.634349    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-097e16-65ddd76cdc-nrwtx" event={"ID":"a1792e8c-6841-4193-9ca0-53ecbe580290","Type":"ContainerStarted","Data":"f54fd4f4ae90757abd1e4a009b70707a38ed07e5ed5cfd5e388155e4661df592"}
Sep 10 02:58:59.634662 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:59.634390    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-097e16-65ddd76cdc-nrwtx"
Sep 10 02:58:59.647865 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:59.647823    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-097e16-546949c98-csmxk" podStartSLOduration=1.6478098540000001 podStartE2EDuration="1.647809854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:58:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:58:59.646853399 +0000 UTC m=+730.021939320" watchObservedRunningTime="2026-09-10 02:58:59.647809854 +0000 UTC m=+730.022895775"
Sep 10 02:58:59.651263 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:59.651242    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:58:59.663373 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:59.663335    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-097e16-65ddd76cdc-nrwtx" podStartSLOduration=1.6633224389999999 podStartE2EDuration="1.663322439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:58:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:58:59.662532185 +0000 UTC m=+730.037618110" watchObservedRunningTime="2026-09-10 02:58:59.663322439 +0000 UTC m=+730.038408398"
Sep 10 02:58:59.897888 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:59.897807    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:58:59.995411 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:59.995382    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:58:59.996280 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:59.996259    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e6e6b8-7978b8f866-wxwzs"]
Sep 10 02:58:59.999724 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:58:59.999706    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e6e6b8-7978b8f866-wxwzs"
Sep 10 02:59:00.001685 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.001664    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-e6e6b8-dockercfg-q4wxg\""
Sep 10 02:59:00.001796 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.001704    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-e6e6b8\""
Sep 10 02:59:00.008716 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.008696    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e6e6b8-7978b8f866-wxwzs"]
Sep 10 02:59:00.021655 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.021636    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e6e6b8-b8cf98d95-dtvhn"]
Sep 10 02:59:00.024930 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.024916    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e6e6b8-b8cf98d95-dtvhn"
Sep 10 02:59:00.035896 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.035876    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e6e6b8-b8cf98d95-dtvhn"]
Sep 10 02:59:00.156988 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.156891    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb1f35e-640b-49f1-bd86-38a8d56b56a6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-e6e6b8-b8cf98d95-dtvhn\" (UID: \"9eb1f35e-640b-49f1-bd86-38a8d56b56a6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e6e6b8-b8cf98d95-dtvhn"
Sep 10 02:59:00.156988 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.156933    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6pmg\" (UniqueName: \"kubernetes.io/projected/9eb1f35e-640b-49f1-bd86-38a8d56b56a6-kube-api-access-k6pmg\") pod \"payload-processing-e2e-worker-w4-e6e6b8-b8cf98d95-dtvhn\" (UID: \"9eb1f35e-640b-49f1-bd86-38a8d56b56a6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e6e6b8-b8cf98d95-dtvhn"
Sep 10 02:59:00.156988 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.156970    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd55b6aa-8203-4025-97ed-0fa65ca98c73-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-e6e6b8-7978b8f866-wxwzs\" (UID: \"bd55b6aa-8203-4025-97ed-0fa65ca98c73\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e6e6b8-7978b8f866-wxwzs"
Sep 10 02:59:00.157314 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.157007    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znk2q\" (UniqueName: \"kubernetes.io/projected/bd55b6aa-8203-4025-97ed-0fa65ca98c73-kube-api-access-znk2q\") pod \"payload-pre-processing-e2e-worker-w4-e6e6b8-7978b8f866-wxwzs\" (UID: \"bd55b6aa-8203-4025-97ed-0fa65ca98c73\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e6e6b8-7978b8f866-wxwzs"
Sep 10 02:59:00.259244 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.258163    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb1f35e-640b-49f1-bd86-38a8d56b56a6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-e6e6b8-b8cf98d95-dtvhn\" (UID: \"9eb1f35e-640b-49f1-bd86-38a8d56b56a6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e6e6b8-b8cf98d95-dtvhn"
Sep 10 02:59:00.259244 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.258212    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6pmg\" (UniqueName: \"kubernetes.io/projected/9eb1f35e-640b-49f1-bd86-38a8d56b56a6-kube-api-access-k6pmg\") pod \"payload-processing-e2e-worker-w4-e6e6b8-b8cf98d95-dtvhn\" (UID: \"9eb1f35e-640b-49f1-bd86-38a8d56b56a6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e6e6b8-b8cf98d95-dtvhn"
Sep 10 02:59:00.259244 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.258252    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd55b6aa-8203-4025-97ed-0fa65ca98c73-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-e6e6b8-7978b8f866-wxwzs\" (UID: \"bd55b6aa-8203-4025-97ed-0fa65ca98c73\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e6e6b8-7978b8f866-wxwzs"
Sep 10 02:59:00.259244 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.258297    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znk2q\" (UniqueName: \"kubernetes.io/projected/bd55b6aa-8203-4025-97ed-0fa65ca98c73-kube-api-access-znk2q\") pod \"payload-pre-processing-e2e-worker-w4-e6e6b8-7978b8f866-wxwzs\" (UID: \"bd55b6aa-8203-4025-97ed-0fa65ca98c73\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e6e6b8-7978b8f866-wxwzs"
Sep 10 02:59:00.259244 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.259178    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb1f35e-640b-49f1-bd86-38a8d56b56a6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-e6e6b8-b8cf98d95-dtvhn\" (UID: \"9eb1f35e-640b-49f1-bd86-38a8d56b56a6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e6e6b8-b8cf98d95-dtvhn"
Sep 10 02:59:00.259683 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.259364    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd55b6aa-8203-4025-97ed-0fa65ca98c73-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-e6e6b8-7978b8f866-wxwzs\" (UID: \"bd55b6aa-8203-4025-97ed-0fa65ca98c73\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e6e6b8-7978b8f866-wxwzs"
Sep 10 02:59:00.268634 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.268598    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6pmg\" (UniqueName: \"kubernetes.io/projected/9eb1f35e-640b-49f1-bd86-38a8d56b56a6-kube-api-access-k6pmg\") pod \"payload-processing-e2e-worker-w4-e6e6b8-b8cf98d95-dtvhn\" (UID: \"9eb1f35e-640b-49f1-bd86-38a8d56b56a6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e6e6b8-b8cf98d95-dtvhn"
Sep 10 02:59:00.269585 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.269561    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znk2q\" (UniqueName: \"kubernetes.io/projected/bd55b6aa-8203-4025-97ed-0fa65ca98c73-kube-api-access-znk2q\") pod \"payload-pre-processing-e2e-worker-w4-e6e6b8-7978b8f866-wxwzs\" (UID: \"bd55b6aa-8203-4025-97ed-0fa65ca98c73\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e6e6b8-7978b8f866-wxwzs"
Sep 10 02:59:00.282470 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.282438    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:00.312803 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.312777    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e6e6b8-7978b8f866-wxwzs"
Sep 10 02:59:00.337360 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.337219    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e6e6b8-b8cf98d95-dtvhn"
Sep 10 02:59:00.383941 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.383907    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:00.591990 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.591967    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e6e6b8-7978b8f866-wxwzs"]
Sep 10 02:59:00.593464 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:59:00.593442    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd55b6aa_8203_4025_97ed_0fa65ca98c73.slice/crio-ac9b767bde29536c5f3b44813f1bbd85955906bd786115b29e19818f68131b69 WatchSource:0}: Error finding container ac9b767bde29536c5f3b44813f1bbd85955906bd786115b29e19818f68131b69: Status 404 returned error can't find the container with id ac9b767bde29536c5f3b44813f1bbd85955906bd786115b29e19818f68131b69
Sep 10 02:59:00.627821 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.627784    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e6e6b8-b8cf98d95-dtvhn"]
Sep 10 02:59:00.630635 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:59:00.630611    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eb1f35e_640b_49f1_bd86_38a8d56b56a6.slice/crio-e8e93d47a8a37412cbeff44819322a89e76f9c581762878af506b5a5722e5253 WatchSource:0}: Error finding container e8e93d47a8a37412cbeff44819322a89e76f9c581762878af506b5a5722e5253: Status 404 returned error can't find the container with id e8e93d47a8a37412cbeff44819322a89e76f9c581762878af506b5a5722e5253
Sep 10 02:59:00.641603 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.641579    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e6e6b8-b8cf98d95-dtvhn" event={"ID":"9eb1f35e-640b-49f1-bd86-38a8d56b56a6","Type":"ContainerStarted","Data":"e8e93d47a8a37412cbeff44819322a89e76f9c581762878af506b5a5722e5253"}
Sep 10 02:59:00.642981 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.642959    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e6e6b8-7978b8f866-wxwzs" event={"ID":"bd55b6aa-8203-4025-97ed-0fa65ca98c73","Type":"ContainerStarted","Data":"ac9b767bde29536c5f3b44813f1bbd85955906bd786115b29e19818f68131b69"}
Sep 10 02:59:00.675991 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.675966    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:00.781931 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.781895    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:00.864476 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.864445    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b0f4da-856f79778-8zd5g"]
Sep 10 02:59:00.866596 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.866566    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:00.868478 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.868457    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0f4da-856f79778-8zd5g"
Sep 10 02:59:00.870781 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.870759    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-b0f4da\""
Sep 10 02:59:00.881018 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.880992    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b0f4da-856f79778-8zd5g"]
Sep 10 02:59:00.901577 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.901552    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b0f4da-77f559485f-2qpw7"]
Sep 10 02:59:00.905650 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.905630    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b0f4da-77f559485f-2qpw7"
Sep 10 02:59:00.907922 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.907897    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-b0f4da-dockercfg-9m84f\""
Sep 10 02:59:00.912270 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.911974    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b0f4da-77f559485f-2qpw7"]
Sep 10 02:59:00.964996 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.964965    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d007f3d-7158-4d17-913b-be4d1f7015ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b0f4da-856f79778-8zd5g\" (UID: \"0d007f3d-7158-4d17-913b-be4d1f7015ef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0f4da-856f79778-8zd5g"
Sep 10 02:59:00.965148 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:00.965125    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jjz6\" (UniqueName: \"kubernetes.io/projected/0d007f3d-7158-4d17-913b-be4d1f7015ef-kube-api-access-2jjz6\") pod \"payload-pre-processing-e2e-inf-a-b0f4da-856f79778-8zd5g\" (UID: \"0d007f3d-7158-4d17-913b-be4d1f7015ef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0f4da-856f79778-8zd5g"
Sep 10 02:59:01.065868 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:01.065831    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d007f3d-7158-4d17-913b-be4d1f7015ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b0f4da-856f79778-8zd5g\" (UID: \"0d007f3d-7158-4d17-913b-be4d1f7015ef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0f4da-856f79778-8zd5g"
Sep 10 02:59:01.066001 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:01.065886    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prqbg\" (UniqueName: \"kubernetes.io/projected/7b487b15-49fd-4368-811a-f8df96477cf1-kube-api-access-prqbg\") pod \"payload-processing-e2e-inf-a-b0f4da-77f559485f-2qpw7\" (UID: \"7b487b15-49fd-4368-811a-f8df96477cf1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b0f4da-77f559485f-2qpw7"
Sep 10 02:59:01.066001 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:01.065987    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jjz6\" (UniqueName: \"kubernetes.io/projected/0d007f3d-7158-4d17-913b-be4d1f7015ef-kube-api-access-2jjz6\") pod \"payload-pre-processing-e2e-inf-a-b0f4da-856f79778-8zd5g\" (UID: \"0d007f3d-7158-4d17-913b-be4d1f7015ef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0f4da-856f79778-8zd5g"
Sep 10 02:59:01.066134 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:01.066033    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b487b15-49fd-4368-811a-f8df96477cf1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b0f4da-77f559485f-2qpw7\" (UID: \"7b487b15-49fd-4368-811a-f8df96477cf1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b0f4da-77f559485f-2qpw7"
Sep 10 02:59:01.066767 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:01.066738    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d007f3d-7158-4d17-913b-be4d1f7015ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b0f4da-856f79778-8zd5g\" (UID: \"0d007f3d-7158-4d17-913b-be4d1f7015ef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0f4da-856f79778-8zd5g"
Sep 10 02:59:01.075597 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:01.075575    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jjz6\" (UniqueName: \"kubernetes.io/projected/0d007f3d-7158-4d17-913b-be4d1f7015ef-kube-api-access-2jjz6\") pod \"payload-pre-processing-e2e-inf-a-b0f4da-856f79778-8zd5g\" (UID: \"0d007f3d-7158-4d17-913b-be4d1f7015ef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0f4da-856f79778-8zd5g"
Sep 10 02:59:01.166877 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:01.166808    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prqbg\" (UniqueName: \"kubernetes.io/projected/7b487b15-49fd-4368-811a-f8df96477cf1-kube-api-access-prqbg\") pod \"payload-processing-e2e-inf-a-b0f4da-77f559485f-2qpw7\" (UID: \"7b487b15-49fd-4368-811a-f8df96477cf1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b0f4da-77f559485f-2qpw7"
Sep 10 02:59:01.166999 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:01.166919    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b487b15-49fd-4368-811a-f8df96477cf1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b0f4da-77f559485f-2qpw7\" (UID: \"7b487b15-49fd-4368-811a-f8df96477cf1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b0f4da-77f559485f-2qpw7"
Sep 10 02:59:01.167640 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:01.167615    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b487b15-49fd-4368-811a-f8df96477cf1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b0f4da-77f559485f-2qpw7\" (UID: \"7b487b15-49fd-4368-811a-f8df96477cf1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b0f4da-77f559485f-2qpw7"
Sep 10 02:59:01.174737 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:01.174710    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prqbg\" (UniqueName: \"kubernetes.io/projected/7b487b15-49fd-4368-811a-f8df96477cf1-kube-api-access-prqbg\") pod \"payload-processing-e2e-inf-a-b0f4da-77f559485f-2qpw7\" (UID: \"7b487b15-49fd-4368-811a-f8df96477cf1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b0f4da-77f559485f-2qpw7"
Sep 10 02:59:01.181552 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:01.181525    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0f4da-856f79778-8zd5g"
Sep 10 02:59:01.217807 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:01.217739    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b0f4da-77f559485f-2qpw7"
Sep 10 02:59:01.372955 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:01.372928    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:01.455865 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:01.455816    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b0f4da-856f79778-8zd5g"]
Sep 10 02:59:01.457961 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:59:01.457931    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d007f3d_7158_4d17_913b_be4d1f7015ef.slice/crio-e5671bc33ef539b809cf4da99636979524ee814de992d3a954e1981e8c9fdca2 WatchSource:0}: Error finding container e5671bc33ef539b809cf4da99636979524ee814de992d3a954e1981e8c9fdca2: Status 404 returned error can't find the container with id e5671bc33ef539b809cf4da99636979524ee814de992d3a954e1981e8c9fdca2
Sep 10 02:59:01.504203 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:01.504182    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b0f4da-77f559485f-2qpw7"]
Sep 10 02:59:01.505770 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:59:01.505746    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b487b15_49fd_4368_811a_f8df96477cf1.slice/crio-642d3ebbfe99a4b0f3ee8c225bfae5496013bdb1ac3e24cbaf21f68d3f154f67 WatchSource:0}: Error finding container 642d3ebbfe99a4b0f3ee8c225bfae5496013bdb1ac3e24cbaf21f68d3f154f67: Status 404 returned error can't find the container with id 642d3ebbfe99a4b0f3ee8c225bfae5496013bdb1ac3e24cbaf21f68d3f154f67
Sep 10 02:59:01.653492 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:01.653456    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e6e6b8-7978b8f866-wxwzs" event={"ID":"bd55b6aa-8203-4025-97ed-0fa65ca98c73","Type":"ContainerStarted","Data":"e3d4684c53cef23f359f4ac32240164acfb83a68a6855d194a8bc5ebd25124f5"}
Sep 10 02:59:01.653658 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:01.653553    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e6e6b8-7978b8f866-wxwzs"
Sep 10 02:59:01.655708 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:01.655679    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0f4da-856f79778-8zd5g" event={"ID":"0d007f3d-7158-4d17-913b-be4d1f7015ef","Type":"ContainerStarted","Data":"9f1c798664477224ec155261416c99bd785d0c076c24801436dd0285aa122a46"}
Sep 10 02:59:01.655856 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:01.655731    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0f4da-856f79778-8zd5g"
Sep 10 02:59:01.655856 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:01.655756    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0f4da-856f79778-8zd5g" event={"ID":"0d007f3d-7158-4d17-913b-be4d1f7015ef","Type":"ContainerStarted","Data":"e5671bc33ef539b809cf4da99636979524ee814de992d3a954e1981e8c9fdca2"}
Sep 10 02:59:01.661652 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:01.661625    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b0f4da-77f559485f-2qpw7" event={"ID":"7b487b15-49fd-4368-811a-f8df96477cf1","Type":"ContainerStarted","Data":"af0ae470f0eda6bd2deef1bc77730b274c866d8cf925826c4165bdde69dfd61c"}
Sep 10 02:59:01.661793 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:01.661674    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b0f4da-77f559485f-2qpw7" event={"ID":"7b487b15-49fd-4368-811a-f8df96477cf1","Type":"ContainerStarted","Data":"642d3ebbfe99a4b0f3ee8c225bfae5496013bdb1ac3e24cbaf21f68d3f154f67"}
Sep 10 02:59:01.661793 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:01.661753    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b0f4da-77f559485f-2qpw7"
Sep 10 02:59:01.663651 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:01.663607    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e6e6b8-b8cf98d95-dtvhn" event={"ID":"9eb1f35e-640b-49f1-bd86-38a8d56b56a6","Type":"ContainerStarted","Data":"6c5aadab4c633ce15da418359ebc680d79fa7d0003fd44eb2fca38917d9e1853"}
Sep 10 02:59:01.673270 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:01.673193    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e6e6b8-7978b8f866-wxwzs" podStartSLOduration=2.673172637 podStartE2EDuration="2.673172637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:59:01.671992716 +0000 UTC m=+732.047078637" watchObservedRunningTime="2026-09-10 02:59:01.673172637 +0000 UTC m=+732.048258558"
Sep 10 02:59:01.687569 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:01.687475    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0f4da-856f79778-8zd5g" podStartSLOduration=1.687457619 podStartE2EDuration="1.687457619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:59:01.686952795 +0000 UTC m=+732.062038716" watchObservedRunningTime="2026-09-10 02:59:01.687457619 +0000 UTC m=+732.062543542"
Sep 10 02:59:01.703381 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:01.703330    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-e6e6b8-b8cf98d95-dtvhn" podStartSLOduration=1.703313637 podStartE2EDuration="1.703313637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:59:01.70261827 +0000 UTC m=+732.077704195" watchObservedRunningTime="2026-09-10 02:59:01.703313637 +0000 UTC m=+732.078399558"
Sep 10 02:59:01.723216 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:01.723149    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-b0f4da-77f559485f-2qpw7" podStartSLOduration=1.723137632 podStartE2EDuration="1.723137632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:59:01.721123812 +0000 UTC m=+732.096209727" watchObservedRunningTime="2026-09-10 02:59:01.723137632 +0000 UTC m=+732.098223554"
Sep 10 02:59:02.096234 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:02.096127    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:02.185550 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:02.185502    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:02.285250 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:02.285220    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:02.385636 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:02.385568    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:02.638980 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:02.638906    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:02.670071 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:02.670047    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-e6e6b8-b8cf98d95-dtvhn"
Sep 10 02:59:02.982935 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:02.982909    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:03.078902 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:03.078868    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:03.199544 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:03.199515    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:03.300405 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:03.300319    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:03.452041 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:03.452010    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:05.212511 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:05.212476    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:05.317159 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:05.317090    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:06.402122 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:06.402062    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:06.502823 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:06.502793    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:06.754997 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:06.754967    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:06.855833 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:06.855799    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:07.011264 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:07.011156    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:07.112575 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:07.112525    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:07.135963 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:07.135933    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-644ff039-6d756dfb58-xsh5l"]
Sep 10 02:59:07.140738 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:07.140713    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-644ff039-6d756dfb58-xsh5l"
Sep 10 02:59:07.143060 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:07.142980    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-644ff039-5b957d9799-qcxcn"]
Sep 10 02:59:07.143213 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:07.143084    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-644ff039\""
Sep 10 02:59:07.143213 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:07.143156    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-644ff039-dockercfg-zspzx\""
Sep 10 02:59:07.146412 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:07.146394    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-644ff039-5b957d9799-qcxcn"
Sep 10 02:59:07.152463 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:07.152443    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-644ff039-6d756dfb58-xsh5l"]
Sep 10 02:59:07.160933 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:07.160782    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-644ff039-5b957d9799-qcxcn"]
Sep 10 02:59:07.321858 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:07.321781    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f041de4d-d7c8-406a-b032-051c4d91087c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-644ff039-6d756dfb58-xsh5l\" (UID: \"f041de4d-d7c8-406a-b032-051c4d91087c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-644ff039-6d756dfb58-xsh5l"
Sep 10 02:59:07.321858 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:07.321820    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6vm7\" (UniqueName: \"kubernetes.io/projected/72a28ae9-0d97-4245-9611-bb2326dfcef7-kube-api-access-q6vm7\") pod \"payload-processing-e2e-ait-644ff039-5b957d9799-qcxcn\" (UID: \"72a28ae9-0d97-4245-9611-bb2326dfcef7\") " pod="openshift-ingress/payload-processing-e2e-ait-644ff039-5b957d9799-qcxcn"
Sep 10 02:59:07.321858 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:07.321845    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dvb8\" (UniqueName: \"kubernetes.io/projected/f041de4d-d7c8-406a-b032-051c4d91087c-kube-api-access-5dvb8\") pod \"payload-pre-processing-e2e-ait-644ff039-6d756dfb58-xsh5l\" (UID: \"f041de4d-d7c8-406a-b032-051c4d91087c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-644ff039-6d756dfb58-xsh5l"
Sep 10 02:59:07.322070 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:07.321926    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72a28ae9-0d97-4245-9611-bb2326dfcef7-plugins-config-volume\") pod \"payload-processing-e2e-ait-644ff039-5b957d9799-qcxcn\" (UID: \"72a28ae9-0d97-4245-9611-bb2326dfcef7\") " pod="openshift-ingress/payload-processing-e2e-ait-644ff039-5b957d9799-qcxcn"
Sep 10 02:59:07.422451 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:07.422417    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72a28ae9-0d97-4245-9611-bb2326dfcef7-plugins-config-volume\") pod \"payload-processing-e2e-ait-644ff039-5b957d9799-qcxcn\" (UID: \"72a28ae9-0d97-4245-9611-bb2326dfcef7\") " pod="openshift-ingress/payload-processing-e2e-ait-644ff039-5b957d9799-qcxcn"
Sep 10 02:59:07.422922 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:07.422501    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f041de4d-d7c8-406a-b032-051c4d91087c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-644ff039-6d756dfb58-xsh5l\" (UID: \"f041de4d-d7c8-406a-b032-051c4d91087c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-644ff039-6d756dfb58-xsh5l"
Sep 10 02:59:07.422922 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:07.422598    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6vm7\" (UniqueName: \"kubernetes.io/projected/72a28ae9-0d97-4245-9611-bb2326dfcef7-kube-api-access-q6vm7\") pod \"payload-processing-e2e-ait-644ff039-5b957d9799-qcxcn\" (UID: \"72a28ae9-0d97-4245-9611-bb2326dfcef7\") " pod="openshift-ingress/payload-processing-e2e-ait-644ff039-5b957d9799-qcxcn"
Sep 10 02:59:07.422922 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:07.422642    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dvb8\" (UniqueName: \"kubernetes.io/projected/f041de4d-d7c8-406a-b032-051c4d91087c-kube-api-access-5dvb8\") pod \"payload-pre-processing-e2e-ait-644ff039-6d756dfb58-xsh5l\" (UID: \"f041de4d-d7c8-406a-b032-051c4d91087c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-644ff039-6d756dfb58-xsh5l"
Sep 10 02:59:07.423203 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:07.423184    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f041de4d-d7c8-406a-b032-051c4d91087c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-644ff039-6d756dfb58-xsh5l\" (UID: \"f041de4d-d7c8-406a-b032-051c4d91087c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-644ff039-6d756dfb58-xsh5l"
Sep 10 02:59:07.423329 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:07.423309    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72a28ae9-0d97-4245-9611-bb2326dfcef7-plugins-config-volume\") pod \"payload-processing-e2e-ait-644ff039-5b957d9799-qcxcn\" (UID: \"72a28ae9-0d97-4245-9611-bb2326dfcef7\") " pod="openshift-ingress/payload-processing-e2e-ait-644ff039-5b957d9799-qcxcn"
Sep 10 02:59:07.431802 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:07.431779    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dvb8\" (UniqueName: \"kubernetes.io/projected/f041de4d-d7c8-406a-b032-051c4d91087c-kube-api-access-5dvb8\") pod \"payload-pre-processing-e2e-ait-644ff039-6d756dfb58-xsh5l\" (UID: \"f041de4d-d7c8-406a-b032-051c4d91087c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-644ff039-6d756dfb58-xsh5l"
Sep 10 02:59:07.431905 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:07.431884    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6vm7\" (UniqueName: \"kubernetes.io/projected/72a28ae9-0d97-4245-9611-bb2326dfcef7-kube-api-access-q6vm7\") pod \"payload-processing-e2e-ait-644ff039-5b957d9799-qcxcn\" (UID: \"72a28ae9-0d97-4245-9611-bb2326dfcef7\") " pod="openshift-ingress/payload-processing-e2e-ait-644ff039-5b957d9799-qcxcn"
Sep 10 02:59:07.454879 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:07.454853    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-644ff039-6d756dfb58-xsh5l"
Sep 10 02:59:07.466218 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:07.466195    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-644ff039-5b957d9799-qcxcn"
Sep 10 02:59:07.752795 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:07.752772    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-644ff039-6d756dfb58-xsh5l"]
Sep 10 02:59:07.754163 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:59:07.754133    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf041de4d_d7c8_406a_b032_051c4d91087c.slice/crio-7bc560d0b21f8c7c8a72a9fdf446bfe0647db0634a700abf93eff7a190afec18 WatchSource:0}: Error finding container 7bc560d0b21f8c7c8a72a9fdf446bfe0647db0634a700abf93eff7a190afec18: Status 404 returned error can't find the container with id 7bc560d0b21f8c7c8a72a9fdf446bfe0647db0634a700abf93eff7a190afec18
Sep 10 02:59:07.762423 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:07.762341    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-644ff039-5b957d9799-qcxcn"]
Sep 10 02:59:07.764542 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:59:07.764522    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72a28ae9_0d97_4245_9611_bb2326dfcef7.slice/crio-cdd664e3379a0ca8bf94ae48a6e18ed005816b52a8b86efab8d8850f6d371763 WatchSource:0}: Error finding container cdd664e3379a0ca8bf94ae48a6e18ed005816b52a8b86efab8d8850f6d371763: Status 404 returned error can't find the container with id cdd664e3379a0ca8bf94ae48a6e18ed005816b52a8b86efab8d8850f6d371763
Sep 10 02:59:08.237538 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:08.237510    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:08.337504 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:08.337478    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:08.571729 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:08.571657    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:08.706954 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:08.706927    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:08.709227 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:08.709203    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-644ff039-5b957d9799-qcxcn" event={"ID":"72a28ae9-0d97-4245-9611-bb2326dfcef7","Type":"ContainerStarted","Data":"66a4cf9c72c8d04a744bec546eea0a8a26ccbfb0483df714b4741b8acf38e13c"}
Sep 10 02:59:08.709355 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:08.709272    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-644ff039-5b957d9799-qcxcn" event={"ID":"72a28ae9-0d97-4245-9611-bb2326dfcef7","Type":"ContainerStarted","Data":"cdd664e3379a0ca8bf94ae48a6e18ed005816b52a8b86efab8d8850f6d371763"}
Sep 10 02:59:08.709355 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:08.709303    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-644ff039-5b957d9799-qcxcn"
Sep 10 02:59:08.710840 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:08.710821    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-644ff039-6d756dfb58-xsh5l" event={"ID":"f041de4d-d7c8-406a-b032-051c4d91087c","Type":"ContainerStarted","Data":"b9b858fcb619b5024af968e9d1c65d7311fb68b2a93a686c4c97aef86a391096"}
Sep 10 02:59:08.710920 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:08.710857    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-644ff039-6d756dfb58-xsh5l" event={"ID":"f041de4d-d7c8-406a-b032-051c4d91087c","Type":"ContainerStarted","Data":"7bc560d0b21f8c7c8a72a9fdf446bfe0647db0634a700abf93eff7a190afec18"}
Sep 10 02:59:08.710920 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:08.710883    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-644ff039-6d756dfb58-xsh5l"
Sep 10 02:59:08.731660 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:08.731617    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-644ff039-5b957d9799-qcxcn" podStartSLOduration=1.7316009 podStartE2EDuration="1.7316009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:59:08.731017968 +0000 UTC m=+739.106103890" watchObservedRunningTime="2026-09-10 02:59:08.7316009 +0000 UTC m=+739.106686821"
Sep 10 02:59:08.751444 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:08.751403    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-644ff039-6d756dfb58-xsh5l" podStartSLOduration=1.7513889489999999 podStartE2EDuration="1.751388949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:59:08.749934747 +0000 UTC m=+739.125020667" watchObservedRunningTime="2026-09-10 02:59:08.751388949 +0000 UTC m=+739.126474871"
Sep 10 02:59:08.807434 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:08.807398    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:08.851988 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:08.851924    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5ab28b-c7d5fcf85-svr5v"]
Sep 10 02:59:08.855591 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:08.855572    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-5ab28b-c7d5fcf85-svr5v"
Sep 10 02:59:08.857869 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:08.857813    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-5ab28b\""
Sep 10 02:59:08.868910 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:08.868888    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5ab28b-c7d5fcf85-svr5v"]
Sep 10 02:59:09.041508 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:09.041476    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ffcda36-f226-4591-bac6-3c472c0aaee8-plugins-config-volume\") pod \"payload-processing-e2e-autores-5ab28b-c7d5fcf85-svr5v\" (UID: \"1ffcda36-f226-4591-bac6-3c472c0aaee8\") " pod="openshift-ingress/payload-processing-e2e-autores-5ab28b-c7d5fcf85-svr5v"
Sep 10 02:59:09.041633 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:09.041561    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfn6x\" (UniqueName: \"kubernetes.io/projected/1ffcda36-f226-4591-bac6-3c472c0aaee8-kube-api-access-zfn6x\") pod \"payload-processing-e2e-autores-5ab28b-c7d5fcf85-svr5v\" (UID: \"1ffcda36-f226-4591-bac6-3c472c0aaee8\") " pod="openshift-ingress/payload-processing-e2e-autores-5ab28b-c7d5fcf85-svr5v"
Sep 10 02:59:09.142148 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:09.142085    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ffcda36-f226-4591-bac6-3c472c0aaee8-plugins-config-volume\") pod \"payload-processing-e2e-autores-5ab28b-c7d5fcf85-svr5v\" (UID: \"1ffcda36-f226-4591-bac6-3c472c0aaee8\") " pod="openshift-ingress/payload-processing-e2e-autores-5ab28b-c7d5fcf85-svr5v"
Sep 10 02:59:09.142265 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:09.142190    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfn6x\" (UniqueName: \"kubernetes.io/projected/1ffcda36-f226-4591-bac6-3c472c0aaee8-kube-api-access-zfn6x\") pod \"payload-processing-e2e-autores-5ab28b-c7d5fcf85-svr5v\" (UID: \"1ffcda36-f226-4591-bac6-3c472c0aaee8\") " pod="openshift-ingress/payload-processing-e2e-autores-5ab28b-c7d5fcf85-svr5v"
Sep 10 02:59:09.142779 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:09.142761    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ffcda36-f226-4591-bac6-3c472c0aaee8-plugins-config-volume\") pod \"payload-processing-e2e-autores-5ab28b-c7d5fcf85-svr5v\" (UID: \"1ffcda36-f226-4591-bac6-3c472c0aaee8\") " pod="openshift-ingress/payload-processing-e2e-autores-5ab28b-c7d5fcf85-svr5v"
Sep 10 02:59:09.149883 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:09.149855    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfn6x\" (UniqueName: \"kubernetes.io/projected/1ffcda36-f226-4591-bac6-3c472c0aaee8-kube-api-access-zfn6x\") pod \"payload-processing-e2e-autores-5ab28b-c7d5fcf85-svr5v\" (UID: \"1ffcda36-f226-4591-bac6-3c472c0aaee8\") " pod="openshift-ingress/payload-processing-e2e-autores-5ab28b-c7d5fcf85-svr5v"
Sep 10 02:59:09.169731 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:09.169706    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-5ab28b-c7d5fcf85-svr5v"
Sep 10 02:59:09.441129 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:09.441084    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5ab28b-c7d5fcf85-svr5v"]
Sep 10 02:59:09.442537 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:59:09.442513    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ffcda36_f226_4591_bac6_3c472c0aaee8.slice/crio-881564a67807f9a6849207b463632dc791c70696f3fe21c788e9ad8535d532d5 WatchSource:0}: Error finding container 881564a67807f9a6849207b463632dc791c70696f3fe21c788e9ad8535d532d5: Status 404 returned error can't find the container with id 881564a67807f9a6849207b463632dc791c70696f3fe21c788e9ad8535d532d5
Sep 10 02:59:09.718261 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:09.718226    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-5ab28b-c7d5fcf85-svr5v" event={"ID":"1ffcda36-f226-4591-bac6-3c472c0aaee8","Type":"ContainerStarted","Data":"45dd2bad9b2d1a381172f694f5f97f56e93c471b46374db2fe3947365c03ced3"}
Sep 10 02:59:09.718725 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:09.718288    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-5ab28b-c7d5fcf85-svr5v" event={"ID":"1ffcda36-f226-4591-bac6-3c472c0aaee8","Type":"ContainerStarted","Data":"881564a67807f9a6849207b463632dc791c70696f3fe21c788e9ad8535d532d5"}
Sep 10 02:59:09.718725 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:09.718626    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-5ab28b-c7d5fcf85-svr5v"
Sep 10 02:59:09.733546 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:09.733490    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-5ab28b-c7d5fcf85-svr5v" podStartSLOduration=1.733472718 podStartE2EDuration="1.733472718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:59:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:59:09.732240731 +0000 UTC m=+740.107326679" watchObservedRunningTime="2026-09-10 02:59:09.733472718 +0000 UTC m=+740.108558642"
Sep 10 02:59:10.645479 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:10.645447    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-097e16-546949c98-csmxk"
Sep 10 02:59:10.645640 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:10.645517    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-097e16-65ddd76cdc-nrwtx"
Sep 10 02:59:12.426498 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:12.426460    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:12.586590 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:12.586559    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:12.672629 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:12.672603    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0f4da-856f79778-8zd5g"
Sep 10 02:59:12.672801 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:12.672682    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b0f4da-77f559485f-2qpw7"
Sep 10 02:59:12.672874 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:12.672833    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e6e6b8-7978b8f866-wxwzs"
Sep 10 02:59:12.680089 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:12.680027    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:12.779467 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:12.779429    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:13.681601 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:13.681572    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-e6e6b8-b8cf98d95-dtvhn"
Sep 10 02:59:15.509497 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:15.509469    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:17.478572 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:17.478540    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:19.203453 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:19.203420    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:19.720289 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:19.720257    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-644ff039-6d756dfb58-xsh5l"
Sep 10 02:59:19.720440 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:19.720346    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-644ff039-5b957d9799-qcxcn"
Sep 10 02:59:20.497304 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:20.497264    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:21.295473 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:21.295443    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:21.732622 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:21.732596    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-5ab28b-c7d5fcf85-svr5v"
Sep 10 02:59:21.988548 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:21.988452    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:22.090254 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:22.090226    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:22.519861 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:22.519801    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:22.662437 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:22.662388    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:23.294532 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:23.294482    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:23.954722 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:23.954691    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:26.802316 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:26.802277    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:26.852965 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:26.852937    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:27.136880 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:27.136804    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:27.501994 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:27.501970    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:27.602395 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:27.602368    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:27.843586 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:27.843505    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:28.086716 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:28.086681    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:29.363856 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:29.363814    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:29.607668 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:29.607637    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:29.695645 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:29.695568    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:29.837414 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:29.837389    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:31.052765 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:31.052732    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:31.213091 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:31.213065    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:31.817747 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:31.817714    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:32.267516 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:32.267484    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:32.331335 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:32.331301    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:32.582694 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:32.582620    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:32.745442 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:32.745414    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:32.823617 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:32.823591    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:35.547025 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:35.546991    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:35.662043 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:35.662016    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:35.747010 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:35.746976    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:35.918952 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:35.918877    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:37.103403 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:37.103366    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:42.023138 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:42.023089    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:42.121018 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:42.120983    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:46.679382 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:46.679350    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:47.283516 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.283479    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:47.311661 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.311631    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-644ff039-5b957d9799-qcxcn"]
Sep 10 02:59:47.311868 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.311842    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-644ff039-5b957d9799-qcxcn" podUID="72a28ae9-0d97-4245-9611-bb2326dfcef7" containerName="payload-processing" containerID="cri-o://66a4cf9c72c8d04a744bec546eea0a8a26ccbfb0483df714b4741b8acf38e13c" gracePeriod=30
Sep 10 02:59:47.317030 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.317005    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-644ff039-6d756dfb58-xsh5l"]
Sep 10 02:59:47.317247 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.317203    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-644ff039-6d756dfb58-xsh5l" podUID="f041de4d-d7c8-406a-b032-051c4d91087c" containerName="payload-pre-processing" containerID="cri-o://b9b858fcb619b5024af968e9d1c65d7311fb68b2a93a686c4c97aef86a391096" gracePeriod=30
Sep 10 02:59:47.394697 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.394666    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:47.495383 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.495328    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:47.562515 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.562491    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-644ff039-5b957d9799-qcxcn"
Sep 10 02:59:47.583343 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.583314    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:47.591391 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.591371    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-644ff039-6d756dfb58-xsh5l"
Sep 10 02:59:47.651220 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.651193    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72a28ae9-0d97-4245-9611-bb2326dfcef7-plugins-config-volume\") pod \"72a28ae9-0d97-4245-9611-bb2326dfcef7\" (UID: \"72a28ae9-0d97-4245-9611-bb2326dfcef7\") "
Sep 10 02:59:47.651396 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.651260    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dvb8\" (UniqueName: \"kubernetes.io/projected/f041de4d-d7c8-406a-b032-051c4d91087c-kube-api-access-5dvb8\") pod \"f041de4d-d7c8-406a-b032-051c4d91087c\" (UID: \"f041de4d-d7c8-406a-b032-051c4d91087c\") "
Sep 10 02:59:47.651396 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.651334    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6vm7\" (UniqueName: \"kubernetes.io/projected/72a28ae9-0d97-4245-9611-bb2326dfcef7-kube-api-access-q6vm7\") pod \"72a28ae9-0d97-4245-9611-bb2326dfcef7\" (UID: \"72a28ae9-0d97-4245-9611-bb2326dfcef7\") "
Sep 10 02:59:47.651396 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.651377    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f041de4d-d7c8-406a-b032-051c4d91087c-plugins-config-volume\") pod \"f041de4d-d7c8-406a-b032-051c4d91087c\" (UID: \"f041de4d-d7c8-406a-b032-051c4d91087c\") "
Sep 10 02:59:47.651966 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.651938    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f041de4d-d7c8-406a-b032-051c4d91087c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f041de4d-d7c8-406a-b032-051c4d91087c" (UID: "f041de4d-d7c8-406a-b032-051c4d91087c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 02:59:47.652209 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.652180    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72a28ae9-0d97-4245-9611-bb2326dfcef7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "72a28ae9-0d97-4245-9611-bb2326dfcef7" (UID: "72a28ae9-0d97-4245-9611-bb2326dfcef7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 02:59:47.654220 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.654190    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f041de4d-d7c8-406a-b032-051c4d91087c-kube-api-access-5dvb8" (OuterVolumeSpecName: "kube-api-access-5dvb8") pod "f041de4d-d7c8-406a-b032-051c4d91087c" (UID: "f041de4d-d7c8-406a-b032-051c4d91087c"). InnerVolumeSpecName "kube-api-access-5dvb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 02:59:47.654337 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.654315    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a28ae9-0d97-4245-9611-bb2326dfcef7-kube-api-access-q6vm7" (OuterVolumeSpecName: "kube-api-access-q6vm7") pod "72a28ae9-0d97-4245-9611-bb2326dfcef7" (UID: "72a28ae9-0d97-4245-9611-bb2326dfcef7"). InnerVolumeSpecName "kube-api-access-q6vm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 02:59:47.673657 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.673632    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:47.752723 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.752698    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72a28ae9-0d97-4245-9611-bb2326dfcef7-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:59:47.752723 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.752721    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dvb8\" (UniqueName: \"kubernetes.io/projected/f041de4d-d7c8-406a-b032-051c4d91087c-kube-api-access-5dvb8\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:59:47.753058 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.752731    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6vm7\" (UniqueName: \"kubernetes.io/projected/72a28ae9-0d97-4245-9611-bb2326dfcef7-kube-api-access-q6vm7\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:59:47.753058 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.752740    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f041de4d-d7c8-406a-b032-051c4d91087c-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 02:59:47.939292 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.939221    2558 generic.go:356] "Generic (PLEG): container finished" podID="72a28ae9-0d97-4245-9611-bb2326dfcef7" containerID="66a4cf9c72c8d04a744bec546eea0a8a26ccbfb0483df714b4741b8acf38e13c" exitCode=0
Sep 10 02:59:47.939292 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.939276    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-644ff039-5b957d9799-qcxcn"
Sep 10 02:59:47.939292 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.939289    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-644ff039-5b957d9799-qcxcn" event={"ID":"72a28ae9-0d97-4245-9611-bb2326dfcef7","Type":"ContainerDied","Data":"66a4cf9c72c8d04a744bec546eea0a8a26ccbfb0483df714b4741b8acf38e13c"}
Sep 10 02:59:47.939481 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.939374    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-644ff039-5b957d9799-qcxcn" event={"ID":"72a28ae9-0d97-4245-9611-bb2326dfcef7","Type":"ContainerDied","Data":"cdd664e3379a0ca8bf94ae48a6e18ed005816b52a8b86efab8d8850f6d371763"}
Sep 10 02:59:47.939481 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.939408    2558 scope.go:117] "RemoveContainer" containerID="66a4cf9c72c8d04a744bec546eea0a8a26ccbfb0483df714b4741b8acf38e13c"
Sep 10 02:59:47.940677 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.940655    2558 generic.go:356] "Generic (PLEG): container finished" podID="f041de4d-d7c8-406a-b032-051c4d91087c" containerID="b9b858fcb619b5024af968e9d1c65d7311fb68b2a93a686c4c97aef86a391096" exitCode=0
Sep 10 02:59:47.940784 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.940715    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-644ff039-6d756dfb58-xsh5l"
Sep 10 02:59:47.940784 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.940739    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-644ff039-6d756dfb58-xsh5l" event={"ID":"f041de4d-d7c8-406a-b032-051c4d91087c","Type":"ContainerDied","Data":"b9b858fcb619b5024af968e9d1c65d7311fb68b2a93a686c4c97aef86a391096"}
Sep 10 02:59:47.940784 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.940781    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-644ff039-6d756dfb58-xsh5l" event={"ID":"f041de4d-d7c8-406a-b032-051c4d91087c","Type":"ContainerDied","Data":"7bc560d0b21f8c7c8a72a9fdf446bfe0647db0634a700abf93eff7a190afec18"}
Sep 10 02:59:47.949001 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.948983    2558 scope.go:117] "RemoveContainer" containerID="66a4cf9c72c8d04a744bec546eea0a8a26ccbfb0483df714b4741b8acf38e13c"
Sep 10 02:59:47.949351 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:59:47.949325    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66a4cf9c72c8d04a744bec546eea0a8a26ccbfb0483df714b4741b8acf38e13c\": container with ID starting with 66a4cf9c72c8d04a744bec546eea0a8a26ccbfb0483df714b4741b8acf38e13c not found: ID does not exist" containerID="66a4cf9c72c8d04a744bec546eea0a8a26ccbfb0483df714b4741b8acf38e13c"
Sep 10 02:59:47.949436 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.949361    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66a4cf9c72c8d04a744bec546eea0a8a26ccbfb0483df714b4741b8acf38e13c"} err="failed to get container status \"66a4cf9c72c8d04a744bec546eea0a8a26ccbfb0483df714b4741b8acf38e13c\": rpc error: code = NotFound desc = could not find container \"66a4cf9c72c8d04a744bec546eea0a8a26ccbfb0483df714b4741b8acf38e13c\": container with ID starting with 66a4cf9c72c8d04a744bec546eea0a8a26ccbfb0483df714b4741b8acf38e13c not found: ID does not exist"
Sep 10 02:59:47.949436 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.949386    2558 scope.go:117] "RemoveContainer" containerID="b9b858fcb619b5024af968e9d1c65d7311fb68b2a93a686c4c97aef86a391096"
Sep 10 02:59:47.958776 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.958759    2558 scope.go:117] "RemoveContainer" containerID="b9b858fcb619b5024af968e9d1c65d7311fb68b2a93a686c4c97aef86a391096"
Sep 10 02:59:47.959085 ip-10-0-128-101 kubenswrapper[2558]: E0910 02:59:47.959062    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9b858fcb619b5024af968e9d1c65d7311fb68b2a93a686c4c97aef86a391096\": container with ID starting with b9b858fcb619b5024af968e9d1c65d7311fb68b2a93a686c4c97aef86a391096 not found: ID does not exist" containerID="b9b858fcb619b5024af968e9d1c65d7311fb68b2a93a686c4c97aef86a391096"
Sep 10 02:59:47.959170 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.959094    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9b858fcb619b5024af968e9d1c65d7311fb68b2a93a686c4c97aef86a391096"} err="failed to get container status \"b9b858fcb619b5024af968e9d1c65d7311fb68b2a93a686c4c97aef86a391096\": rpc error: code = NotFound desc = could not find container \"b9b858fcb619b5024af968e9d1c65d7311fb68b2a93a686c4c97aef86a391096\": container with ID starting with b9b858fcb619b5024af968e9d1c65d7311fb68b2a93a686c4c97aef86a391096 not found: ID does not exist"
Sep 10 02:59:47.964524 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.964493    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-644ff039-6d756dfb58-xsh5l"]
Sep 10 02:59:47.968365 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.968343    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-644ff039-6d756dfb58-xsh5l"]
Sep 10 02:59:47.980239 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.980220    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-644ff039-5b957d9799-qcxcn"]
Sep 10 02:59:47.985339 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:47.985314    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-644ff039-5b957d9799-qcxcn"]
Sep 10 02:59:48.068922 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:48.068892    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72a28ae9-0d97-4245-9611-bb2326dfcef7" path="/var/lib/kubelet/pods/72a28ae9-0d97-4245-9611-bb2326dfcef7/volumes"
Sep 10 02:59:48.069342 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:48.069324    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f041de4d-d7c8-406a-b032-051c4d91087c" path="/var/lib/kubelet/pods/f041de4d-d7c8-406a-b032-051c4d91087c/volumes"
Sep 10 02:59:48.233411 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:48.233369    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:48.335073 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:48.335050    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:48.470674 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:48.470640    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:48.790641 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:48.790610    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:49.265904 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:49.265878    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:51.666402 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:51.666358    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:51.770871 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:51.770839    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:51.865866 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:51.865834    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:52.095370 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:52.095334    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:52.300212 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:52.300178    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:52.403775 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:52.403701    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:52.414651 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:52.414623    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-387ddcaa-55ff99db6-jrxbs"]
Sep 10 02:59:52.415018 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:52.415005    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72a28ae9-0d97-4245-9611-bb2326dfcef7" containerName="payload-processing"
Sep 10 02:59:52.415083 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:52.415019    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="72a28ae9-0d97-4245-9611-bb2326dfcef7" containerName="payload-processing"
Sep 10 02:59:52.415083 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:52.415055    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f041de4d-d7c8-406a-b032-051c4d91087c" containerName="payload-pre-processing"
Sep 10 02:59:52.415083 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:52.415062    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f041de4d-d7c8-406a-b032-051c4d91087c" containerName="payload-pre-processing"
Sep 10 02:59:52.415229 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:52.415140    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="72a28ae9-0d97-4245-9611-bb2326dfcef7" containerName="payload-processing"
Sep 10 02:59:52.415229 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:52.415158    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f041de4d-d7c8-406a-b032-051c4d91087c" containerName="payload-pre-processing"
Sep 10 02:59:52.420041 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:52.420021    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-387ddcaa-55ff99db6-jrxbs"
Sep 10 02:59:52.422314 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:52.422296    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-387ddcaa\""
Sep 10 02:59:52.422416 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:52.422395    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-387ddcaa-dockercfg-j7m6b\""
Sep 10 02:59:52.427334 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:52.427316    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-387ddcaa-8596dcf594-mwxvf"]
Sep 10 02:59:52.430636 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:52.430522    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-387ddcaa-8596dcf594-mwxvf"
Sep 10 02:59:52.433657 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:52.433635    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-387ddcaa-55ff99db6-jrxbs"]
Sep 10 02:59:52.442192 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:52.442165    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-387ddcaa-8596dcf594-mwxvf"]
Sep 10 02:59:52.495397 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:52.495366    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e60d8eb-3c5d-4849-8125-6d1d9b037751-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-387ddcaa-55ff99db6-jrxbs\" (UID: \"1e60d8eb-3c5d-4849-8125-6d1d9b037751\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-387ddcaa-55ff99db6-jrxbs"
Sep 10 02:59:52.495506 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:52.495412    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlrs6\" (UniqueName: \"kubernetes.io/projected/274bdac5-2654-4854-bdaa-f343b61c2dd2-kube-api-access-tlrs6\") pod \"payload-processing-e2e-migrate-387ddcaa-8596dcf594-mwxvf\" (UID: \"274bdac5-2654-4854-bdaa-f343b61c2dd2\") " pod="openshift-ingress/payload-processing-e2e-migrate-387ddcaa-8596dcf594-mwxvf"
Sep 10 02:59:52.495506 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:52.495463    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjlmk\" (UniqueName: \"kubernetes.io/projected/1e60d8eb-3c5d-4849-8125-6d1d9b037751-kube-api-access-rjlmk\") pod \"payload-pre-processing-e2e-migrate-387ddcaa-55ff99db6-jrxbs\" (UID: \"1e60d8eb-3c5d-4849-8125-6d1d9b037751\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-387ddcaa-55ff99db6-jrxbs"
Sep 10 02:59:52.495593 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:52.495519    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/274bdac5-2654-4854-bdaa-f343b61c2dd2-plugins-config-volume\") pod \"payload-processing-e2e-migrate-387ddcaa-8596dcf594-mwxvf\" (UID: \"274bdac5-2654-4854-bdaa-f343b61c2dd2\") " pod="openshift-ingress/payload-processing-e2e-migrate-387ddcaa-8596dcf594-mwxvf"
Sep 10 02:59:52.596954 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:52.596924    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlrs6\" (UniqueName: \"kubernetes.io/projected/274bdac5-2654-4854-bdaa-f343b61c2dd2-kube-api-access-tlrs6\") pod \"payload-processing-e2e-migrate-387ddcaa-8596dcf594-mwxvf\" (UID: \"274bdac5-2654-4854-bdaa-f343b61c2dd2\") " pod="openshift-ingress/payload-processing-e2e-migrate-387ddcaa-8596dcf594-mwxvf"
Sep 10 02:59:52.597071 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:52.596969    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjlmk\" (UniqueName: \"kubernetes.io/projected/1e60d8eb-3c5d-4849-8125-6d1d9b037751-kube-api-access-rjlmk\") pod \"payload-pre-processing-e2e-migrate-387ddcaa-55ff99db6-jrxbs\" (UID: \"1e60d8eb-3c5d-4849-8125-6d1d9b037751\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-387ddcaa-55ff99db6-jrxbs"
Sep 10 02:59:52.597071 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:52.597015    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/274bdac5-2654-4854-bdaa-f343b61c2dd2-plugins-config-volume\") pod \"payload-processing-e2e-migrate-387ddcaa-8596dcf594-mwxvf\" (UID: \"274bdac5-2654-4854-bdaa-f343b61c2dd2\") " pod="openshift-ingress/payload-processing-e2e-migrate-387ddcaa-8596dcf594-mwxvf"
Sep 10 02:59:52.597211 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:52.597099    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e60d8eb-3c5d-4849-8125-6d1d9b037751-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-387ddcaa-55ff99db6-jrxbs\" (UID: \"1e60d8eb-3c5d-4849-8125-6d1d9b037751\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-387ddcaa-55ff99db6-jrxbs"
Sep 10 02:59:52.597794 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:52.597770    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/274bdac5-2654-4854-bdaa-f343b61c2dd2-plugins-config-volume\") pod \"payload-processing-e2e-migrate-387ddcaa-8596dcf594-mwxvf\" (UID: \"274bdac5-2654-4854-bdaa-f343b61c2dd2\") " pod="openshift-ingress/payload-processing-e2e-migrate-387ddcaa-8596dcf594-mwxvf"
Sep 10 02:59:52.597913 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:52.597893    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e60d8eb-3c5d-4849-8125-6d1d9b037751-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-387ddcaa-55ff99db6-jrxbs\" (UID: \"1e60d8eb-3c5d-4849-8125-6d1d9b037751\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-387ddcaa-55ff99db6-jrxbs"
Sep 10 02:59:52.605596 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:52.605574    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlrs6\" (UniqueName: \"kubernetes.io/projected/274bdac5-2654-4854-bdaa-f343b61c2dd2-kube-api-access-tlrs6\") pod \"payload-processing-e2e-migrate-387ddcaa-8596dcf594-mwxvf\" (UID: \"274bdac5-2654-4854-bdaa-f343b61c2dd2\") " pod="openshift-ingress/payload-processing-e2e-migrate-387ddcaa-8596dcf594-mwxvf"
Sep 10 02:59:52.606027 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:52.606006    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjlmk\" (UniqueName: \"kubernetes.io/projected/1e60d8eb-3c5d-4849-8125-6d1d9b037751-kube-api-access-rjlmk\") pod \"payload-pre-processing-e2e-migrate-387ddcaa-55ff99db6-jrxbs\" (UID: \"1e60d8eb-3c5d-4849-8125-6d1d9b037751\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-387ddcaa-55ff99db6-jrxbs"
Sep 10 02:59:52.730405 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:52.730379    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-387ddcaa-55ff99db6-jrxbs"
Sep 10 02:59:52.741414 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:52.741394    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-387ddcaa-8596dcf594-mwxvf"
Sep 10 02:59:53.017255 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:53.017225    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-387ddcaa-55ff99db6-jrxbs"]
Sep 10 02:59:53.018582 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:59:53.018555    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e60d8eb_3c5d_4849_8125_6d1d9b037751.slice/crio-cc41198c9c22544d432979b8d392390453b32febd194824e797fa819328ad31d WatchSource:0}: Error finding container cc41198c9c22544d432979b8d392390453b32febd194824e797fa819328ad31d: Status 404 returned error can't find the container with id cc41198c9c22544d432979b8d392390453b32febd194824e797fa819328ad31d
Sep 10 02:59:53.060376 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:53.060357    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-387ddcaa-8596dcf594-mwxvf"]
Sep 10 02:59:53.071056 ip-10-0-128-101 kubenswrapper[2558]: W0910 02:59:53.071025    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod274bdac5_2654_4854_bdaa_f343b61c2dd2.slice/crio-696b84e77841ca0da9879f4420a482b75b66118574ac913ea614eb6d11cf82d3 WatchSource:0}: Error finding container 696b84e77841ca0da9879f4420a482b75b66118574ac913ea614eb6d11cf82d3: Status 404 returned error can't find the container with id 696b84e77841ca0da9879f4420a482b75b66118574ac913ea614eb6d11cf82d3
Sep 10 02:59:53.243704 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:53.243675    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:53.981774 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:53.981731    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-387ddcaa-8596dcf594-mwxvf" event={"ID":"274bdac5-2654-4854-bdaa-f343b61c2dd2","Type":"ContainerStarted","Data":"38394bc93e9950a280c56ea9936b667b77e9fd3f5691d78ec64487e4f0cfdc5d"}
Sep 10 02:59:53.982229 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:53.981804    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-387ddcaa-8596dcf594-mwxvf"
Sep 10 02:59:53.982229 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:53.981741    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:53.982229 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:53.981832    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-387ddcaa-8596dcf594-mwxvf" event={"ID":"274bdac5-2654-4854-bdaa-f343b61c2dd2","Type":"ContainerStarted","Data":"696b84e77841ca0da9879f4420a482b75b66118574ac913ea614eb6d11cf82d3"}
Sep 10 02:59:53.983479 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:53.983454    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-387ddcaa-55ff99db6-jrxbs" event={"ID":"1e60d8eb-3c5d-4849-8125-6d1d9b037751","Type":"ContainerStarted","Data":"92725f59b4080d6cca5a0b2dcd2857c4bd0733f04f1bbe9fe08ef0dd22ba69ff"}
Sep 10 02:59:53.983604 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:53.983497    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-387ddcaa-55ff99db6-jrxbs" event={"ID":"1e60d8eb-3c5d-4849-8125-6d1d9b037751","Type":"ContainerStarted","Data":"cc41198c9c22544d432979b8d392390453b32febd194824e797fa819328ad31d"}
Sep 10 02:59:53.983604 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:53.983543    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-387ddcaa-55ff99db6-jrxbs"
Sep 10 02:59:54.000045 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:54.000007    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-387ddcaa-8596dcf594-mwxvf" podStartSLOduration=1.9999936969999998 podStartE2EDuration="1.999993697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:59:53.997971861 +0000 UTC m=+784.373057783" watchObservedRunningTime="2026-09-10 02:59:53.999993697 +0000 UTC m=+784.375079687"
Sep 10 02:59:54.015359 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:54.015320    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-387ddcaa-55ff99db6-jrxbs" podStartSLOduration=2.015308525 podStartE2EDuration="2.015308525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:59:54.013145846 +0000 UTC m=+784.388231769" watchObservedRunningTime="2026-09-10 02:59:54.015308525 +0000 UTC m=+784.390394446"
Sep 10 02:59:54.775946 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:54.775916    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 02:59:56.029394 ip-10-0-128-101 kubenswrapper[2558]: I0910 02:59:56.029360    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:00:00.137099 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:00.137068    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ce5f9d-29816820-79v9c"]
Sep 10 03:00:00.140993 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:00.140974    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-387ddcaa-29816820-5554m"]
Sep 10 03:00:00.141127 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:00.141084    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ce5f9d-29816820-79v9c"
Sep 10 03:00:00.143209 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:00.143191    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8jjl\""
Sep 10 03:00:00.144231 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:00.144212    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-387ddcaa-29816820-5554m"
Sep 10 03:00:00.149974 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:00.149661    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ce5f9d-29816820-79v9c"]
Sep 10 03:00:00.155455 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:00.155437    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-387ddcaa-29816820-5554m"]
Sep 10 03:00:00.156214 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:00.156197    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ce5f9d-29816820-79v9c"
Sep 10 03:00:00.164350 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:00.163754    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e6e6b8-29816820-4xkmz"]
Sep 10 03:00:00.168648 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:00.168626    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e6e6b8-29816820-4xkmz"
Sep 10 03:00:00.168744 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:00.168626    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-387ddcaa-29816820-5554m"
Sep 10 03:00:00.174950 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:00.174927    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e6e6b8-29816820-4xkmz"]
Sep 10 03:00:00.205838 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:00.205815    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e6e6b8-29816820-4xkmz"
Sep 10 03:00:00.446632 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:00.446599    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ce5f9d-29816820-79v9c"]
Sep 10 03:00:00.446964 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:00:00.446939    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae36d3f_86b7_4695_9604_68ce874095d1.slice/crio-b7db198e7c63f9bdf73805e9682db334b2a82eb910cb9d9c7b10506e8eba4338 WatchSource:0}: Error finding container b7db198e7c63f9bdf73805e9682db334b2a82eb910cb9d9c7b10506e8eba4338: Status 404 returned error can't find the container with id b7db198e7c63f9bdf73805e9682db334b2a82eb910cb9d9c7b10506e8eba4338
Sep 10 03:00:00.466975 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:00.466511    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-387ddcaa-29816820-5554m"]
Sep 10 03:00:00.474970 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:00:00.474944    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a1224e4_599b_42f9_b695_e899ed0de37f.slice/crio-dd306969a13220e91b29dc3edb14ccd24ae787cd1e654098878c03d33a5a94f6 WatchSource:0}: Error finding container dd306969a13220e91b29dc3edb14ccd24ae787cd1e654098878c03d33a5a94f6: Status 404 returned error can't find the container with id dd306969a13220e91b29dc3edb14ccd24ae787cd1e654098878c03d33a5a94f6
Sep 10 03:00:00.502242 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:00.502223    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e6e6b8-29816820-4xkmz"]
Sep 10 03:00:00.503548 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:00:00.503526    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73eee88_4bcf_4c52_9676_1d6b8c3e8557.slice/crio-28cfba970e294df115ab4797ee40477c4ff9b3f5b591c300575d76c8ffa98692 WatchSource:0}: Error finding container 28cfba970e294df115ab4797ee40477c4ff9b3f5b591c300575d76c8ffa98692: Status 404 returned error can't find the container with id 28cfba970e294df115ab4797ee40477c4ff9b3f5b591c300575d76c8ffa98692
Sep 10 03:00:01.028988 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:01.028950    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-387ddcaa-29816820-5554m" event={"ID":"4a1224e4-599b-42f9-b695-e899ed0de37f","Type":"ContainerStarted","Data":"dd306969a13220e91b29dc3edb14ccd24ae787cd1e654098878c03d33a5a94f6"}
Sep 10 03:00:01.030155 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:01.030126    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ce5f9d-29816820-79v9c" event={"ID":"bae36d3f-86b7-4695-9604-68ce874095d1","Type":"ContainerStarted","Data":"b7db198e7c63f9bdf73805e9682db334b2a82eb910cb9d9c7b10506e8eba4338"}
Sep 10 03:00:01.031275 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:01.031250    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e6e6b8-29816820-4xkmz" event={"ID":"c73eee88-4bcf-4c52-9676-1d6b8c3e8557","Type":"ContainerStarted","Data":"28cfba970e294df115ab4797ee40477c4ff9b3f5b591c300575d76c8ffa98692"}
Sep 10 03:00:04.991176 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:04.991143    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-387ddcaa-8596dcf594-mwxvf"
Sep 10 03:00:04.991583 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:04.991207    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-387ddcaa-55ff99db6-jrxbs"
Sep 10 03:00:05.064909 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:05.064870    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e6e6b8-29816820-4xkmz" event={"ID":"c73eee88-4bcf-4c52-9676-1d6b8c3e8557","Type":"ContainerStarted","Data":"eecb95d629d987d38267fba13f113cd709a880466b20f54452e3fc9375f2e663"}
Sep 10 03:00:05.067342 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:05.067314    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-387ddcaa-29816820-5554m" event={"ID":"4a1224e4-599b-42f9-b695-e899ed0de37f","Type":"ContainerStarted","Data":"8771d712c5a08e126d54836e54c2c8fc732ce8d9b9015b9d56a09716809ca584"}
Sep 10 03:00:05.069121 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:05.069057    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ce5f9d-29816820-79v9c" event={"ID":"bae36d3f-86b7-4695-9604-68ce874095d1","Type":"ContainerStarted","Data":"5a7d018081a638f280292662ab479d1a49e77ffe248d0233ebb6ef7026b448f5"}
Sep 10 03:00:05.079083 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:05.079033    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e6e6b8-29816820-4xkmz" podStartSLOduration=1.661565527 podStartE2EDuration="5.079015614s" totalImagesPullingTime="3.417450087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:00:05.078524915 +0000 UTC m=+795.453610836" watchObservedRunningTime="2026-09-10 03:00:05.079015614 +0000 UTC m=+795.454101547"
Sep 10 03:00:05.092645 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:05.092607    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-387ddcaa-29816820-5554m" podStartSLOduration=1.650891197 podStartE2EDuration="5.092593716s" totalImagesPullingTime="3.441702519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:00:05.091395207 +0000 UTC m=+795.466481129" watchObservedRunningTime="2026-09-10 03:00:05.092593716 +0000 UTC m=+795.467679637"
Sep 10 03:00:05.105547 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:05.105513    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ce5f9d-29816820-79v9c" podStartSLOduration=1.636284702 podStartE2EDuration="5.105501533s" totalImagesPullingTime="3.469216831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:00:05.103870892 +0000 UTC m=+795.478956812" watchObservedRunningTime="2026-09-10 03:00:05.105501533 +0000 UTC m=+795.480587454"
Sep 10 03:00:06.076666 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:06.076634    2558 generic.go:356] "Generic (PLEG): container finished" podID="c73eee88-4bcf-4c52-9676-1d6b8c3e8557" containerID="eecb95d629d987d38267fba13f113cd709a880466b20f54452e3fc9375f2e663" exitCode=0
Sep 10 03:00:06.077094 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:06.076707    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e6e6b8-29816820-4xkmz" event={"ID":"c73eee88-4bcf-4c52-9676-1d6b8c3e8557","Type":"ContainerDied","Data":"eecb95d629d987d38267fba13f113cd709a880466b20f54452e3fc9375f2e663"}
Sep 10 03:00:06.078424 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:06.078399    2558 generic.go:356] "Generic (PLEG): container finished" podID="4a1224e4-599b-42f9-b695-e899ed0de37f" containerID="8771d712c5a08e126d54836e54c2c8fc732ce8d9b9015b9d56a09716809ca584" exitCode=0
Sep 10 03:00:06.078518 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:06.078477    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-387ddcaa-29816820-5554m" event={"ID":"4a1224e4-599b-42f9-b695-e899ed0de37f","Type":"ContainerDied","Data":"8771d712c5a08e126d54836e54c2c8fc732ce8d9b9015b9d56a09716809ca584"}
Sep 10 03:00:06.080381 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:06.080358    2558 generic.go:356] "Generic (PLEG): container finished" podID="bae36d3f-86b7-4695-9604-68ce874095d1" containerID="5a7d018081a638f280292662ab479d1a49e77ffe248d0233ebb6ef7026b448f5" exitCode=0
Sep 10 03:00:06.080480 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:06.080393    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ce5f9d-29816820-79v9c" event={"ID":"bae36d3f-86b7-4695-9604-68ce874095d1","Type":"ContainerDied","Data":"5a7d018081a638f280292662ab479d1a49e77ffe248d0233ebb6ef7026b448f5"}
Sep 10 03:00:07.244240 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:07.244193    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-387ddcaa-29816820-5554m"
Sep 10 03:00:07.297662 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:07.297642    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ce5f9d-29816820-79v9c"
Sep 10 03:00:07.300623 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:07.300592    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e6e6b8-29816820-4xkmz"
Sep 10 03:00:08.092413 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:08.092394    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ce5f9d-29816820-79v9c"
Sep 10 03:00:08.092533 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:08.092414    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ce5f9d-29816820-79v9c" event={"ID":"bae36d3f-86b7-4695-9604-68ce874095d1","Type":"ContainerDied","Data":"b7db198e7c63f9bdf73805e9682db334b2a82eb910cb9d9c7b10506e8eba4338"}
Sep 10 03:00:08.092533 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:08.092454    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7db198e7c63f9bdf73805e9682db334b2a82eb910cb9d9c7b10506e8eba4338"
Sep 10 03:00:08.093893 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:08.093871    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e6e6b8-29816820-4xkmz"
Sep 10 03:00:08.094001 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:08.093895    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e6e6b8-29816820-4xkmz" event={"ID":"c73eee88-4bcf-4c52-9676-1d6b8c3e8557","Type":"ContainerDied","Data":"28cfba970e294df115ab4797ee40477c4ff9b3f5b591c300575d76c8ffa98692"}
Sep 10 03:00:08.094001 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:08.093953    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28cfba970e294df115ab4797ee40477c4ff9b3f5b591c300575d76c8ffa98692"
Sep 10 03:00:08.095341 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:08.095322    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-387ddcaa-29816820-5554m" event={"ID":"4a1224e4-599b-42f9-b695-e899ed0de37f","Type":"ContainerDied","Data":"dd306969a13220e91b29dc3edb14ccd24ae787cd1e654098878c03d33a5a94f6"}
Sep 10 03:00:08.095341 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:08.095335    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-387ddcaa-29816820-5554m"
Sep 10 03:00:08.095506 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:08.095353    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd306969a13220e91b29dc3edb14ccd24ae787cd1e654098878c03d33a5a94f6"
Sep 10 03:00:12.702691 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:12.702615    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:00:23.079789 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:23.079749    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-387ddcaa-29816820-5554m"]
Sep 10 03:00:23.083923 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:23.083889    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-387ddcaa-29816820-5554m"]
Sep 10 03:00:23.096734 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:23.096026    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-387ddcaa-8596dcf594-mwxvf"]
Sep 10 03:00:23.096734 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:23.096283    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-387ddcaa-8596dcf594-mwxvf" podUID="274bdac5-2654-4854-bdaa-f343b61c2dd2" containerName="payload-processing" containerID="cri-o://38394bc93e9950a280c56ea9936b667b77e9fd3f5691d78ec64487e4f0cfdc5d" gracePeriod=30
Sep 10 03:00:23.103958 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:23.103936    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-387ddcaa-55ff99db6-jrxbs"]
Sep 10 03:00:23.104343 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:23.104314    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-387ddcaa-55ff99db6-jrxbs" podUID="1e60d8eb-3c5d-4849-8125-6d1d9b037751" containerName="payload-pre-processing" containerID="cri-o://92725f59b4080d6cca5a0b2dcd2857c4bd0733f04f1bbe9fe08ef0dd22ba69ff" gracePeriod=30
Sep 10 03:00:23.170351 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:23.170301    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:00:23.231556 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:23.231525    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:00:23.282570 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:23.282542    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:00:23.401878 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:23.401856    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-387ddcaa-8596dcf594-mwxvf"
Sep 10 03:00:23.404648 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:23.404629    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-387ddcaa-55ff99db6-jrxbs"
Sep 10 03:00:23.407770 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:23.407748    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:00:23.546811 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:23.546788    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjlmk\" (UniqueName: \"kubernetes.io/projected/1e60d8eb-3c5d-4849-8125-6d1d9b037751-kube-api-access-rjlmk\") pod \"1e60d8eb-3c5d-4849-8125-6d1d9b037751\" (UID: \"1e60d8eb-3c5d-4849-8125-6d1d9b037751\") "
Sep 10 03:00:23.546931 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:23.546841    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e60d8eb-3c5d-4849-8125-6d1d9b037751-plugins-config-volume\") pod \"1e60d8eb-3c5d-4849-8125-6d1d9b037751\" (UID: \"1e60d8eb-3c5d-4849-8125-6d1d9b037751\") "
Sep 10 03:00:23.546931 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:23.546880    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/274bdac5-2654-4854-bdaa-f343b61c2dd2-plugins-config-volume\") pod \"274bdac5-2654-4854-bdaa-f343b61c2dd2\" (UID: \"274bdac5-2654-4854-bdaa-f343b61c2dd2\") "
Sep 10 03:00:23.546931 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:23.546919    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlrs6\" (UniqueName: \"kubernetes.io/projected/274bdac5-2654-4854-bdaa-f343b61c2dd2-kube-api-access-tlrs6\") pod \"274bdac5-2654-4854-bdaa-f343b61c2dd2\" (UID: \"274bdac5-2654-4854-bdaa-f343b61c2dd2\") "
Sep 10 03:00:23.547277 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:23.547249    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e60d8eb-3c5d-4849-8125-6d1d9b037751-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1e60d8eb-3c5d-4849-8125-6d1d9b037751" (UID: "1e60d8eb-3c5d-4849-8125-6d1d9b037751"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:00:23.547277 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:23.547262    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/274bdac5-2654-4854-bdaa-f343b61c2dd2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "274bdac5-2654-4854-bdaa-f343b61c2dd2" (UID: "274bdac5-2654-4854-bdaa-f343b61c2dd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:00:23.548919 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:23.548897    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/274bdac5-2654-4854-bdaa-f343b61c2dd2-kube-api-access-tlrs6" (OuterVolumeSpecName: "kube-api-access-tlrs6") pod "274bdac5-2654-4854-bdaa-f343b61c2dd2" (UID: "274bdac5-2654-4854-bdaa-f343b61c2dd2"). InnerVolumeSpecName "kube-api-access-tlrs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:00:23.548986 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:23.548972    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e60d8eb-3c5d-4849-8125-6d1d9b037751-kube-api-access-rjlmk" (OuterVolumeSpecName: "kube-api-access-rjlmk") pod "1e60d8eb-3c5d-4849-8125-6d1d9b037751" (UID: "1e60d8eb-3c5d-4849-8125-6d1d9b037751"). InnerVolumeSpecName "kube-api-access-rjlmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:00:23.647505 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:23.647442    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e60d8eb-3c5d-4849-8125-6d1d9b037751-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:00:23.647505 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:23.647474    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/274bdac5-2654-4854-bdaa-f343b61c2dd2-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:00:23.647505 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:23.647484    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlrs6\" (UniqueName: \"kubernetes.io/projected/274bdac5-2654-4854-bdaa-f343b61c2dd2-kube-api-access-tlrs6\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:00:23.647505 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:23.647493    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjlmk\" (UniqueName: \"kubernetes.io/projected/1e60d8eb-3c5d-4849-8125-6d1d9b037751-kube-api-access-rjlmk\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:00:23.851940 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:23.851914    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:00:24.069169 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:24.069134    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a1224e4-599b-42f9-b695-e899ed0de37f" path="/var/lib/kubelet/pods/4a1224e4-599b-42f9-b695-e899ed0de37f/volumes"
Sep 10 03:00:24.182854 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:24.182816    2558 generic.go:356] "Generic (PLEG): container finished" podID="1e60d8eb-3c5d-4849-8125-6d1d9b037751" containerID="92725f59b4080d6cca5a0b2dcd2857c4bd0733f04f1bbe9fe08ef0dd22ba69ff" exitCode=0
Sep 10 03:00:24.183309 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:24.182865    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-387ddcaa-55ff99db6-jrxbs"
Sep 10 03:00:24.183309 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:24.182897    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-387ddcaa-55ff99db6-jrxbs" event={"ID":"1e60d8eb-3c5d-4849-8125-6d1d9b037751","Type":"ContainerDied","Data":"92725f59b4080d6cca5a0b2dcd2857c4bd0733f04f1bbe9fe08ef0dd22ba69ff"}
Sep 10 03:00:24.183309 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:24.182945    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-387ddcaa-55ff99db6-jrxbs" event={"ID":"1e60d8eb-3c5d-4849-8125-6d1d9b037751","Type":"ContainerDied","Data":"cc41198c9c22544d432979b8d392390453b32febd194824e797fa819328ad31d"}
Sep 10 03:00:24.183309 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:24.182970    2558 scope.go:117] "RemoveContainer" containerID="92725f59b4080d6cca5a0b2dcd2857c4bd0733f04f1bbe9fe08ef0dd22ba69ff"
Sep 10 03:00:24.184594 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:24.184569    2558 generic.go:356] "Generic (PLEG): container finished" podID="274bdac5-2654-4854-bdaa-f343b61c2dd2" containerID="38394bc93e9950a280c56ea9936b667b77e9fd3f5691d78ec64487e4f0cfdc5d" exitCode=0
Sep 10 03:00:24.184673 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:24.184602    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-387ddcaa-8596dcf594-mwxvf"
Sep 10 03:00:24.184673 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:24.184624    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-387ddcaa-8596dcf594-mwxvf" event={"ID":"274bdac5-2654-4854-bdaa-f343b61c2dd2","Type":"ContainerDied","Data":"38394bc93e9950a280c56ea9936b667b77e9fd3f5691d78ec64487e4f0cfdc5d"}
Sep 10 03:00:24.184673 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:24.184660    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-387ddcaa-8596dcf594-mwxvf" event={"ID":"274bdac5-2654-4854-bdaa-f343b61c2dd2","Type":"ContainerDied","Data":"696b84e77841ca0da9879f4420a482b75b66118574ac913ea614eb6d11cf82d3"}
Sep 10 03:00:24.192340 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:24.192321    2558 scope.go:117] "RemoveContainer" containerID="92725f59b4080d6cca5a0b2dcd2857c4bd0733f04f1bbe9fe08ef0dd22ba69ff"
Sep 10 03:00:24.192622 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:00:24.192602    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92725f59b4080d6cca5a0b2dcd2857c4bd0733f04f1bbe9fe08ef0dd22ba69ff\": container with ID starting with 92725f59b4080d6cca5a0b2dcd2857c4bd0733f04f1bbe9fe08ef0dd22ba69ff not found: ID does not exist" containerID="92725f59b4080d6cca5a0b2dcd2857c4bd0733f04f1bbe9fe08ef0dd22ba69ff"
Sep 10 03:00:24.192675 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:24.192630    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92725f59b4080d6cca5a0b2dcd2857c4bd0733f04f1bbe9fe08ef0dd22ba69ff"} err="failed to get container status \"92725f59b4080d6cca5a0b2dcd2857c4bd0733f04f1bbe9fe08ef0dd22ba69ff\": rpc error: code = NotFound desc = could not find container \"92725f59b4080d6cca5a0b2dcd2857c4bd0733f04f1bbe9fe08ef0dd22ba69ff\": container with ID starting with 92725f59b4080d6cca5a0b2dcd2857c4bd0733f04f1bbe9fe08ef0dd22ba69ff not found: ID does not exist"
Sep 10 03:00:24.192675 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:24.192646    2558 scope.go:117] "RemoveContainer" containerID="38394bc93e9950a280c56ea9936b667b77e9fd3f5691d78ec64487e4f0cfdc5d"
Sep 10 03:00:24.201988 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:24.201959    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-387ddcaa-8596dcf594-mwxvf"]
Sep 10 03:00:24.203433 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:24.203416    2558 scope.go:117] "RemoveContainer" containerID="38394bc93e9950a280c56ea9936b667b77e9fd3f5691d78ec64487e4f0cfdc5d"
Sep 10 03:00:24.203742 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:00:24.203720    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38394bc93e9950a280c56ea9936b667b77e9fd3f5691d78ec64487e4f0cfdc5d\": container with ID starting with 38394bc93e9950a280c56ea9936b667b77e9fd3f5691d78ec64487e4f0cfdc5d not found: ID does not exist" containerID="38394bc93e9950a280c56ea9936b667b77e9fd3f5691d78ec64487e4f0cfdc5d"
Sep 10 03:00:24.203814 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:24.203748    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38394bc93e9950a280c56ea9936b667b77e9fd3f5691d78ec64487e4f0cfdc5d"} err="failed to get container status \"38394bc93e9950a280c56ea9936b667b77e9fd3f5691d78ec64487e4f0cfdc5d\": rpc error: code = NotFound desc = could not find container \"38394bc93e9950a280c56ea9936b667b77e9fd3f5691d78ec64487e4f0cfdc5d\": container with ID starting with 38394bc93e9950a280c56ea9936b667b77e9fd3f5691d78ec64487e4f0cfdc5d not found: ID does not exist"
Sep 10 03:00:24.205265 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:24.205247    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-387ddcaa-8596dcf594-mwxvf"]
Sep 10 03:00:24.212543 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:24.212522    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-387ddcaa-55ff99db6-jrxbs"]
Sep 10 03:00:24.214467 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:24.214449    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-387ddcaa-55ff99db6-jrxbs"]
Sep 10 03:00:24.417297 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:24.417230    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:00:24.473323 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:24.473298    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:00:26.069734 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:26.069688    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e60d8eb-3c5d-4849-8125-6d1d9b037751" path="/var/lib/kubelet/pods/1e60d8eb-3c5d-4849-8125-6d1d9b037751/volumes"
Sep 10 03:00:26.070207 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:26.070018    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="274bdac5-2654-4854-bdaa-f343b61c2dd2" path="/var/lib/kubelet/pods/274bdac5-2654-4854-bdaa-f343b61c2dd2/volumes"
Sep 10 03:00:26.786441 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:26.786404    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:00:26.900604 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:26.900576    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:00:27.613511 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.613441    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw"]
Sep 10 03:00:27.613869 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.613818    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c73eee88-4bcf-4c52-9676-1d6b8c3e8557" containerName="cleanup"
Sep 10 03:00:27.613869 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.613828    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c73eee88-4bcf-4c52-9676-1d6b8c3e8557" containerName="cleanup"
Sep 10 03:00:27.613869 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.613837    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a1224e4-599b-42f9-b695-e899ed0de37f" containerName="cleanup"
Sep 10 03:00:27.613869 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.613844    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a1224e4-599b-42f9-b695-e899ed0de37f" containerName="cleanup"
Sep 10 03:00:27.613869 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.613854    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bae36d3f-86b7-4695-9604-68ce874095d1" containerName="cleanup"
Sep 10 03:00:27.613869 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.613859    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bae36d3f-86b7-4695-9604-68ce874095d1" containerName="cleanup"
Sep 10 03:00:27.614086 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.613873    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e60d8eb-3c5d-4849-8125-6d1d9b037751" containerName="payload-pre-processing"
Sep 10 03:00:27.614086 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.613879    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e60d8eb-3c5d-4849-8125-6d1d9b037751" containerName="payload-pre-processing"
Sep 10 03:00:27.614086 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.613885    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="274bdac5-2654-4854-bdaa-f343b61c2dd2" containerName="payload-processing"
Sep 10 03:00:27.614086 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.613891    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="274bdac5-2654-4854-bdaa-f343b61c2dd2" containerName="payload-processing"
Sep 10 03:00:27.614086 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.613952    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bae36d3f-86b7-4695-9604-68ce874095d1" containerName="cleanup"
Sep 10 03:00:27.614086 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.613962    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e60d8eb-3c5d-4849-8125-6d1d9b037751" containerName="payload-pre-processing"
Sep 10 03:00:27.614086 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.613969    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="274bdac5-2654-4854-bdaa-f343b61c2dd2" containerName="payload-processing"
Sep 10 03:00:27.614086 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.613975    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c73eee88-4bcf-4c52-9676-1d6b8c3e8557" containerName="cleanup"
Sep 10 03:00:27.614086 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.613980    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a1224e4-599b-42f9-b695-e899ed0de37f" containerName="cleanup"
Sep 10 03:00:27.618240 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.618224    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw"
Sep 10 03:00:27.621378 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.621353    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-e6e6b8\"/\"openshift-service-ca.crt\""
Sep 10 03:00:27.621378 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.621367    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e6e6b8\"/\"simulator-w4-e6e6b8-kserve-self-signed-certs\""
Sep 10 03:00:27.621564 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.621353    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e6e6b8\"/\"default-dockercfg-vbcwd\""
Sep 10 03:00:27.621564 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.621397    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-e6e6b8\"/\"kube-root-ca.crt\""
Sep 10 03:00:27.628666 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.628644    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw"]
Sep 10 03:00:27.681787 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.681759    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19549650-bd12-4ea4-83b6-30df64ab2587-tmp-dir\") pod \"simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw\" (UID: \"19549650-bd12-4ea4-83b6-30df64ab2587\") " pod="e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw"
Sep 10 03:00:27.681919 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.681797    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gp7l\" (UniqueName: \"kubernetes.io/projected/19549650-bd12-4ea4-83b6-30df64ab2587-kube-api-access-8gp7l\") pod \"simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw\" (UID: \"19549650-bd12-4ea4-83b6-30df64ab2587\") " pod="e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw"
Sep 10 03:00:27.681919 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.681826    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19549650-bd12-4ea4-83b6-30df64ab2587-dshm\") pod \"simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw\" (UID: \"19549650-bd12-4ea4-83b6-30df64ab2587\") " pod="e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw"
Sep 10 03:00:27.681919 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.681915    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19549650-bd12-4ea4-83b6-30df64ab2587-model-cache\") pod \"simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw\" (UID: \"19549650-bd12-4ea4-83b6-30df64ab2587\") " pod="e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw"
Sep 10 03:00:27.682037 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.681945    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19549650-bd12-4ea4-83b6-30df64ab2587-home\") pod \"simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw\" (UID: \"19549650-bd12-4ea4-83b6-30df64ab2587\") " pod="e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw"
Sep 10 03:00:27.682037 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.681962    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19549650-bd12-4ea4-83b6-30df64ab2587-tls-certs\") pod \"simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw\" (UID: \"19549650-bd12-4ea4-83b6-30df64ab2587\") " pod="e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw"
Sep 10 03:00:27.782784 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.782748    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gp7l\" (UniqueName: \"kubernetes.io/projected/19549650-bd12-4ea4-83b6-30df64ab2587-kube-api-access-8gp7l\") pod \"simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw\" (UID: \"19549650-bd12-4ea4-83b6-30df64ab2587\") " pod="e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw"
Sep 10 03:00:27.782942 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.782811    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19549650-bd12-4ea4-83b6-30df64ab2587-dshm\") pod \"simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw\" (UID: \"19549650-bd12-4ea4-83b6-30df64ab2587\") " pod="e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw"
Sep 10 03:00:27.782942 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.782877    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19549650-bd12-4ea4-83b6-30df64ab2587-model-cache\") pod \"simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw\" (UID: \"19549650-bd12-4ea4-83b6-30df64ab2587\") " pod="e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw"
Sep 10 03:00:27.782942 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.782920    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19549650-bd12-4ea4-83b6-30df64ab2587-home\") pod \"simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw\" (UID: \"19549650-bd12-4ea4-83b6-30df64ab2587\") " pod="e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw"
Sep 10 03:00:27.783172 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.782946    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19549650-bd12-4ea4-83b6-30df64ab2587-tls-certs\") pod \"simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw\" (UID: \"19549650-bd12-4ea4-83b6-30df64ab2587\") " pod="e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw"
Sep 10 03:00:27.783172 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.782987    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19549650-bd12-4ea4-83b6-30df64ab2587-tmp-dir\") pod \"simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw\" (UID: \"19549650-bd12-4ea4-83b6-30df64ab2587\") " pod="e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw"
Sep 10 03:00:27.783508 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.783476    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19549650-bd12-4ea4-83b6-30df64ab2587-tmp-dir\") pod \"simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw\" (UID: \"19549650-bd12-4ea4-83b6-30df64ab2587\") " pod="e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw"
Sep 10 03:00:27.783611 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.783506    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19549650-bd12-4ea4-83b6-30df64ab2587-model-cache\") pod \"simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw\" (UID: \"19549650-bd12-4ea4-83b6-30df64ab2587\") " pod="e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw"
Sep 10 03:00:27.783832 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.783814    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19549650-bd12-4ea4-83b6-30df64ab2587-home\") pod \"simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw\" (UID: \"19549650-bd12-4ea4-83b6-30df64ab2587\") " pod="e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw"
Sep 10 03:00:27.785658 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.785635    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19549650-bd12-4ea4-83b6-30df64ab2587-dshm\") pod \"simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw\" (UID: \"19549650-bd12-4ea4-83b6-30df64ab2587\") " pod="e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw"
Sep 10 03:00:27.785871 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.785854    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19549650-bd12-4ea4-83b6-30df64ab2587-tls-certs\") pod \"simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw\" (UID: \"19549650-bd12-4ea4-83b6-30df64ab2587\") " pod="e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw"
Sep 10 03:00:27.791278 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.791253    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gp7l\" (UniqueName: \"kubernetes.io/projected/19549650-bd12-4ea4-83b6-30df64ab2587-kube-api-access-8gp7l\") pod \"simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw\" (UID: \"19549650-bd12-4ea4-83b6-30df64ab2587\") " pod="e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw"
Sep 10 03:00:27.813356 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.813333    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:00:27.929957 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.929897    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw"
Sep 10 03:00:27.994282 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:27.994237    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:00:28.208536 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:28.208512    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw"]
Sep 10 03:00:28.210136 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:00:28.210084    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19549650_bd12_4ea4_83b6_30df64ab2587.slice/crio-1064179d3b86a6edf9ef9147eff782db1eec36c8ad5a198aeff017d4f188828c WatchSource:0}: Error finding container 1064179d3b86a6edf9ef9147eff782db1eec36c8ad5a198aeff017d4f188828c: Status 404 returned error can't find the container with id 1064179d3b86a6edf9ef9147eff782db1eec36c8ad5a198aeff017d4f188828c
Sep 10 03:00:28.297671 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:28.297641    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:00:28.398442 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:28.398408    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:00:28.562145 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:28.562046    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:00:28.643777 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:28.643741    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:00:28.690616 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:28.690583    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-30dae5eb-674555df7c-lv2q2"]
Sep 10 03:00:28.694154 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:28.694132    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-30dae5eb-674555df7c-lv2q2"
Sep 10 03:00:28.696313 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:28.696289    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-30dae5eb\""
Sep 10 03:00:28.696410 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:28.696290    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-30dae5eb-dockercfg-2n7jc\""
Sep 10 03:00:28.704809 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:28.704785    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-30dae5eb-674555df7c-lv2q2"]
Sep 10 03:00:28.789633 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:28.789599    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqpvs\" (UniqueName: \"kubernetes.io/projected/afcce4db-e7ac-45cb-bc92-b1c0956cca56-kube-api-access-lqpvs\") pod \"payload-processing-e2e-ait-30dae5eb-674555df7c-lv2q2\" (UID: \"afcce4db-e7ac-45cb-bc92-b1c0956cca56\") " pod="openshift-ingress/payload-processing-e2e-ait-30dae5eb-674555df7c-lv2q2"
Sep 10 03:00:28.789633 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:28.789635    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afcce4db-e7ac-45cb-bc92-b1c0956cca56-plugins-config-volume\") pod \"payload-processing-e2e-ait-30dae5eb-674555df7c-lv2q2\" (UID: \"afcce4db-e7ac-45cb-bc92-b1c0956cca56\") " pod="openshift-ingress/payload-processing-e2e-ait-30dae5eb-674555df7c-lv2q2"
Sep 10 03:00:28.891016 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:28.890942    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqpvs\" (UniqueName: \"kubernetes.io/projected/afcce4db-e7ac-45cb-bc92-b1c0956cca56-kube-api-access-lqpvs\") pod \"payload-processing-e2e-ait-30dae5eb-674555df7c-lv2q2\" (UID: \"afcce4db-e7ac-45cb-bc92-b1c0956cca56\") " pod="openshift-ingress/payload-processing-e2e-ait-30dae5eb-674555df7c-lv2q2"
Sep 10 03:00:28.891016 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:28.890973    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afcce4db-e7ac-45cb-bc92-b1c0956cca56-plugins-config-volume\") pod \"payload-processing-e2e-ait-30dae5eb-674555df7c-lv2q2\" (UID: \"afcce4db-e7ac-45cb-bc92-b1c0956cca56\") " pod="openshift-ingress/payload-processing-e2e-ait-30dae5eb-674555df7c-lv2q2"
Sep 10 03:00:28.891712 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:28.891694    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afcce4db-e7ac-45cb-bc92-b1c0956cca56-plugins-config-volume\") pod \"payload-processing-e2e-ait-30dae5eb-674555df7c-lv2q2\" (UID: \"afcce4db-e7ac-45cb-bc92-b1c0956cca56\") " pod="openshift-ingress/payload-processing-e2e-ait-30dae5eb-674555df7c-lv2q2"
Sep 10 03:00:28.898384 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:28.898361    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqpvs\" (UniqueName: \"kubernetes.io/projected/afcce4db-e7ac-45cb-bc92-b1c0956cca56-kube-api-access-lqpvs\") pod \"payload-processing-e2e-ait-30dae5eb-674555df7c-lv2q2\" (UID: \"afcce4db-e7ac-45cb-bc92-b1c0956cca56\") " pod="openshift-ingress/payload-processing-e2e-ait-30dae5eb-674555df7c-lv2q2"
Sep 10 03:00:29.006248 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:29.006222    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-30dae5eb-674555df7c-lv2q2"
Sep 10 03:00:29.216376 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:29.216338    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw" event={"ID":"19549650-bd12-4ea4-83b6-30df64ab2587","Type":"ContainerStarted","Data":"4e3f382c3f1323da3130efdab465987dc8916b15df1b36b7406c2956a83e6f94"}
Sep 10 03:00:29.216376 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:29.216376    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw" event={"ID":"19549650-bd12-4ea4-83b6-30df64ab2587","Type":"ContainerStarted","Data":"1064179d3b86a6edf9ef9147eff782db1eec36c8ad5a198aeff017d4f188828c"}
Sep 10 03:00:29.256786 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:29.256724    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw" podStartSLOduration=2.256706066 podStartE2EDuration="2.256706066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:00:29.254630854 +0000 UTC m=+819.629716772" watchObservedRunningTime="2026-09-10 03:00:29.256706066 +0000 UTC m=+819.631791988"
Sep 10 03:00:29.280870 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:00:29.280839    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafcce4db_e7ac_45cb_bc92_b1c0956cca56.slice/crio-ee1bb0e632b8503f1159e7637abfff79883ce05c657574a21d7c195e2f8fdd4e WatchSource:0}: Error finding container ee1bb0e632b8503f1159e7637abfff79883ce05c657574a21d7c195e2f8fdd4e: Status 404 returned error can't find the container with id ee1bb0e632b8503f1159e7637abfff79883ce05c657574a21d7c195e2f8fdd4e
Sep 10 03:00:29.288706 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:29.288684    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-30dae5eb-674555df7c-lv2q2"]
Sep 10 03:00:30.224789 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:30.224748    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-30dae5eb-674555df7c-lv2q2" event={"ID":"afcce4db-e7ac-45cb-bc92-b1c0956cca56","Type":"ContainerStarted","Data":"6aedd33a2cc379d89fc09ef92248ba247fffa839df199d4214a20dc662d109c3"}
Sep 10 03:00:30.225300 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:30.224807    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-30dae5eb-674555df7c-lv2q2" event={"ID":"afcce4db-e7ac-45cb-bc92-b1c0956cca56","Type":"ContainerStarted","Data":"ee1bb0e632b8503f1159e7637abfff79883ce05c657574a21d7c195e2f8fdd4e"}
Sep 10 03:00:30.240091 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:30.240047    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-30dae5eb-674555df7c-lv2q2" podStartSLOduration=2.240030734 podStartE2EDuration="2.240030734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:00:30.239056705 +0000 UTC m=+820.614142612" watchObservedRunningTime="2026-09-10 03:00:30.240030734 +0000 UTC m=+820.615116655"
Sep 10 03:00:31.231382 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:31.231354    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-30dae5eb-674555df7c-lv2q2"
Sep 10 03:00:37.930556 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:37.930511    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw"
Sep 10 03:00:37.930990 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:37.930570    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw"
Sep 10 03:00:37.942779 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:37.942755    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw"
Sep 10 03:00:38.282636 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:38.282613    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw"
Sep 10 03:00:42.237674 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:42.237642    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-30dae5eb-674555df7c-lv2q2"
Sep 10 03:00:42.882215 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:42.882188    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:00:44.827638 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:44.827598    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:00:52.661710 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:52.661676    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30dae5eb-s9bmg"]
Sep 10 03:00:52.665367 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:52.665350    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30dae5eb-s9bmg"
Sep 10 03:00:52.668005 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:52.667982    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8jjl\""
Sep 10 03:00:52.672414 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:52.672394    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30dae5eb-s9bmg"]
Sep 10 03:00:52.770624 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:52.770596    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0764156e-6387-4df9-92f4-b639dfa6273e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-30dae5eb-s9bmg\" (UID: \"0764156e-6387-4df9-92f4-b639dfa6273e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30dae5eb-s9bmg"
Sep 10 03:00:52.871200 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:52.871174    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0764156e-6387-4df9-92f4-b639dfa6273e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-30dae5eb-s9bmg\" (UID: \"0764156e-6387-4df9-92f4-b639dfa6273e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30dae5eb-s9bmg"
Sep 10 03:00:52.871719 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:52.871699    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0764156e-6387-4df9-92f4-b639dfa6273e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-30dae5eb-s9bmg\" (UID: \"0764156e-6387-4df9-92f4-b639dfa6273e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30dae5eb-s9bmg"
Sep 10 03:00:52.977166 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:52.977138    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30dae5eb-s9bmg"
Sep 10 03:00:53.249901 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:53.249877    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30dae5eb-s9bmg"]
Sep 10 03:00:53.251336 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:00:53.251305    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0764156e_6387_4df9_92f4_b639dfa6273e.slice/crio-d17c96c0cf1bd10fd916ff5158f542b1872ed19f2bad108418e8fc1da1b1324e WatchSource:0}: Error finding container d17c96c0cf1bd10fd916ff5158f542b1872ed19f2bad108418e8fc1da1b1324e: Status 404 returned error can't find the container with id d17c96c0cf1bd10fd916ff5158f542b1872ed19f2bad108418e8fc1da1b1324e
Sep 10 03:00:53.358458 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:53.358416    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30dae5eb-s9bmg" event={"ID":"0764156e-6387-4df9-92f4-b639dfa6273e","Type":"ContainerStarted","Data":"d17c96c0cf1bd10fd916ff5158f542b1872ed19f2bad108418e8fc1da1b1324e"}
Sep 10 03:00:54.367325 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:54.367285    2558 generic.go:356] "Generic (PLEG): container finished" podID="0764156e-6387-4df9-92f4-b639dfa6273e" containerID="47dd7c745f067fa436d78d79cdc6101557c1a72f5bf10037c7833506198555f7" exitCode=0
Sep 10 03:00:54.367720 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:54.367375    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30dae5eb-s9bmg" event={"ID":"0764156e-6387-4df9-92f4-b639dfa6273e","Type":"ContainerDied","Data":"47dd7c745f067fa436d78d79cdc6101557c1a72f5bf10037c7833506198555f7"}
Sep 10 03:00:55.510777 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:55.510753    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30dae5eb-s9bmg"
Sep 10 03:00:55.594078 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:55.594042    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0764156e-6387-4df9-92f4-b639dfa6273e-maas-api-service-ca\") pod \"0764156e-6387-4df9-92f4-b639dfa6273e\" (UID: \"0764156e-6387-4df9-92f4-b639dfa6273e\") "
Sep 10 03:00:55.594402 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:55.594378    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0764156e-6387-4df9-92f4-b639dfa6273e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0764156e-6387-4df9-92f4-b639dfa6273e" (UID: "0764156e-6387-4df9-92f4-b639dfa6273e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:00:55.695125 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:55.695035    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0764156e-6387-4df9-92f4-b639dfa6273e-maas-api-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:00:56.381030 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:56.380953    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30dae5eb-s9bmg"
Sep 10 03:00:56.381181 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:56.380961    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30dae5eb-s9bmg" event={"ID":"0764156e-6387-4df9-92f4-b639dfa6273e","Type":"ContainerDied","Data":"d17c96c0cf1bd10fd916ff5158f542b1872ed19f2bad108418e8fc1da1b1324e"}
Sep 10 03:00:56.381181 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:56.381093    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d17c96c0cf1bd10fd916ff5158f542b1872ed19f2bad108418e8fc1da1b1324e"
Sep 10 03:00:58.587668 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:58.587631    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:00:58.616438 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:58.616409    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-30dae5eb-674555df7c-lv2q2"]
Sep 10 03:00:58.616785 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:58.616640    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-30dae5eb-674555df7c-lv2q2" podUID="afcce4db-e7ac-45cb-bc92-b1c0956cca56" containerName="payload-processing" containerID="cri-o://6aedd33a2cc379d89fc09ef92248ba247fffa839df199d4214a20dc662d109c3" gracePeriod=30
Sep 10 03:00:58.687237 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:58.687208    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:00:58.774163 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:58.774129    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30dae5eb-s9bmg"]
Sep 10 03:00:58.775814 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:58.775783    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:00:58.776606 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:58.776585    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30dae5eb-s9bmg"]
Sep 10 03:00:58.839041 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:58.838961    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:00:58.877496 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:58.877474    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-30dae5eb-674555df7c-lv2q2"
Sep 10 03:00:58.919939 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:58.919914    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqpvs\" (UniqueName: \"kubernetes.io/projected/afcce4db-e7ac-45cb-bc92-b1c0956cca56-kube-api-access-lqpvs\") pod \"afcce4db-e7ac-45cb-bc92-b1c0956cca56\" (UID: \"afcce4db-e7ac-45cb-bc92-b1c0956cca56\") "
Sep 10 03:00:58.920046 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:58.919959    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afcce4db-e7ac-45cb-bc92-b1c0956cca56-plugins-config-volume\") pod \"afcce4db-e7ac-45cb-bc92-b1c0956cca56\" (UID: \"afcce4db-e7ac-45cb-bc92-b1c0956cca56\") "
Sep 10 03:00:58.920337 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:58.920310    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afcce4db-e7ac-45cb-bc92-b1c0956cca56-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "afcce4db-e7ac-45cb-bc92-b1c0956cca56" (UID: "afcce4db-e7ac-45cb-bc92-b1c0956cca56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:00:58.921897 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:58.921876    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afcce4db-e7ac-45cb-bc92-b1c0956cca56-kube-api-access-lqpvs" (OuterVolumeSpecName: "kube-api-access-lqpvs") pod "afcce4db-e7ac-45cb-bc92-b1c0956cca56" (UID: "afcce4db-e7ac-45cb-bc92-b1c0956cca56"). InnerVolumeSpecName "kube-api-access-lqpvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:00:59.020853 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:59.020828    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqpvs\" (UniqueName: \"kubernetes.io/projected/afcce4db-e7ac-45cb-bc92-b1c0956cca56-kube-api-access-lqpvs\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:00:59.020942 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:59.020856    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afcce4db-e7ac-45cb-bc92-b1c0956cca56-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:00:59.399742 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:59.399709    2558 generic.go:356] "Generic (PLEG): container finished" podID="afcce4db-e7ac-45cb-bc92-b1c0956cca56" containerID="6aedd33a2cc379d89fc09ef92248ba247fffa839df199d4214a20dc662d109c3" exitCode=0
Sep 10 03:00:59.399887 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:59.399768    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-30dae5eb-674555df7c-lv2q2"
Sep 10 03:00:59.399887 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:59.399789    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-30dae5eb-674555df7c-lv2q2" event={"ID":"afcce4db-e7ac-45cb-bc92-b1c0956cca56","Type":"ContainerDied","Data":"6aedd33a2cc379d89fc09ef92248ba247fffa839df199d4214a20dc662d109c3"}
Sep 10 03:00:59.399887 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:59.399863    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-30dae5eb-674555df7c-lv2q2" event={"ID":"afcce4db-e7ac-45cb-bc92-b1c0956cca56","Type":"ContainerDied","Data":"ee1bb0e632b8503f1159e7637abfff79883ce05c657574a21d7c195e2f8fdd4e"}
Sep 10 03:00:59.399887 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:59.399885    2558 scope.go:117] "RemoveContainer" containerID="6aedd33a2cc379d89fc09ef92248ba247fffa839df199d4214a20dc662d109c3"
Sep 10 03:00:59.409791 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:59.409776    2558 scope.go:117] "RemoveContainer" containerID="6aedd33a2cc379d89fc09ef92248ba247fffa839df199d4214a20dc662d109c3"
Sep 10 03:00:59.410059 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:00:59.410041    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aedd33a2cc379d89fc09ef92248ba247fffa839df199d4214a20dc662d109c3\": container with ID starting with 6aedd33a2cc379d89fc09ef92248ba247fffa839df199d4214a20dc662d109c3 not found: ID does not exist" containerID="6aedd33a2cc379d89fc09ef92248ba247fffa839df199d4214a20dc662d109c3"
Sep 10 03:00:59.410125 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:59.410068    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aedd33a2cc379d89fc09ef92248ba247fffa839df199d4214a20dc662d109c3"} err="failed to get container status \"6aedd33a2cc379d89fc09ef92248ba247fffa839df199d4214a20dc662d109c3\": rpc error: code = NotFound desc = could not find container \"6aedd33a2cc379d89fc09ef92248ba247fffa839df199d4214a20dc662d109c3\": container with ID starting with 6aedd33a2cc379d89fc09ef92248ba247fffa839df199d4214a20dc662d109c3 not found: ID does not exist"
Sep 10 03:00:59.421814 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:59.421794    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-30dae5eb-674555df7c-lv2q2"]
Sep 10 03:00:59.424317 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:00:59.424296    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-30dae5eb-674555df7c-lv2q2"]
Sep 10 03:01:00.017700 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:00.017660    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:00.068892 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:00.068857    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0764156e-6387-4df9-92f4-b639dfa6273e" path="/var/lib/kubelet/pods/0764156e-6387-4df9-92f4-b639dfa6273e/volumes"
Sep 10 03:01:00.069296 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:00.069274    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afcce4db-e7ac-45cb-bc92-b1c0956cca56" path="/var/lib/kubelet/pods/afcce4db-e7ac-45cb-bc92-b1c0956cca56/volumes"
Sep 10 03:01:00.481835 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:00.481810    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:00.792047 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:00.791965    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:01.540683 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:01.540654    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:03.014694 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:03.014663    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:03.107792 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:03.107757    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:04.358264 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:04.358233    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:04.458116 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:04.458074    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:04.625379 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:04.625301    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:04.723799 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:04.723759    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:04.724949 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:04.724931    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-08482c4c-679998bcb6-c64z5"]
Sep 10 03:01:04.725359 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:04.725345    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0764156e-6387-4df9-92f4-b639dfa6273e" containerName="revoke-keys"
Sep 10 03:01:04.725405 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:04.725361    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="0764156e-6387-4df9-92f4-b639dfa6273e" containerName="revoke-keys"
Sep 10 03:01:04.725405 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:04.725378    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afcce4db-e7ac-45cb-bc92-b1c0956cca56" containerName="payload-processing"
Sep 10 03:01:04.725405 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:04.725383    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="afcce4db-e7ac-45cb-bc92-b1c0956cca56" containerName="payload-processing"
Sep 10 03:01:04.725501 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:04.725439    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="afcce4db-e7ac-45cb-bc92-b1c0956cca56" containerName="payload-processing"
Sep 10 03:01:04.725501 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:04.725447    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="0764156e-6387-4df9-92f4-b639dfa6273e" containerName="revoke-keys"
Sep 10 03:01:04.728595 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:04.728580    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-08482c4c-679998bcb6-c64z5"
Sep 10 03:01:04.730723 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:04.730704    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-08482c4c-dockercfg-ftbdz\""
Sep 10 03:01:04.731147 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:04.731130    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-08482c4c\""
Sep 10 03:01:04.740040 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:04.740020    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-08482c4c-679998bcb6-c64z5"]
Sep 10 03:01:04.747786 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:04.747762    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-08482c4c-5cbcb75765-r6h2h"]
Sep 10 03:01:04.750349 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:04.750329    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-08482c4c-5cbcb75765-r6h2h"
Sep 10 03:01:04.763927 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:04.763905    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-08482c4c-5cbcb75765-r6h2h"]
Sep 10 03:01:04.861219 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:04.861189    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9238a60b-2b10-433e-bec5-10cf7d0eb3dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-08482c4c-679998bcb6-c64z5\" (UID: \"9238a60b-2b10-433e-bec5-10cf7d0eb3dd\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-08482c4c-679998bcb6-c64z5"
Sep 10 03:01:04.861386 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:04.861240    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c28261-58d6-431c-9a16-b7d88640bed0-plugins-config-volume\") pod \"payload-processing-e2e-derive-08482c4c-5cbcb75765-r6h2h\" (UID: \"84c28261-58d6-431c-9a16-b7d88640bed0\") " pod="openshift-ingress/payload-processing-e2e-derive-08482c4c-5cbcb75765-r6h2h"
Sep 10 03:01:04.861386 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:04.861346    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmqx5\" (UniqueName: \"kubernetes.io/projected/84c28261-58d6-431c-9a16-b7d88640bed0-kube-api-access-hmqx5\") pod \"payload-processing-e2e-derive-08482c4c-5cbcb75765-r6h2h\" (UID: \"84c28261-58d6-431c-9a16-b7d88640bed0\") " pod="openshift-ingress/payload-processing-e2e-derive-08482c4c-5cbcb75765-r6h2h"
Sep 10 03:01:04.861505 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:04.861431    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sllzp\" (UniqueName: \"kubernetes.io/projected/9238a60b-2b10-433e-bec5-10cf7d0eb3dd-kube-api-access-sllzp\") pod \"payload-pre-processing-e2e-derive-08482c4c-679998bcb6-c64z5\" (UID: \"9238a60b-2b10-433e-bec5-10cf7d0eb3dd\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-08482c4c-679998bcb6-c64z5"
Sep 10 03:01:04.962869 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:04.962834    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sllzp\" (UniqueName: \"kubernetes.io/projected/9238a60b-2b10-433e-bec5-10cf7d0eb3dd-kube-api-access-sllzp\") pod \"payload-pre-processing-e2e-derive-08482c4c-679998bcb6-c64z5\" (UID: \"9238a60b-2b10-433e-bec5-10cf7d0eb3dd\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-08482c4c-679998bcb6-c64z5"
Sep 10 03:01:04.963011 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:04.962910    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9238a60b-2b10-433e-bec5-10cf7d0eb3dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-08482c4c-679998bcb6-c64z5\" (UID: \"9238a60b-2b10-433e-bec5-10cf7d0eb3dd\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-08482c4c-679998bcb6-c64z5"
Sep 10 03:01:04.963011 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:04.962945    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c28261-58d6-431c-9a16-b7d88640bed0-plugins-config-volume\") pod \"payload-processing-e2e-derive-08482c4c-5cbcb75765-r6h2h\" (UID: \"84c28261-58d6-431c-9a16-b7d88640bed0\") " pod="openshift-ingress/payload-processing-e2e-derive-08482c4c-5cbcb75765-r6h2h"
Sep 10 03:01:04.963011 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:04.962997    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmqx5\" (UniqueName: \"kubernetes.io/projected/84c28261-58d6-431c-9a16-b7d88640bed0-kube-api-access-hmqx5\") pod \"payload-processing-e2e-derive-08482c4c-5cbcb75765-r6h2h\" (UID: \"84c28261-58d6-431c-9a16-b7d88640bed0\") " pod="openshift-ingress/payload-processing-e2e-derive-08482c4c-5cbcb75765-r6h2h"
Sep 10 03:01:04.963633 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:04.963610    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9238a60b-2b10-433e-bec5-10cf7d0eb3dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-08482c4c-679998bcb6-c64z5\" (UID: \"9238a60b-2b10-433e-bec5-10cf7d0eb3dd\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-08482c4c-679998bcb6-c64z5"
Sep 10 03:01:04.963697 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:04.963654    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c28261-58d6-431c-9a16-b7d88640bed0-plugins-config-volume\") pod \"payload-processing-e2e-derive-08482c4c-5cbcb75765-r6h2h\" (UID: \"84c28261-58d6-431c-9a16-b7d88640bed0\") " pod="openshift-ingress/payload-processing-e2e-derive-08482c4c-5cbcb75765-r6h2h"
Sep 10 03:01:04.971686 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:04.971659    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sllzp\" (UniqueName: \"kubernetes.io/projected/9238a60b-2b10-433e-bec5-10cf7d0eb3dd-kube-api-access-sllzp\") pod \"payload-pre-processing-e2e-derive-08482c4c-679998bcb6-c64z5\" (UID: \"9238a60b-2b10-433e-bec5-10cf7d0eb3dd\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-08482c4c-679998bcb6-c64z5"
Sep 10 03:01:04.971795 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:04.971687    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmqx5\" (UniqueName: \"kubernetes.io/projected/84c28261-58d6-431c-9a16-b7d88640bed0-kube-api-access-hmqx5\") pod \"payload-processing-e2e-derive-08482c4c-5cbcb75765-r6h2h\" (UID: \"84c28261-58d6-431c-9a16-b7d88640bed0\") " pod="openshift-ingress/payload-processing-e2e-derive-08482c4c-5cbcb75765-r6h2h"
Sep 10 03:01:05.039595 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:05.039565    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-08482c4c-679998bcb6-c64z5"
Sep 10 03:01:05.061833 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:05.061811    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-08482c4c-5cbcb75765-r6h2h"
Sep 10 03:01:05.320987 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:05.320964    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-08482c4c-679998bcb6-c64z5"]
Sep 10 03:01:05.322744 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:01:05.322721    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9238a60b_2b10_433e_bec5_10cf7d0eb3dd.slice/crio-9848a3c01a58f89f2c0d130d2da075a25e0f58464bfdd11d6067f4a3ae6e8cb3 WatchSource:0}: Error finding container 9848a3c01a58f89f2c0d130d2da075a25e0f58464bfdd11d6067f4a3ae6e8cb3: Status 404 returned error can't find the container with id 9848a3c01a58f89f2c0d130d2da075a25e0f58464bfdd11d6067f4a3ae6e8cb3
Sep 10 03:01:05.346897 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:05.346877    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-08482c4c-5cbcb75765-r6h2h"]
Sep 10 03:01:05.359422 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:01:05.359398    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c28261_58d6_431c_9a16_b7d88640bed0.slice/crio-02f16648eacc7094caf92a0175618f832f2e3b869a71b514870efffb8e2375c2 WatchSource:0}: Error finding container 02f16648eacc7094caf92a0175618f832f2e3b869a71b514870efffb8e2375c2: Status 404 returned error can't find the container with id 02f16648eacc7094caf92a0175618f832f2e3b869a71b514870efffb8e2375c2
Sep 10 03:01:05.435262 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:05.435233    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-08482c4c-5cbcb75765-r6h2h" event={"ID":"84c28261-58d6-431c-9a16-b7d88640bed0","Type":"ContainerStarted","Data":"119056953b1769c989e13e1a6ee14d08df90a8edbd924f6ea2eb0ea752114f2b"}
Sep 10 03:01:05.435397 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:05.435301    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-08482c4c-5cbcb75765-r6h2h"
Sep 10 03:01:05.435397 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:05.435327    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-08482c4c-5cbcb75765-r6h2h" event={"ID":"84c28261-58d6-431c-9a16-b7d88640bed0","Type":"ContainerStarted","Data":"02f16648eacc7094caf92a0175618f832f2e3b869a71b514870efffb8e2375c2"}
Sep 10 03:01:05.436960 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:05.436932    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-08482c4c-679998bcb6-c64z5" event={"ID":"9238a60b-2b10-433e-bec5-10cf7d0eb3dd","Type":"ContainerStarted","Data":"f95997c2e9b24444cb92bc40c770ef958f6345cdebb9e16bdcd4b64124cfabbe"}
Sep 10 03:01:05.437045 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:05.436990    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-08482c4c-679998bcb6-c64z5" event={"ID":"9238a60b-2b10-433e-bec5-10cf7d0eb3dd","Type":"ContainerStarted","Data":"9848a3c01a58f89f2c0d130d2da075a25e0f58464bfdd11d6067f4a3ae6e8cb3"}
Sep 10 03:01:05.437045 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:05.437021    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-08482c4c-679998bcb6-c64z5"
Sep 10 03:01:05.450944 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:05.450906    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-08482c4c-5cbcb75765-r6h2h" podStartSLOduration=1.45088837 podStartE2EDuration="1.45088837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:01:05.449383439 +0000 UTC m=+855.824469360" watchObservedRunningTime="2026-09-10 03:01:05.45088837 +0000 UTC m=+855.825974290"
Sep 10 03:01:05.467629 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:05.467580    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-08482c4c-679998bcb6-c64z5" podStartSLOduration=1.467562134 podStartE2EDuration="1.467562134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:01:05.465332102 +0000 UTC m=+855.840418024" watchObservedRunningTime="2026-09-10 03:01:05.467562134 +0000 UTC m=+855.842648055"
Sep 10 03:01:16.444830 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:16.444788    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-08482c4c-5cbcb75765-r6h2h"
Sep 10 03:01:16.445311 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:16.444856    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-08482c4c-679998bcb6-c64z5"
Sep 10 03:01:19.537041 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:19.537010    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:21.040050 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:21.040020    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:21.585993 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:21.585959    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:22.366199 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:22.366165    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:25.120704 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:25.120669    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:30.137933 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:30.137893    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:30.773573 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:30.773538    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:31.303935 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:31.303895    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:31.349098 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:31.349068    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5ab28b-c7d5fcf85-svr5v"]
Sep 10 03:01:31.349300 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:31.349275    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-5ab28b-c7d5fcf85-svr5v" podUID="1ffcda36-f226-4591-bac6-3c472c0aaee8" containerName="payload-processing" containerID="cri-o://45dd2bad9b2d1a381172f694f5f97f56e93c471b46374db2fe3947365c03ced3" gracePeriod=30
Sep 10 03:01:31.384142 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:31.384096    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:31.490630 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:31.490601    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:31.595051 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:31.594804    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:31.597089 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:31.597064    2558 generic.go:356] "Generic (PLEG): container finished" podID="1ffcda36-f226-4591-bac6-3c472c0aaee8" containerID="45dd2bad9b2d1a381172f694f5f97f56e93c471b46374db2fe3947365c03ced3" exitCode=0
Sep 10 03:01:31.597222 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:31.597148    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-5ab28b-c7d5fcf85-svr5v" event={"ID":"1ffcda36-f226-4591-bac6-3c472c0aaee8","Type":"ContainerDied","Data":"45dd2bad9b2d1a381172f694f5f97f56e93c471b46374db2fe3947365c03ced3"}
Sep 10 03:01:31.597222 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:31.597209    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-5ab28b-c7d5fcf85-svr5v" event={"ID":"1ffcda36-f226-4591-bac6-3c472c0aaee8","Type":"ContainerDied","Data":"881564a67807f9a6849207b463632dc791c70696f3fe21c788e9ad8535d532d5"}
Sep 10 03:01:31.597300 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:31.597233    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="881564a67807f9a6849207b463632dc791c70696f3fe21c788e9ad8535d532d5"
Sep 10 03:01:31.602822 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:31.602798    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-5ab28b-c7d5fcf85-svr5v"
Sep 10 03:01:31.782057 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:31.782027    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ffcda36-f226-4591-bac6-3c472c0aaee8-plugins-config-volume\") pod \"1ffcda36-f226-4591-bac6-3c472c0aaee8\" (UID: \"1ffcda36-f226-4591-bac6-3c472c0aaee8\") "
Sep 10 03:01:31.782202 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:31.782066    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfn6x\" (UniqueName: \"kubernetes.io/projected/1ffcda36-f226-4591-bac6-3c472c0aaee8-kube-api-access-zfn6x\") pod \"1ffcda36-f226-4591-bac6-3c472c0aaee8\" (UID: \"1ffcda36-f226-4591-bac6-3c472c0aaee8\") "
Sep 10 03:01:31.782417 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:31.782395    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ffcda36-f226-4591-bac6-3c472c0aaee8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1ffcda36-f226-4591-bac6-3c472c0aaee8" (UID: "1ffcda36-f226-4591-bac6-3c472c0aaee8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:01:31.784044 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:31.784026    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ffcda36-f226-4591-bac6-3c472c0aaee8-kube-api-access-zfn6x" (OuterVolumeSpecName: "kube-api-access-zfn6x") pod "1ffcda36-f226-4591-bac6-3c472c0aaee8" (UID: "1ffcda36-f226-4591-bac6-3c472c0aaee8"). InnerVolumeSpecName "kube-api-access-zfn6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:01:31.883317 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:31.883256    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ffcda36-f226-4591-bac6-3c472c0aaee8-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:01:31.883317 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:31.883282    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfn6x\" (UniqueName: \"kubernetes.io/projected/1ffcda36-f226-4591-bac6-3c472c0aaee8-kube-api-access-zfn6x\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:01:31.923239 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:31.923213    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:32.602976 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:32.602944    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-5ab28b-c7d5fcf85-svr5v"
Sep 10 03:01:32.622483 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:32.622453    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5ab28b-c7d5fcf85-svr5v"]
Sep 10 03:01:32.626023 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:32.626002    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5ab28b-c7d5fcf85-svr5v"]
Sep 10 03:01:34.068536 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:34.068492    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ffcda36-f226-4591-bac6-3c472c0aaee8" path="/var/lib/kubelet/pods/1ffcda36-f226-4591-bac6-3c472c0aaee8/volumes"
Sep 10 03:01:34.304714 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:34.304686    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:34.848551 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:34.848492    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:34.880177 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:34.880094    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-08482c4c-5cbcb75765-r6h2h"]
Sep 10 03:01:34.880657 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:34.880625    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-08482c4c-5cbcb75765-r6h2h" podUID="84c28261-58d6-431c-9a16-b7d88640bed0" containerName="payload-processing" containerID="cri-o://119056953b1769c989e13e1a6ee14d08df90a8edbd924f6ea2eb0ea752114f2b" gracePeriod=30
Sep 10 03:01:34.888839 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:34.888812    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-08482c4c-679998bcb6-c64z5"]
Sep 10 03:01:34.889082 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:34.889057    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-08482c4c-679998bcb6-c64z5" podUID="9238a60b-2b10-433e-bec5-10cf7d0eb3dd" containerName="payload-pre-processing" containerID="cri-o://f95997c2e9b24444cb92bc40c770ef958f6345cdebb9e16bdcd4b64124cfabbe" gracePeriod=30
Sep 10 03:01:34.977136 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:34.977092    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:35.038340 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.038165    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:35.092890 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.092830    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:35.187036 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.186999    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-08482c4c-5cbcb75765-r6h2h"
Sep 10 03:01:35.188499 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.188470    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:35.190147 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.190127    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-08482c4c-679998bcb6-c64z5"
Sep 10 03:01:35.316218 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.316194    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9238a60b-2b10-433e-bec5-10cf7d0eb3dd-plugins-config-volume\") pod \"9238a60b-2b10-433e-bec5-10cf7d0eb3dd\" (UID: \"9238a60b-2b10-433e-bec5-10cf7d0eb3dd\") "
Sep 10 03:01:35.316382 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.316224    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmqx5\" (UniqueName: \"kubernetes.io/projected/84c28261-58d6-431c-9a16-b7d88640bed0-kube-api-access-hmqx5\") pod \"84c28261-58d6-431c-9a16-b7d88640bed0\" (UID: \"84c28261-58d6-431c-9a16-b7d88640bed0\") "
Sep 10 03:01:35.316382 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.316248    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c28261-58d6-431c-9a16-b7d88640bed0-plugins-config-volume\") pod \"84c28261-58d6-431c-9a16-b7d88640bed0\" (UID: \"84c28261-58d6-431c-9a16-b7d88640bed0\") "
Sep 10 03:01:35.316382 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.316273    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sllzp\" (UniqueName: \"kubernetes.io/projected/9238a60b-2b10-433e-bec5-10cf7d0eb3dd-kube-api-access-sllzp\") pod \"9238a60b-2b10-433e-bec5-10cf7d0eb3dd\" (UID: \"9238a60b-2b10-433e-bec5-10cf7d0eb3dd\") "
Sep 10 03:01:35.316615 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.316591    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9238a60b-2b10-433e-bec5-10cf7d0eb3dd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9238a60b-2b10-433e-bec5-10cf7d0eb3dd" (UID: "9238a60b-2b10-433e-bec5-10cf7d0eb3dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:01:35.316696 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.316669    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c28261-58d6-431c-9a16-b7d88640bed0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "84c28261-58d6-431c-9a16-b7d88640bed0" (UID: "84c28261-58d6-431c-9a16-b7d88640bed0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:01:35.318420 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.318391    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c28261-58d6-431c-9a16-b7d88640bed0-kube-api-access-hmqx5" (OuterVolumeSpecName: "kube-api-access-hmqx5") pod "84c28261-58d6-431c-9a16-b7d88640bed0" (UID: "84c28261-58d6-431c-9a16-b7d88640bed0"). InnerVolumeSpecName "kube-api-access-hmqx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:01:35.318503 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.318426    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9238a60b-2b10-433e-bec5-10cf7d0eb3dd-kube-api-access-sllzp" (OuterVolumeSpecName: "kube-api-access-sllzp") pod "9238a60b-2b10-433e-bec5-10cf7d0eb3dd" (UID: "9238a60b-2b10-433e-bec5-10cf7d0eb3dd"). InnerVolumeSpecName "kube-api-access-sllzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:01:35.417236 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.417184    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9238a60b-2b10-433e-bec5-10cf7d0eb3dd-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:01:35.417236 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.417205    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmqx5\" (UniqueName: \"kubernetes.io/projected/84c28261-58d6-431c-9a16-b7d88640bed0-kube-api-access-hmqx5\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:01:35.417236 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.417214    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c28261-58d6-431c-9a16-b7d88640bed0-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:01:35.417236 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.417222    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sllzp\" (UniqueName: \"kubernetes.io/projected/9238a60b-2b10-433e-bec5-10cf7d0eb3dd-kube-api-access-sllzp\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:01:35.621124 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.621073    2558 generic.go:356] "Generic (PLEG): container finished" podID="84c28261-58d6-431c-9a16-b7d88640bed0" containerID="119056953b1769c989e13e1a6ee14d08df90a8edbd924f6ea2eb0ea752114f2b" exitCode=0
Sep 10 03:01:35.621227 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.621132    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-08482c4c-5cbcb75765-r6h2h"
Sep 10 03:01:35.621227 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.621146    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-08482c4c-5cbcb75765-r6h2h" event={"ID":"84c28261-58d6-431c-9a16-b7d88640bed0","Type":"ContainerDied","Data":"119056953b1769c989e13e1a6ee14d08df90a8edbd924f6ea2eb0ea752114f2b"}
Sep 10 03:01:35.621227 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.621204    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-08482c4c-5cbcb75765-r6h2h" event={"ID":"84c28261-58d6-431c-9a16-b7d88640bed0","Type":"ContainerDied","Data":"02f16648eacc7094caf92a0175618f832f2e3b869a71b514870efffb8e2375c2"}
Sep 10 03:01:35.621391 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.621229    2558 scope.go:117] "RemoveContainer" containerID="119056953b1769c989e13e1a6ee14d08df90a8edbd924f6ea2eb0ea752114f2b"
Sep 10 03:01:35.622647 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.622622    2558 generic.go:356] "Generic (PLEG): container finished" podID="9238a60b-2b10-433e-bec5-10cf7d0eb3dd" containerID="f95997c2e9b24444cb92bc40c770ef958f6345cdebb9e16bdcd4b64124cfabbe" exitCode=0
Sep 10 03:01:35.622758 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.622673    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-08482c4c-679998bcb6-c64z5"
Sep 10 03:01:35.622758 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.622678    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-08482c4c-679998bcb6-c64z5" event={"ID":"9238a60b-2b10-433e-bec5-10cf7d0eb3dd","Type":"ContainerDied","Data":"f95997c2e9b24444cb92bc40c770ef958f6345cdebb9e16bdcd4b64124cfabbe"}
Sep 10 03:01:35.622758 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.622720    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-08482c4c-679998bcb6-c64z5" event={"ID":"9238a60b-2b10-433e-bec5-10cf7d0eb3dd","Type":"ContainerDied","Data":"9848a3c01a58f89f2c0d130d2da075a25e0f58464bfdd11d6067f4a3ae6e8cb3"}
Sep 10 03:01:35.631067 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.631050    2558 scope.go:117] "RemoveContainer" containerID="119056953b1769c989e13e1a6ee14d08df90a8edbd924f6ea2eb0ea752114f2b"
Sep 10 03:01:35.631399 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:01:35.631381    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"119056953b1769c989e13e1a6ee14d08df90a8edbd924f6ea2eb0ea752114f2b\": container with ID starting with 119056953b1769c989e13e1a6ee14d08df90a8edbd924f6ea2eb0ea752114f2b not found: ID does not exist" containerID="119056953b1769c989e13e1a6ee14d08df90a8edbd924f6ea2eb0ea752114f2b"
Sep 10 03:01:35.631489 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.631405    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"119056953b1769c989e13e1a6ee14d08df90a8edbd924f6ea2eb0ea752114f2b"} err="failed to get container status \"119056953b1769c989e13e1a6ee14d08df90a8edbd924f6ea2eb0ea752114f2b\": rpc error: code = NotFound desc = could not find container \"119056953b1769c989e13e1a6ee14d08df90a8edbd924f6ea2eb0ea752114f2b\": container with ID starting with 119056953b1769c989e13e1a6ee14d08df90a8edbd924f6ea2eb0ea752114f2b not found: ID does not exist"
Sep 10 03:01:35.631489 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.631422    2558 scope.go:117] "RemoveContainer" containerID="f95997c2e9b24444cb92bc40c770ef958f6345cdebb9e16bdcd4b64124cfabbe"
Sep 10 03:01:35.639225 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.639206    2558 scope.go:117] "RemoveContainer" containerID="f95997c2e9b24444cb92bc40c770ef958f6345cdebb9e16bdcd4b64124cfabbe"
Sep 10 03:01:35.639492 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:01:35.639473    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f95997c2e9b24444cb92bc40c770ef958f6345cdebb9e16bdcd4b64124cfabbe\": container with ID starting with f95997c2e9b24444cb92bc40c770ef958f6345cdebb9e16bdcd4b64124cfabbe not found: ID does not exist" containerID="f95997c2e9b24444cb92bc40c770ef958f6345cdebb9e16bdcd4b64124cfabbe"
Sep 10 03:01:35.639549 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.639495    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f95997c2e9b24444cb92bc40c770ef958f6345cdebb9e16bdcd4b64124cfabbe"} err="failed to get container status \"f95997c2e9b24444cb92bc40c770ef958f6345cdebb9e16bdcd4b64124cfabbe\": rpc error: code = NotFound desc = could not find container \"f95997c2e9b24444cb92bc40c770ef958f6345cdebb9e16bdcd4b64124cfabbe\": container with ID starting with f95997c2e9b24444cb92bc40c770ef958f6345cdebb9e16bdcd4b64124cfabbe not found: ID does not exist"
Sep 10 03:01:35.644301 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.644282    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-08482c4c-5cbcb75765-r6h2h"]
Sep 10 03:01:35.650083 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.650065    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-08482c4c-5cbcb75765-r6h2h"]
Sep 10 03:01:35.660970 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.660949    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-08482c4c-679998bcb6-c64z5"]
Sep 10 03:01:35.664333 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.664315    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-08482c4c-679998bcb6-c64z5"]
Sep 10 03:01:35.905295 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:35.905265    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:36.008635 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:36.008612    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:36.069873 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:36.069847    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84c28261-58d6-431c-9a16-b7d88640bed0" path="/var/lib/kubelet/pods/84c28261-58d6-431c-9a16-b7d88640bed0/volumes"
Sep 10 03:01:36.070221 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:36.070207    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9238a60b-2b10-433e-bec5-10cf7d0eb3dd" path="/var/lib/kubelet/pods/9238a60b-2b10-433e-bec5-10cf7d0eb3dd/volumes"
Sep 10 03:01:36.127309 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:36.127279    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:36.996511 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:36.996474    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:37.495280 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:37.495249    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:43.240037 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:43.239951    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:43.347579 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:43.347541    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:44.213014 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:44.212982    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:44.313642 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:44.313606    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:55.001195 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.001158    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m"]
Sep 10 03:01:55.002704 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.001559    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ffcda36-f226-4591-bac6-3c472c0aaee8" containerName="payload-processing"
Sep 10 03:01:55.002704 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.001571    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ffcda36-f226-4591-bac6-3c472c0aaee8" containerName="payload-processing"
Sep 10 03:01:55.002704 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.001584    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84c28261-58d6-431c-9a16-b7d88640bed0" containerName="payload-processing"
Sep 10 03:01:55.002704 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.001590    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="84c28261-58d6-431c-9a16-b7d88640bed0" containerName="payload-processing"
Sep 10 03:01:55.002704 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.001600    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9238a60b-2b10-433e-bec5-10cf7d0eb3dd" containerName="payload-pre-processing"
Sep 10 03:01:55.002704 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.001605    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9238a60b-2b10-433e-bec5-10cf7d0eb3dd" containerName="payload-pre-processing"
Sep 10 03:01:55.002704 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.001663    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9238a60b-2b10-433e-bec5-10cf7d0eb3dd" containerName="payload-pre-processing"
Sep 10 03:01:55.002704 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.001671    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ffcda36-f226-4591-bac6-3c472c0aaee8" containerName="payload-processing"
Sep 10 03:01:55.002704 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.001682    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="84c28261-58d6-431c-9a16-b7d88640bed0" containerName="payload-processing"
Sep 10 03:01:55.003971 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.003951    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m"
Sep 10 03:01:55.007112 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.007078    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-097e16\"/\"premium-w0-097e16-kserve-self-signed-certs\""
Sep 10 03:01:55.007216 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.007079    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-097e16\"/\"kube-root-ca.crt\""
Sep 10 03:01:55.007216 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.007079    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-097e16\"/\"default-dockercfg-lj6l4\""
Sep 10 03:01:55.007216 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.007078    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-097e16\"/\"openshift-service-ca.crt\""
Sep 10 03:01:55.013811 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.013790    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m"]
Sep 10 03:01:55.031314 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.031291    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:01:55.062229 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.062204    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-model-cache\") pod \"premium-w0-097e16-kserve-747bf48f7b-rv44m\" (UID: \"fc8b91c3-7d8d-4379-87e6-4d02d59401b8\") " pod="e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m"
Sep 10 03:01:55.062331 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.062266    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-dshm\") pod \"premium-w0-097e16-kserve-747bf48f7b-rv44m\" (UID: \"fc8b91c3-7d8d-4379-87e6-4d02d59401b8\") " pod="e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m"
Sep 10 03:01:55.062331 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.062290    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bndlk\" (UniqueName: \"kubernetes.io/projected/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-kube-api-access-bndlk\") pod \"premium-w0-097e16-kserve-747bf48f7b-rv44m\" (UID: \"fc8b91c3-7d8d-4379-87e6-4d02d59401b8\") " pod="e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m"
Sep 10 03:01:55.062411 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.062332    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-tmp-dir\") pod \"premium-w0-097e16-kserve-747bf48f7b-rv44m\" (UID: \"fc8b91c3-7d8d-4379-87e6-4d02d59401b8\") " pod="e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m"
Sep 10 03:01:55.062411 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.062356    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-home\") pod \"premium-w0-097e16-kserve-747bf48f7b-rv44m\" (UID: \"fc8b91c3-7d8d-4379-87e6-4d02d59401b8\") " pod="e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m"
Sep 10 03:01:55.062411 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.062384    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-tls-certs\") pod \"premium-w0-097e16-kserve-747bf48f7b-rv44m\" (UID: \"fc8b91c3-7d8d-4379-87e6-4d02d59401b8\") " pod="e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m"
Sep 10 03:01:55.163223 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.163197    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-model-cache\") pod \"premium-w0-097e16-kserve-747bf48f7b-rv44m\" (UID: \"fc8b91c3-7d8d-4379-87e6-4d02d59401b8\") " pod="e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m"
Sep 10 03:01:55.163339 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.163232    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-dshm\") pod \"premium-w0-097e16-kserve-747bf48f7b-rv44m\" (UID: \"fc8b91c3-7d8d-4379-87e6-4d02d59401b8\") " pod="e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m"
Sep 10 03:01:55.163339 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.163251    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bndlk\" (UniqueName: \"kubernetes.io/projected/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-kube-api-access-bndlk\") pod \"premium-w0-097e16-kserve-747bf48f7b-rv44m\" (UID: \"fc8b91c3-7d8d-4379-87e6-4d02d59401b8\") " pod="e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m"
Sep 10 03:01:55.163339 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.163283    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-tmp-dir\") pod \"premium-w0-097e16-kserve-747bf48f7b-rv44m\" (UID: \"fc8b91c3-7d8d-4379-87e6-4d02d59401b8\") " pod="e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m"
Sep 10 03:01:55.163339 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.163298    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-home\") pod \"premium-w0-097e16-kserve-747bf48f7b-rv44m\" (UID: \"fc8b91c3-7d8d-4379-87e6-4d02d59401b8\") " pod="e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m"
Sep 10 03:01:55.163611 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.163441    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-tls-certs\") pod \"premium-w0-097e16-kserve-747bf48f7b-rv44m\" (UID: \"fc8b91c3-7d8d-4379-87e6-4d02d59401b8\") " pod="e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m"
Sep 10 03:01:55.163668 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.163625    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-model-cache\") pod \"premium-w0-097e16-kserve-747bf48f7b-rv44m\" (UID: \"fc8b91c3-7d8d-4379-87e6-4d02d59401b8\") " pod="e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m"
Sep 10 03:01:55.163668 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.163635    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-tmp-dir\") pod \"premium-w0-097e16-kserve-747bf48f7b-rv44m\" (UID: \"fc8b91c3-7d8d-4379-87e6-4d02d59401b8\") " pod="e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m"
Sep 10 03:01:55.163829 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.163811    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-home\") pod \"premium-w0-097e16-kserve-747bf48f7b-rv44m\" (UID: \"fc8b91c3-7d8d-4379-87e6-4d02d59401b8\") " pod="e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m"
Sep 10 03:01:55.165557 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.165533    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-dshm\") pod \"premium-w0-097e16-kserve-747bf48f7b-rv44m\" (UID: \"fc8b91c3-7d8d-4379-87e6-4d02d59401b8\") " pod="e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m"
Sep 10 03:01:55.165981 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.165958    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-tls-certs\") pod \"premium-w0-097e16-kserve-747bf48f7b-rv44m\" (UID: \"fc8b91c3-7d8d-4379-87e6-4d02d59401b8\") " pod="e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m"
Sep 10 03:01:55.170134 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.170096    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bndlk\" (UniqueName: \"kubernetes.io/projected/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-kube-api-access-bndlk\") pod \"premium-w0-097e16-kserve-747bf48f7b-rv44m\" (UID: \"fc8b91c3-7d8d-4379-87e6-4d02d59401b8\") " pod="e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m"
Sep 10 03:01:55.314590 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.314540    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m"
Sep 10 03:01:55.585842 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.585814    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m"]
Sep 10 03:01:55.587865 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:01:55.587828    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc8b91c3_7d8d_4379_87e6_4d02d59401b8.slice/crio-7033dd931ca0ae507ffaf7d857e2528678bedfe98a8aecb85d20c9557eac2cdb WatchSource:0}: Error finding container 7033dd931ca0ae507ffaf7d857e2528678bedfe98a8aecb85d20c9557eac2cdb: Status 404 returned error can't find the container with id 7033dd931ca0ae507ffaf7d857e2528678bedfe98a8aecb85d20c9557eac2cdb
Sep 10 03:01:55.739488 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.739448    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m" event={"ID":"fc8b91c3-7d8d-4379-87e6-4d02d59401b8","Type":"ContainerStarted","Data":"b92115c1f6a983cf015fc62df89762c110982f5ca4cf796bad0ffaffa6e75f37"}
Sep 10 03:01:55.739660 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.739504    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m" event={"ID":"fc8b91c3-7d8d-4379-87e6-4d02d59401b8","Type":"ContainerStarted","Data":"7033dd931ca0ae507ffaf7d857e2528678bedfe98a8aecb85d20c9557eac2cdb"}
Sep 10 03:01:55.757977 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:01:55.757925    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m" podStartSLOduration=1.757911006 podStartE2EDuration="1.757911006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:01:55.757664184 +0000 UTC m=+906.132750105" watchObservedRunningTime="2026-09-10 03:01:55.757911006 +0000 UTC m=+906.132996928"
Sep 10 03:02:05.315175 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:05.315142    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m"
Sep 10 03:02:05.315551 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:05.315190    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m"
Sep 10 03:02:05.327681 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:05.327657    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m"
Sep 10 03:02:05.813079 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:05.813052    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m"
Sep 10 03:02:09.962240 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:09.962209    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:02:13.878620 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:13.878587    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:02:14.159163 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:14.159065    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:02:14.625260 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:14.625227    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:02:14.947771 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:14.947693    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:02:15.208385 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:15.208343    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:02:16.293319 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:16.293286    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:02:16.408391 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:16.408362    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:02:16.502627 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:16.502597    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:02:17.308665 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:17.308635    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:02:17.408565 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:17.408537    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:02:18.033847 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:18.033815    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:02:18.127897 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:18.127868    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:02:19.302004 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:19.301968    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:02:19.512483 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:19.512447    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:02:19.612700 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:19.612622    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:02:19.734847 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:19.734819    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:02:19.895091 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:19.895002    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:02:31.743510 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:31.743464    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:02:32.633228 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:32.633198    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:02:38.309634 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:38.309601    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:02:38.418234 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:38.418203    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:02:38.509322 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:38.509290    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:02:43.039917 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:43.039875    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:02:49.058421 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:49.058386    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:02:49.839911 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:49.839856    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:02:55.415385 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:55.415346    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:02:55.525294 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:55.525258    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:02:55.615762 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:55.615733    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:02:55.749380 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:55.749359    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:02:56.212339 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:56.212295    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h"]
Sep 10 03:02:56.215858 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:56.215838    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h"
Sep 10 03:02:56.218321 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:56.218292    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ce5f9d\"/\"test-model-ce5f9d-kserve-self-signed-certs\""
Sep 10 03:02:56.219061 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:56.219042    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ce5f9d\"/\"openshift-service-ca.crt\""
Sep 10 03:02:56.219202 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:56.219182    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ce5f9d\"/\"default-dockercfg-cr9q4\""
Sep 10 03:02:56.219292 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:56.219276    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ce5f9d\"/\"kube-root-ca.crt\""
Sep 10 03:02:56.228953 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:56.228930    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h"]
Sep 10 03:02:56.238905 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:56.238849    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxxg7\" (UniqueName: \"kubernetes.io/projected/a35c892b-01e1-402c-804c-71301f820d81-kube-api-access-fxxg7\") pod \"test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h\" (UID: \"a35c892b-01e1-402c-804c-71301f820d81\") " pod="ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h"
Sep 10 03:02:56.238905 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:56.238885    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a35c892b-01e1-402c-804c-71301f820d81-home\") pod \"test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h\" (UID: \"a35c892b-01e1-402c-804c-71301f820d81\") " pod="ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h"
Sep 10 03:02:56.239087 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:56.238938    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a35c892b-01e1-402c-804c-71301f820d81-dshm\") pod \"test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h\" (UID: \"a35c892b-01e1-402c-804c-71301f820d81\") " pod="ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h"
Sep 10 03:02:56.239087 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:56.238961    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a35c892b-01e1-402c-804c-71301f820d81-tmp-dir\") pod \"test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h\" (UID: \"a35c892b-01e1-402c-804c-71301f820d81\") " pod="ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h"
Sep 10 03:02:56.239087 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:56.239009    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a35c892b-01e1-402c-804c-71301f820d81-tls-certs\") pod \"test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h\" (UID: \"a35c892b-01e1-402c-804c-71301f820d81\") " pod="ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h"
Sep 10 03:02:56.239087 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:56.239044    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a35c892b-01e1-402c-804c-71301f820d81-model-cache\") pod \"test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h\" (UID: \"a35c892b-01e1-402c-804c-71301f820d81\") " pod="ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h"
Sep 10 03:02:56.244400 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:56.244379    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:02:56.339807 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:56.339778    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxxg7\" (UniqueName: \"kubernetes.io/projected/a35c892b-01e1-402c-804c-71301f820d81-kube-api-access-fxxg7\") pod \"test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h\" (UID: \"a35c892b-01e1-402c-804c-71301f820d81\") " pod="ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h"
Sep 10 03:02:56.339940 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:56.339814    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a35c892b-01e1-402c-804c-71301f820d81-home\") pod \"test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h\" (UID: \"a35c892b-01e1-402c-804c-71301f820d81\") " pod="ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h"
Sep 10 03:02:56.339940 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:56.339845    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a35c892b-01e1-402c-804c-71301f820d81-dshm\") pod \"test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h\" (UID: \"a35c892b-01e1-402c-804c-71301f820d81\") " pod="ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h"
Sep 10 03:02:56.340020 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:56.339954    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a35c892b-01e1-402c-804c-71301f820d81-tmp-dir\") pod \"test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h\" (UID: \"a35c892b-01e1-402c-804c-71301f820d81\") " pod="ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h"
Sep 10 03:02:56.340020 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:56.340013    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a35c892b-01e1-402c-804c-71301f820d81-tls-certs\") pod \"test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h\" (UID: \"a35c892b-01e1-402c-804c-71301f820d81\") " pod="ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h"
Sep 10 03:02:56.340141 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:56.340050    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a35c892b-01e1-402c-804c-71301f820d81-model-cache\") pod \"test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h\" (UID: \"a35c892b-01e1-402c-804c-71301f820d81\") " pod="ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h"
Sep 10 03:02:56.340292 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:56.340271    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a35c892b-01e1-402c-804c-71301f820d81-home\") pod \"test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h\" (UID: \"a35c892b-01e1-402c-804c-71301f820d81\") " pod="ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h"
Sep 10 03:02:56.340362 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:56.340316    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a35c892b-01e1-402c-804c-71301f820d81-tmp-dir\") pod \"test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h\" (UID: \"a35c892b-01e1-402c-804c-71301f820d81\") " pod="ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h"
Sep 10 03:02:56.340421 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:56.340372    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a35c892b-01e1-402c-804c-71301f820d81-model-cache\") pod \"test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h\" (UID: \"a35c892b-01e1-402c-804c-71301f820d81\") " pod="ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h"
Sep 10 03:02:56.342000 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:56.341974    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a35c892b-01e1-402c-804c-71301f820d81-dshm\") pod \"test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h\" (UID: \"a35c892b-01e1-402c-804c-71301f820d81\") " pod="ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h"
Sep 10 03:02:56.342588 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:56.342567    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a35c892b-01e1-402c-804c-71301f820d81-tls-certs\") pod \"test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h\" (UID: \"a35c892b-01e1-402c-804c-71301f820d81\") " pod="ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h"
Sep 10 03:02:56.356336 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:56.356310    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxxg7\" (UniqueName: \"kubernetes.io/projected/a35c892b-01e1-402c-804c-71301f820d81-kube-api-access-fxxg7\") pod \"test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h\" (UID: \"a35c892b-01e1-402c-804c-71301f820d81\") " pod="ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h"
Sep 10 03:02:56.387982 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:56.387954    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:02:56.485906 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:56.485846    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:02:56.525837 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:56.525814    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h"
Sep 10 03:02:56.588239 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:56.588172    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:02:56.798333 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:56.798309    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h"]
Sep 10 03:02:56.800967 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:02:56.800943    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda35c892b_01e1_402c_804c_71301f820d81.slice/crio-4763df444281538600fe7146c1c93e9b7269a5857b5b5c6d8ef24b8c1594ef22 WatchSource:0}: Error finding container 4763df444281538600fe7146c1c93e9b7269a5857b5b5c6d8ef24b8c1594ef22: Status 404 returned error can't find the container with id 4763df444281538600fe7146c1c93e9b7269a5857b5b5c6d8ef24b8c1594ef22
Sep 10 03:02:57.100832 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:57.100747    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h" event={"ID":"a35c892b-01e1-402c-804c-71301f820d81","Type":"ContainerStarted","Data":"16ac3e95047f8fc292dd28ce2b56ea7ce652262b1208bd995155a6c6bb1c8521"}
Sep 10 03:02:57.100832 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:57.100791    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h" event={"ID":"a35c892b-01e1-402c-804c-71301f820d81","Type":"ContainerStarted","Data":"4763df444281538600fe7146c1c93e9b7269a5857b5b5c6d8ef24b8c1594ef22"}
Sep 10 03:02:57.115898 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:57.115846    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h" podStartSLOduration=1.115827698 podStartE2EDuration="1.115827698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:02:57.115433015 +0000 UTC m=+967.490518937" watchObservedRunningTime="2026-09-10 03:02:57.115827698 +0000 UTC m=+967.490913624"
Sep 10 03:02:58.170275 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:02:58.170244    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:03:03.912402 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:03.912365    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:03:04.013396 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:04.013369    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:03:04.624648 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:04.624613    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:03:06.526821 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:06.526788    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h"
Sep 10 03:03:06.527230 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:06.526839    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h"
Sep 10 03:03:06.539547 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:06.539518    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h"
Sep 10 03:03:07.171594 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:07.171558    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h"
Sep 10 03:03:10.379182 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:10.379154    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:03:10.448399 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:10.448369    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:03:10.635895 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:10.635822    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:03:11.083869 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:11.083829    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:03:11.253497 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:11.253463    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:03:17.268776 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:17.268747    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:03:17.365793 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:17.365767    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:03:18.427099 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:18.427058    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:03:18.538485 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:18.538447    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:03:18.889991 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:18.889961    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:03:19.099085 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:19.099057    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:03:19.197900 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:19.197831    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:03:19.233052 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:19.233025    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-842ba8c2-c577fdbf-ftjm4"]
Sep 10 03:03:19.236531 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:19.236516    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-842ba8c2-c577fdbf-ftjm4"
Sep 10 03:03:19.238765 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:19.238744    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-842ba8c2\""
Sep 10 03:03:19.251623 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:19.251602    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-842ba8c2-c577fdbf-ftjm4"]
Sep 10 03:03:19.295252 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:19.295225    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-842ba8c2-6cdf879494-pmjfc"]
Sep 10 03:03:19.298851 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:19.298835    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-842ba8c2-6cdf879494-pmjfc"
Sep 10 03:03:19.301729 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:19.301710    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-842ba8c2-dockercfg-vkwnt\""
Sep 10 03:03:19.312013 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:19.311994    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-842ba8c2-6cdf879494-pmjfc"]
Sep 10 03:03:19.320716 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:19.320697    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b30f51-d720-40fb-b419-6ed7700c42ad-plugins-config-volume\") pod \"payload-processing-e2e-mt-842ba8c2-c577fdbf-ftjm4\" (UID: \"f7b30f51-d720-40fb-b419-6ed7700c42ad\") " pod="openshift-ingress/payload-processing-e2e-mt-842ba8c2-c577fdbf-ftjm4"
Sep 10 03:03:19.320834 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:19.320730    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkdcw\" (UniqueName: \"kubernetes.io/projected/f7b30f51-d720-40fb-b419-6ed7700c42ad-kube-api-access-xkdcw\") pod \"payload-processing-e2e-mt-842ba8c2-c577fdbf-ftjm4\" (UID: \"f7b30f51-d720-40fb-b419-6ed7700c42ad\") " pod="openshift-ingress/payload-processing-e2e-mt-842ba8c2-c577fdbf-ftjm4"
Sep 10 03:03:19.421752 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:19.421725    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b30f51-d720-40fb-b419-6ed7700c42ad-plugins-config-volume\") pod \"payload-processing-e2e-mt-842ba8c2-c577fdbf-ftjm4\" (UID: \"f7b30f51-d720-40fb-b419-6ed7700c42ad\") " pod="openshift-ingress/payload-processing-e2e-mt-842ba8c2-c577fdbf-ftjm4"
Sep 10 03:03:19.421752 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:19.421754    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkdcw\" (UniqueName: \"kubernetes.io/projected/f7b30f51-d720-40fb-b419-6ed7700c42ad-kube-api-access-xkdcw\") pod \"payload-processing-e2e-mt-842ba8c2-c577fdbf-ftjm4\" (UID: \"f7b30f51-d720-40fb-b419-6ed7700c42ad\") " pod="openshift-ingress/payload-processing-e2e-mt-842ba8c2-c577fdbf-ftjm4"
Sep 10 03:03:19.421957 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:19.421784    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsp8l\" (UniqueName: \"kubernetes.io/projected/397bb1a5-c853-4334-8c10-8f4fde03bd9a-kube-api-access-zsp8l\") pod \"payload-pre-processing-e2e-mt-842ba8c2-6cdf879494-pmjfc\" (UID: \"397bb1a5-c853-4334-8c10-8f4fde03bd9a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-842ba8c2-6cdf879494-pmjfc"
Sep 10 03:03:19.421957 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:19.421827    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/397bb1a5-c853-4334-8c10-8f4fde03bd9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-842ba8c2-6cdf879494-pmjfc\" (UID: \"397bb1a5-c853-4334-8c10-8f4fde03bd9a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-842ba8c2-6cdf879494-pmjfc"
Sep 10 03:03:19.422442 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:19.422422    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b30f51-d720-40fb-b419-6ed7700c42ad-plugins-config-volume\") pod \"payload-processing-e2e-mt-842ba8c2-c577fdbf-ftjm4\" (UID: \"f7b30f51-d720-40fb-b419-6ed7700c42ad\") " pod="openshift-ingress/payload-processing-e2e-mt-842ba8c2-c577fdbf-ftjm4"
Sep 10 03:03:19.432036 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:19.432013    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkdcw\" (UniqueName: \"kubernetes.io/projected/f7b30f51-d720-40fb-b419-6ed7700c42ad-kube-api-access-xkdcw\") pod \"payload-processing-e2e-mt-842ba8c2-c577fdbf-ftjm4\" (UID: \"f7b30f51-d720-40fb-b419-6ed7700c42ad\") " pod="openshift-ingress/payload-processing-e2e-mt-842ba8c2-c577fdbf-ftjm4"
Sep 10 03:03:19.522232 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:19.522202    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsp8l\" (UniqueName: \"kubernetes.io/projected/397bb1a5-c853-4334-8c10-8f4fde03bd9a-kube-api-access-zsp8l\") pod \"payload-pre-processing-e2e-mt-842ba8c2-6cdf879494-pmjfc\" (UID: \"397bb1a5-c853-4334-8c10-8f4fde03bd9a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-842ba8c2-6cdf879494-pmjfc"
Sep 10 03:03:19.522333 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:19.522250    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/397bb1a5-c853-4334-8c10-8f4fde03bd9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-842ba8c2-6cdf879494-pmjfc\" (UID: \"397bb1a5-c853-4334-8c10-8f4fde03bd9a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-842ba8c2-6cdf879494-pmjfc"
Sep 10 03:03:19.522850 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:19.522833    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/397bb1a5-c853-4334-8c10-8f4fde03bd9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-842ba8c2-6cdf879494-pmjfc\" (UID: \"397bb1a5-c853-4334-8c10-8f4fde03bd9a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-842ba8c2-6cdf879494-pmjfc"
Sep 10 03:03:19.529812 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:19.529786    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsp8l\" (UniqueName: \"kubernetes.io/projected/397bb1a5-c853-4334-8c10-8f4fde03bd9a-kube-api-access-zsp8l\") pod \"payload-pre-processing-e2e-mt-842ba8c2-6cdf879494-pmjfc\" (UID: \"397bb1a5-c853-4334-8c10-8f4fde03bd9a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-842ba8c2-6cdf879494-pmjfc"
Sep 10 03:03:19.547155 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:19.547098    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-842ba8c2-c577fdbf-ftjm4"
Sep 10 03:03:19.608432 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:19.608404    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-842ba8c2-6cdf879494-pmjfc"
Sep 10 03:03:19.826020 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:19.825994    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-842ba8c2-c577fdbf-ftjm4"]
Sep 10 03:03:19.827254 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:03:19.827228    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b30f51_d720_40fb_b419_6ed7700c42ad.slice/crio-c6860634984945f06ded5605ed5ed70dee2e02576c647178c63dee2a0c5e16eb WatchSource:0}: Error finding container c6860634984945f06ded5605ed5ed70dee2e02576c647178c63dee2a0c5e16eb: Status 404 returned error can't find the container with id c6860634984945f06ded5605ed5ed70dee2e02576c647178c63dee2a0c5e16eb
Sep 10 03:03:19.887700 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:19.887673    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-842ba8c2-6cdf879494-pmjfc"]
Sep 10 03:03:19.889139 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:03:19.889080    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod397bb1a5_c853_4334_8c10_8f4fde03bd9a.slice/crio-0fefbb6cd809e56e30e1f15db21462b47a811a86f2e5f4d2cb204c5b3bf1da32 WatchSource:0}: Error finding container 0fefbb6cd809e56e30e1f15db21462b47a811a86f2e5f4d2cb204c5b3bf1da32: Status 404 returned error can't find the container with id 0fefbb6cd809e56e30e1f15db21462b47a811a86f2e5f4d2cb204c5b3bf1da32
Sep 10 03:03:20.234165 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:20.234132    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-842ba8c2-c577fdbf-ftjm4" event={"ID":"f7b30f51-d720-40fb-b419-6ed7700c42ad","Type":"ContainerStarted","Data":"9c243d925b3da43117e2273648f009dcaf4b99a660bf72e0a77426bb8f5faf3c"}
Sep 10 03:03:20.234328 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:20.234182    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-842ba8c2-c577fdbf-ftjm4" event={"ID":"f7b30f51-d720-40fb-b419-6ed7700c42ad","Type":"ContainerStarted","Data":"c6860634984945f06ded5605ed5ed70dee2e02576c647178c63dee2a0c5e16eb"}
Sep 10 03:03:20.234328 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:20.234214    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-842ba8c2-c577fdbf-ftjm4"
Sep 10 03:03:20.235804 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:20.235772    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-842ba8c2-6cdf879494-pmjfc" event={"ID":"397bb1a5-c853-4334-8c10-8f4fde03bd9a","Type":"ContainerStarted","Data":"d108d86560db56f4440d0758d695588a9e9a2c648f38bf2e35a3ee79b1158f9e"}
Sep 10 03:03:20.235906 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:20.235829    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-842ba8c2-6cdf879494-pmjfc"
Sep 10 03:03:20.235906 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:20.235853    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-842ba8c2-6cdf879494-pmjfc" event={"ID":"397bb1a5-c853-4334-8c10-8f4fde03bd9a","Type":"ContainerStarted","Data":"0fefbb6cd809e56e30e1f15db21462b47a811a86f2e5f4d2cb204c5b3bf1da32"}
Sep 10 03:03:20.252307 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:20.252260    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-842ba8c2-c577fdbf-ftjm4" podStartSLOduration=1.252245422 podStartE2EDuration="1.252245422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:03:20.250710878 +0000 UTC m=+990.625796799" watchObservedRunningTime="2026-09-10 03:03:20.252245422 +0000 UTC m=+990.627331381"
Sep 10 03:03:20.269740 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:20.269689    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-842ba8c2-6cdf879494-pmjfc" podStartSLOduration=1.269672094 podStartE2EDuration="1.269672094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:03:20.267864683 +0000 UTC m=+990.642950605" watchObservedRunningTime="2026-09-10 03:03:20.269672094 +0000 UTC m=+990.644758022"
Sep 10 03:03:31.243550 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:31.243512    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-842ba8c2-c577fdbf-ftjm4"
Sep 10 03:03:31.244166 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:31.243592    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-842ba8c2-6cdf879494-pmjfc"
Sep 10 03:03:32.158086 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:32.158032    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:03:40.744058 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:40.744018    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:03:42.348539 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:42.348506    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:03:43.016268 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:43.016230    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:03:43.111827 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:43.111796    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:03:43.499827 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:43.499796    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:03:49.211491 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:49.211457    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:03:49.311784 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:49.311756    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:03:49.585035 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:49.584963    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-842ba8c2-5b2c2"]
Sep 10 03:03:49.588457 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:49.588441    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-842ba8c2-5b2c2"
Sep 10 03:03:49.590352 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:49.590337    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8jjl\""
Sep 10 03:03:49.594743 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:49.594721    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-842ba8c2-5b2c2"]
Sep 10 03:03:49.758705 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:49.758673    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da2efc9-2440-4796-880f-84931dd01f8b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-842ba8c2-5b2c2\" (UID: \"9da2efc9-2440-4796-880f-84931dd01f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-842ba8c2-5b2c2"
Sep 10 03:03:49.859252 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:49.859191    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da2efc9-2440-4796-880f-84931dd01f8b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-842ba8c2-5b2c2\" (UID: \"9da2efc9-2440-4796-880f-84931dd01f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-842ba8c2-5b2c2"
Sep 10 03:03:49.859767 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:49.859746    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da2efc9-2440-4796-880f-84931dd01f8b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-842ba8c2-5b2c2\" (UID: \"9da2efc9-2440-4796-880f-84931dd01f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-842ba8c2-5b2c2"
Sep 10 03:03:49.901393 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:49.901371    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-842ba8c2-5b2c2"
Sep 10 03:03:50.174794 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:50.174755    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-842ba8c2-5b2c2"]
Sep 10 03:03:50.421575 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:50.421520    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-842ba8c2-5b2c2" event={"ID":"9da2efc9-2440-4796-880f-84931dd01f8b","Type":"ContainerStarted","Data":"328401f7ba18540b576098d40cacdd3250d11fa9deca0c147b7db048f397f6fa"}
Sep 10 03:03:51.429177 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:51.429140    2558 generic.go:356] "Generic (PLEG): container finished" podID="9da2efc9-2440-4796-880f-84931dd01f8b" containerID="91df2f9a93a19607c68852ff59c34584db3f3d89b4a5432b8d388ccb800b6028" exitCode=0
Sep 10 03:03:51.429618 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:51.429229    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-842ba8c2-5b2c2" event={"ID":"9da2efc9-2440-4796-880f-84931dd01f8b","Type":"ContainerDied","Data":"91df2f9a93a19607c68852ff59c34584db3f3d89b4a5432b8d388ccb800b6028"}
Sep 10 03:03:52.582823 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:52.582801    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-842ba8c2-5b2c2"
Sep 10 03:03:52.681576 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:52.681552    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da2efc9-2440-4796-880f-84931dd01f8b-maas-api-service-ca\") pod \"9da2efc9-2440-4796-880f-84931dd01f8b\" (UID: \"9da2efc9-2440-4796-880f-84931dd01f8b\") "
Sep 10 03:03:52.681880 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:52.681855    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9da2efc9-2440-4796-880f-84931dd01f8b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9da2efc9-2440-4796-880f-84931dd01f8b" (UID: "9da2efc9-2440-4796-880f-84931dd01f8b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:03:52.782440 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:52.782418    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da2efc9-2440-4796-880f-84931dd01f8b-maas-api-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:03:53.442481 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:53.442454    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-842ba8c2-5b2c2"
Sep 10 03:03:53.442647 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:53.442458    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-842ba8c2-5b2c2" event={"ID":"9da2efc9-2440-4796-880f-84931dd01f8b","Type":"ContainerDied","Data":"328401f7ba18540b576098d40cacdd3250d11fa9deca0c147b7db048f397f6fa"}
Sep 10 03:03:53.442647 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:03:53.442571    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="328401f7ba18540b576098d40cacdd3250d11fa9deca0c147b7db048f397f6fa"
Sep 10 03:04:00.007797 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.007766    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:00.037004 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.036976    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-842ba8c2-c577fdbf-ftjm4"]
Sep 10 03:04:00.037256 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.037231    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-842ba8c2-c577fdbf-ftjm4" podUID="f7b30f51-d720-40fb-b419-6ed7700c42ad" containerName="payload-processing" containerID="cri-o://9c243d925b3da43117e2273648f009dcaf4b99a660bf72e0a77426bb8f5faf3c" gracePeriod=30
Sep 10 03:04:00.042507 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.042482    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-842ba8c2-6cdf879494-pmjfc"]
Sep 10 03:04:00.042670 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.042644    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-842ba8c2-6cdf879494-pmjfc" podUID="397bb1a5-c853-4334-8c10-8f4fde03bd9a" containerName="payload-pre-processing" containerID="cri-o://d108d86560db56f4440d0758d695588a9e9a2c648f38bf2e35a3ee79b1158f9e" gracePeriod=30
Sep 10 03:04:00.080708 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.080665    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:00.137574 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.137547    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:00.198990 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.198924    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-842ba8c2-5b2c2"]
Sep 10 03:04:00.201767 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.201736    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-842ba8c2-5b2c2"]
Sep 10 03:04:00.214017 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.213989    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:00.324607 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.324585    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-842ba8c2-c577fdbf-ftjm4"
Sep 10 03:04:00.327608 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.327590    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-842ba8c2-6cdf879494-pmjfc"
Sep 10 03:04:00.341324 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.341305    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkdcw\" (UniqueName: \"kubernetes.io/projected/f7b30f51-d720-40fb-b419-6ed7700c42ad-kube-api-access-xkdcw\") pod \"f7b30f51-d720-40fb-b419-6ed7700c42ad\" (UID: \"f7b30f51-d720-40fb-b419-6ed7700c42ad\") "
Sep 10 03:04:00.341406 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.341375    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/397bb1a5-c853-4334-8c10-8f4fde03bd9a-plugins-config-volume\") pod \"397bb1a5-c853-4334-8c10-8f4fde03bd9a\" (UID: \"397bb1a5-c853-4334-8c10-8f4fde03bd9a\") "
Sep 10 03:04:00.341492 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.341472    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b30f51-d720-40fb-b419-6ed7700c42ad-plugins-config-volume\") pod \"f7b30f51-d720-40fb-b419-6ed7700c42ad\" (UID: \"f7b30f51-d720-40fb-b419-6ed7700c42ad\") "
Sep 10 03:04:00.341544 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.341529    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsp8l\" (UniqueName: \"kubernetes.io/projected/397bb1a5-c853-4334-8c10-8f4fde03bd9a-kube-api-access-zsp8l\") pod \"397bb1a5-c853-4334-8c10-8f4fde03bd9a\" (UID: \"397bb1a5-c853-4334-8c10-8f4fde03bd9a\") "
Sep 10 03:04:00.341809 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.341672    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/397bb1a5-c853-4334-8c10-8f4fde03bd9a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "397bb1a5-c853-4334-8c10-8f4fde03bd9a" (UID: "397bb1a5-c853-4334-8c10-8f4fde03bd9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:04:00.341899 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.341855    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7b30f51-d720-40fb-b419-6ed7700c42ad-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f7b30f51-d720-40fb-b419-6ed7700c42ad" (UID: "f7b30f51-d720-40fb-b419-6ed7700c42ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:04:00.352707 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.352666    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/397bb1a5-c853-4334-8c10-8f4fde03bd9a-kube-api-access-zsp8l" (OuterVolumeSpecName: "kube-api-access-zsp8l") pod "397bb1a5-c853-4334-8c10-8f4fde03bd9a" (UID: "397bb1a5-c853-4334-8c10-8f4fde03bd9a"). InnerVolumeSpecName "kube-api-access-zsp8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:04:00.352841 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.352724    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b30f51-d720-40fb-b419-6ed7700c42ad-kube-api-access-xkdcw" (OuterVolumeSpecName: "kube-api-access-xkdcw") pod "f7b30f51-d720-40fb-b419-6ed7700c42ad" (UID: "f7b30f51-d720-40fb-b419-6ed7700c42ad"). InnerVolumeSpecName "kube-api-access-xkdcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:04:00.442843 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.442819    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/397bb1a5-c853-4334-8c10-8f4fde03bd9a-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:04:00.442843 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.442840    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b30f51-d720-40fb-b419-6ed7700c42ad-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:04:00.442983 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.442850    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsp8l\" (UniqueName: \"kubernetes.io/projected/397bb1a5-c853-4334-8c10-8f4fde03bd9a-kube-api-access-zsp8l\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:04:00.442983 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.442860    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkdcw\" (UniqueName: \"kubernetes.io/projected/f7b30f51-d720-40fb-b419-6ed7700c42ad-kube-api-access-xkdcw\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:04:00.486124 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.486085    2558 generic.go:356] "Generic (PLEG): container finished" podID="397bb1a5-c853-4334-8c10-8f4fde03bd9a" containerID="d108d86560db56f4440d0758d695588a9e9a2c648f38bf2e35a3ee79b1158f9e" exitCode=0
Sep 10 03:04:00.486249 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.486150    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-842ba8c2-6cdf879494-pmjfc"
Sep 10 03:04:00.486249 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.486173    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-842ba8c2-6cdf879494-pmjfc" event={"ID":"397bb1a5-c853-4334-8c10-8f4fde03bd9a","Type":"ContainerDied","Data":"d108d86560db56f4440d0758d695588a9e9a2c648f38bf2e35a3ee79b1158f9e"}
Sep 10 03:04:00.486370 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.486265    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-842ba8c2-6cdf879494-pmjfc" event={"ID":"397bb1a5-c853-4334-8c10-8f4fde03bd9a","Type":"ContainerDied","Data":"0fefbb6cd809e56e30e1f15db21462b47a811a86f2e5f4d2cb204c5b3bf1da32"}
Sep 10 03:04:00.486370 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.486299    2558 scope.go:117] "RemoveContainer" containerID="d108d86560db56f4440d0758d695588a9e9a2c648f38bf2e35a3ee79b1158f9e"
Sep 10 03:04:00.487508 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.487491    2558 generic.go:356] "Generic (PLEG): container finished" podID="f7b30f51-d720-40fb-b419-6ed7700c42ad" containerID="9c243d925b3da43117e2273648f009dcaf4b99a660bf72e0a77426bb8f5faf3c" exitCode=0
Sep 10 03:04:00.487581 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.487554    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-842ba8c2-c577fdbf-ftjm4" event={"ID":"f7b30f51-d720-40fb-b419-6ed7700c42ad","Type":"ContainerDied","Data":"9c243d925b3da43117e2273648f009dcaf4b99a660bf72e0a77426bb8f5faf3c"}
Sep 10 03:04:00.487581 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.487568    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-842ba8c2-c577fdbf-ftjm4"
Sep 10 03:04:00.487694 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.487592    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-842ba8c2-c577fdbf-ftjm4" event={"ID":"f7b30f51-d720-40fb-b419-6ed7700c42ad","Type":"ContainerDied","Data":"c6860634984945f06ded5605ed5ed70dee2e02576c647178c63dee2a0c5e16eb"}
Sep 10 03:04:00.495618 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.495602    2558 scope.go:117] "RemoveContainer" containerID="d108d86560db56f4440d0758d695588a9e9a2c648f38bf2e35a3ee79b1158f9e"
Sep 10 03:04:00.495896 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:04:00.495879    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d108d86560db56f4440d0758d695588a9e9a2c648f38bf2e35a3ee79b1158f9e\": container with ID starting with d108d86560db56f4440d0758d695588a9e9a2c648f38bf2e35a3ee79b1158f9e not found: ID does not exist" containerID="d108d86560db56f4440d0758d695588a9e9a2c648f38bf2e35a3ee79b1158f9e"
Sep 10 03:04:00.495963 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.495902    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d108d86560db56f4440d0758d695588a9e9a2c648f38bf2e35a3ee79b1158f9e"} err="failed to get container status \"d108d86560db56f4440d0758d695588a9e9a2c648f38bf2e35a3ee79b1158f9e\": rpc error: code = NotFound desc = could not find container \"d108d86560db56f4440d0758d695588a9e9a2c648f38bf2e35a3ee79b1158f9e\": container with ID starting with d108d86560db56f4440d0758d695588a9e9a2c648f38bf2e35a3ee79b1158f9e not found: ID does not exist"
Sep 10 03:04:00.495963 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.495917    2558 scope.go:117] "RemoveContainer" containerID="9c243d925b3da43117e2273648f009dcaf4b99a660bf72e0a77426bb8f5faf3c"
Sep 10 03:04:00.504213 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.504196    2558 scope.go:117] "RemoveContainer" containerID="9c243d925b3da43117e2273648f009dcaf4b99a660bf72e0a77426bb8f5faf3c"
Sep 10 03:04:00.504448 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:04:00.504429    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c243d925b3da43117e2273648f009dcaf4b99a660bf72e0a77426bb8f5faf3c\": container with ID starting with 9c243d925b3da43117e2273648f009dcaf4b99a660bf72e0a77426bb8f5faf3c not found: ID does not exist" containerID="9c243d925b3da43117e2273648f009dcaf4b99a660bf72e0a77426bb8f5faf3c"
Sep 10 03:04:00.504503 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.504454    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c243d925b3da43117e2273648f009dcaf4b99a660bf72e0a77426bb8f5faf3c"} err="failed to get container status \"9c243d925b3da43117e2273648f009dcaf4b99a660bf72e0a77426bb8f5faf3c\": rpc error: code = NotFound desc = could not find container \"9c243d925b3da43117e2273648f009dcaf4b99a660bf72e0a77426bb8f5faf3c\": container with ID starting with 9c243d925b3da43117e2273648f009dcaf4b99a660bf72e0a77426bb8f5faf3c not found: ID does not exist"
Sep 10 03:04:00.509491 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.509466    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-842ba8c2-6cdf879494-pmjfc"]
Sep 10 03:04:00.513136 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.513115    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-842ba8c2-6cdf879494-pmjfc"]
Sep 10 03:04:00.523761 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.523739    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-842ba8c2-c577fdbf-ftjm4"]
Sep 10 03:04:00.525748 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.525726    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-842ba8c2-c577fdbf-ftjm4"]
Sep 10 03:04:00.573626 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:00.573563    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:02.070207 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:02.070176    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="397bb1a5-c853-4334-8c10-8f4fde03bd9a" path="/var/lib/kubelet/pods/397bb1a5-c853-4334-8c10-8f4fde03bd9a/volumes"
Sep 10 03:04:02.070585 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:02.070501    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9da2efc9-2440-4796-880f-84931dd01f8b" path="/var/lib/kubelet/pods/9da2efc9-2440-4796-880f-84931dd01f8b/volumes"
Sep 10 03:04:02.070812 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:02.070799    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7b30f51-d720-40fb-b419-6ed7700c42ad" path="/var/lib/kubelet/pods/f7b30f51-d720-40fb-b419-6ed7700c42ad/volumes"
Sep 10 03:04:05.371008 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:05.370965    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-af329f-m46vn"]
Sep 10 03:04:05.371438 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:05.371423    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="397bb1a5-c853-4334-8c10-8f4fde03bd9a" containerName="payload-pre-processing"
Sep 10 03:04:05.371438 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:05.371440    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="397bb1a5-c853-4334-8c10-8f4fde03bd9a" containerName="payload-pre-processing"
Sep 10 03:04:05.371529 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:05.371454    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7b30f51-d720-40fb-b419-6ed7700c42ad" containerName="payload-processing"
Sep 10 03:04:05.371529 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:05.371464    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7b30f51-d720-40fb-b419-6ed7700c42ad" containerName="payload-processing"
Sep 10 03:04:05.371529 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:05.371489    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9da2efc9-2440-4796-880f-84931dd01f8b" containerName="revoke-keys"
Sep 10 03:04:05.371529 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:05.371495    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9da2efc9-2440-4796-880f-84931dd01f8b" containerName="revoke-keys"
Sep 10 03:04:05.371655 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:05.371565    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9da2efc9-2440-4796-880f-84931dd01f8b" containerName="revoke-keys"
Sep 10 03:04:05.371655 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:05.371574    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7b30f51-d720-40fb-b419-6ed7700c42ad" containerName="payload-processing"
Sep 10 03:04:05.371655 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:05.371582    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="397bb1a5-c853-4334-8c10-8f4fde03bd9a" containerName="payload-pre-processing"
Sep 10 03:04:05.376377 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:05.376357    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-af329f-m46vn"
Sep 10 03:04:05.378869 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:05.378820    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8jjl\""
Sep 10 03:04:05.381051 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:05.381025    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-af329f-m46vn"]
Sep 10 03:04:05.480228 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:05.480201    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae10cad-7a8f-4782-ade6-0afd0f8731f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-af329f-m46vn\" (UID: \"2ae10cad-7a8f-4782-ade6-0afd0f8731f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-af329f-m46vn"
Sep 10 03:04:05.532503 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:05.532473    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:05.581431 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:05.581394    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae10cad-7a8f-4782-ade6-0afd0f8731f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-af329f-m46vn\" (UID: \"2ae10cad-7a8f-4782-ade6-0afd0f8731f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-af329f-m46vn"
Sep 10 03:04:05.581967 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:05.581948    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae10cad-7a8f-4782-ade6-0afd0f8731f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-af329f-m46vn\" (UID: \"2ae10cad-7a8f-4782-ade6-0afd0f8731f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-af329f-m46vn"
Sep 10 03:04:05.689217 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:05.689150    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-af329f-m46vn"
Sep 10 03:04:05.961088 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:05.961001    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-af329f-m46vn"]
Sep 10 03:04:05.964161 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:04:05.964134    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae10cad_7a8f_4782_ade6_0afd0f8731f6.slice/crio-ee2e296ff0a56b0d4fecca404d8aa990cf32a79f3e447d2c86d3383f8a322191 WatchSource:0}: Error finding container ee2e296ff0a56b0d4fecca404d8aa990cf32a79f3e447d2c86d3383f8a322191: Status 404 returned error can't find the container with id ee2e296ff0a56b0d4fecca404d8aa990cf32a79f3e447d2c86d3383f8a322191
Sep 10 03:04:06.525448 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:06.525415    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-af329f-m46vn" event={"ID":"2ae10cad-7a8f-4782-ade6-0afd0f8731f6","Type":"ContainerStarted","Data":"cc42b57a93366c59e033e17f77e9e634e0ca1c8d11a331fd0068438158377793"}
Sep 10 03:04:06.525822 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:06.525473    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-af329f-m46vn" event={"ID":"2ae10cad-7a8f-4782-ade6-0afd0f8731f6","Type":"ContainerStarted","Data":"ee2e296ff0a56b0d4fecca404d8aa990cf32a79f3e447d2c86d3383f8a322191"}
Sep 10 03:04:06.541072 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:06.541019    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-af329f-m46vn" podStartSLOduration=1.540978907 podStartE2EDuration="1.540978907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:04:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:04:06.539501699 +0000 UTC m=+1036.914587620" watchObservedRunningTime="2026-09-10 03:04:06.540978907 +0000 UTC m=+1036.916064828"
Sep 10 03:04:06.712700 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:06.712663    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:06.770227 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:06.770196    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:06.992639 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:06.992606    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:07.344749 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:07.344669    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:07.444060 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:07.444008    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:07.533145 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:07.533085    2558 generic.go:356] "Generic (PLEG): container finished" podID="2ae10cad-7a8f-4782-ade6-0afd0f8731f6" containerID="cc42b57a93366c59e033e17f77e9e634e0ca1c8d11a331fd0068438158377793" exitCode=0
Sep 10 03:04:07.533565 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:07.533174    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-af329f-m46vn" event={"ID":"2ae10cad-7a8f-4782-ade6-0afd0f8731f6","Type":"ContainerDied","Data":"cc42b57a93366c59e033e17f77e9e634e0ca1c8d11a331fd0068438158377793"}
Sep 10 03:04:07.548272 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:07.548240    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:08.671365 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:08.671336    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-af329f-m46vn"
Sep 10 03:04:08.711098 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:08.711069    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae10cad-7a8f-4782-ade6-0afd0f8731f6-maas-api-service-ca\") pod \"2ae10cad-7a8f-4782-ade6-0afd0f8731f6\" (UID: \"2ae10cad-7a8f-4782-ade6-0afd0f8731f6\") "
Sep 10 03:04:08.711416 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:08.711391    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae10cad-7a8f-4782-ade6-0afd0f8731f6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2ae10cad-7a8f-4782-ade6-0afd0f8731f6" (UID: "2ae10cad-7a8f-4782-ade6-0afd0f8731f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:04:08.812332 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:08.812260    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae10cad-7a8f-4782-ade6-0afd0f8731f6-maas-api-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:04:09.545982 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:09.545949    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-af329f-m46vn"
Sep 10 03:04:09.546174 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:09.545990    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-af329f-m46vn" event={"ID":"2ae10cad-7a8f-4782-ade6-0afd0f8731f6","Type":"ContainerDied","Data":"ee2e296ff0a56b0d4fecca404d8aa990cf32a79f3e447d2c86d3383f8a322191"}
Sep 10 03:04:09.546174 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:09.546040    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee2e296ff0a56b0d4fecca404d8aa990cf32a79f3e447d2c86d3383f8a322191"
Sep 10 03:04:13.061967 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:13.061931    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:13.171181 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:13.171155    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:13.262325 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:13.262290    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:13.424586 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:13.424507    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:15.854331 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:15.854298    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:15.916046 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:15.916011    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:15.974539 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:15.974486    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:16.037089 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:16.037051    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-af329f-m46vn"]
Sep 10 03:04:16.039873 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:16.039842    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:16.040374 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:16.040350    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-af329f-m46vn"]
Sep 10 03:04:16.069987 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:16.069957    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ae10cad-7a8f-4782-ade6-0afd0f8731f6" path="/var/lib/kubelet/pods/2ae10cad-7a8f-4782-ade6-0afd0f8731f6/volumes"
Sep 10 03:04:16.417257 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:16.417222    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:19.073207 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:19.073177    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:19.171932 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:19.171902    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:19.885219 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:19.885192    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:20.074072 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:20.074041    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:20.375925 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:20.375898    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:20.667047 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:20.666958    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:20.768182 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:20.768152    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:21.369700 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:21.369655    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:21.570064 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:21.570037    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:23.076813 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:23.076780    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:23.787374 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:23.787333    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:23.887162 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:23.887128    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:24.226218 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:24.226191    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:24.630166 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:24.630082    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:24.951574 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:24.951542    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:25.052203 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:25.052171    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:25.151994 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:25.151966    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:25.412708 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:25.412635    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:25.531553 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:25.531522    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1e0789-77bbc99758-4fsxv"]
Sep 10 03:04:25.531927 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:25.531914    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ae10cad-7a8f-4782-ade6-0afd0f8731f6" containerName="revoke-keys"
Sep 10 03:04:25.531978 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:25.531928    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ae10cad-7a8f-4782-ade6-0afd0f8731f6" containerName="revoke-keys"
Sep 10 03:04:25.532015 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:25.531989    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ae10cad-7a8f-4782-ade6-0afd0f8731f6" containerName="revoke-keys"
Sep 10 03:04:25.535175 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:25.535158    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1e0789-77bbc99758-4fsxv"
Sep 10 03:04:25.537714 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:25.537690    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-1e0789-dockercfg-xhcrx\""
Sep 10 03:04:25.537970 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:25.537954    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-1e0789\""
Sep 10 03:04:25.559006 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:25.558977    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1e0789-77bbc99758-4fsxv"]
Sep 10 03:04:25.570447 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:25.570421    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1e0789-6595f797b5-wr8p8"]
Sep 10 03:04:25.573815 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:25.573797    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1e0789-6595f797b5-wr8p8"
Sep 10 03:04:25.606499 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:25.606478    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1e0789-6595f797b5-wr8p8"]
Sep 10 03:04:25.643605 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:25.643574    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfzw4\" (UniqueName: \"kubernetes.io/projected/f40da815-568b-44f8-90eb-1bcaf4df063c-kube-api-access-tfzw4\") pod \"payload-pre-processing-e2e-shared-a-w2-1e0789-77bbc99758-4fsxv\" (UID: \"f40da815-568b-44f8-90eb-1bcaf4df063c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1e0789-77bbc99758-4fsxv"
Sep 10 03:04:25.643706 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:25.643635    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjv4r\" (UniqueName: \"kubernetes.io/projected/afdadca6-01cc-481a-bb19-425c3b10d50f-kube-api-access-bjv4r\") pod \"payload-processing-e2e-shared-a-w2-1e0789-6595f797b5-wr8p8\" (UID: \"afdadca6-01cc-481a-bb19-425c3b10d50f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1e0789-6595f797b5-wr8p8"
Sep 10 03:04:25.643753 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:25.643715    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afdadca6-01cc-481a-bb19-425c3b10d50f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-1e0789-6595f797b5-wr8p8\" (UID: \"afdadca6-01cc-481a-bb19-425c3b10d50f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1e0789-6595f797b5-wr8p8"
Sep 10 03:04:25.643794 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:25.643754    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40da815-568b-44f8-90eb-1bcaf4df063c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-1e0789-77bbc99758-4fsxv\" (UID: \"f40da815-568b-44f8-90eb-1bcaf4df063c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1e0789-77bbc99758-4fsxv"
Sep 10 03:04:25.651193 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:25.651175    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:25.744390 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:25.744367    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjv4r\" (UniqueName: \"kubernetes.io/projected/afdadca6-01cc-481a-bb19-425c3b10d50f-kube-api-access-bjv4r\") pod \"payload-processing-e2e-shared-a-w2-1e0789-6595f797b5-wr8p8\" (UID: \"afdadca6-01cc-481a-bb19-425c3b10d50f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1e0789-6595f797b5-wr8p8"
Sep 10 03:04:25.744499 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:25.744400    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afdadca6-01cc-481a-bb19-425c3b10d50f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-1e0789-6595f797b5-wr8p8\" (UID: \"afdadca6-01cc-481a-bb19-425c3b10d50f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1e0789-6595f797b5-wr8p8"
Sep 10 03:04:25.744499 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:25.744421    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40da815-568b-44f8-90eb-1bcaf4df063c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-1e0789-77bbc99758-4fsxv\" (UID: \"f40da815-568b-44f8-90eb-1bcaf4df063c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1e0789-77bbc99758-4fsxv"
Sep 10 03:04:25.744589 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:25.744512    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfzw4\" (UniqueName: \"kubernetes.io/projected/f40da815-568b-44f8-90eb-1bcaf4df063c-kube-api-access-tfzw4\") pod \"payload-pre-processing-e2e-shared-a-w2-1e0789-77bbc99758-4fsxv\" (UID: \"f40da815-568b-44f8-90eb-1bcaf4df063c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1e0789-77bbc99758-4fsxv"
Sep 10 03:04:25.745143 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:25.745125    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40da815-568b-44f8-90eb-1bcaf4df063c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-1e0789-77bbc99758-4fsxv\" (UID: \"f40da815-568b-44f8-90eb-1bcaf4df063c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1e0789-77bbc99758-4fsxv"
Sep 10 03:04:25.745191 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:25.745148    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afdadca6-01cc-481a-bb19-425c3b10d50f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-1e0789-6595f797b5-wr8p8\" (UID: \"afdadca6-01cc-481a-bb19-425c3b10d50f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1e0789-6595f797b5-wr8p8"
Sep 10 03:04:25.752115 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:25.752073    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfzw4\" (UniqueName: \"kubernetes.io/projected/f40da815-568b-44f8-90eb-1bcaf4df063c-kube-api-access-tfzw4\") pod \"payload-pre-processing-e2e-shared-a-w2-1e0789-77bbc99758-4fsxv\" (UID: \"f40da815-568b-44f8-90eb-1bcaf4df063c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1e0789-77bbc99758-4fsxv"
Sep 10 03:04:25.752207 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:25.752154    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjv4r\" (UniqueName: \"kubernetes.io/projected/afdadca6-01cc-481a-bb19-425c3b10d50f-kube-api-access-bjv4r\") pod \"payload-processing-e2e-shared-a-w2-1e0789-6595f797b5-wr8p8\" (UID: \"afdadca6-01cc-481a-bb19-425c3b10d50f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1e0789-6595f797b5-wr8p8"
Sep 10 03:04:25.845199 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:25.845179    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1e0789-77bbc99758-4fsxv"
Sep 10 03:04:25.884240 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:25.884211    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1e0789-6595f797b5-wr8p8"
Sep 10 03:04:26.127308 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:26.124956    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1e0789-77bbc99758-4fsxv"]
Sep 10 03:04:26.128202 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:04:26.128003    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf40da815_568b_44f8_90eb_1bcaf4df063c.slice/crio-c42ed975eaeb2a9e4c5c3e386ff910e4657f017012d4c3ab289d8078a6b39f03 WatchSource:0}: Error finding container c42ed975eaeb2a9e4c5c3e386ff910e4657f017012d4c3ab289d8078a6b39f03: Status 404 returned error can't find the container with id c42ed975eaeb2a9e4c5c3e386ff910e4657f017012d4c3ab289d8078a6b39f03
Sep 10 03:04:26.167854 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:26.167830    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1e0789-6595f797b5-wr8p8"]
Sep 10 03:04:26.170650 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:04:26.170621    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafdadca6_01cc_481a_bb19_425c3b10d50f.slice/crio-dd17690c763aafdd005323f552274d25e0bad4e4cd7877a150025b3f1273b6ba WatchSource:0}: Error finding container dd17690c763aafdd005323f552274d25e0bad4e4cd7877a150025b3f1273b6ba: Status 404 returned error can't find the container with id dd17690c763aafdd005323f552274d25e0bad4e4cd7877a150025b3f1273b6ba
Sep 10 03:04:26.648384 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:26.648348    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1e0789-77bbc99758-4fsxv" event={"ID":"f40da815-568b-44f8-90eb-1bcaf4df063c","Type":"ContainerStarted","Data":"7d05feb393af6ffeb5eea75a66eef91009e8b56649c5c881d9f2d465cf9b8f4d"}
Sep 10 03:04:26.648897 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:26.648415    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1e0789-77bbc99758-4fsxv"
Sep 10 03:04:26.648897 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:26.648439    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1e0789-77bbc99758-4fsxv" event={"ID":"f40da815-568b-44f8-90eb-1bcaf4df063c","Type":"ContainerStarted","Data":"c42ed975eaeb2a9e4c5c3e386ff910e4657f017012d4c3ab289d8078a6b39f03"}
Sep 10 03:04:26.650007 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:26.649983    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1e0789-6595f797b5-wr8p8" event={"ID":"afdadca6-01cc-481a-bb19-425c3b10d50f","Type":"ContainerStarted","Data":"5c67abab628a2fcde826306ace0c1f365e55a447582f24e07f7efbe1865906fa"}
Sep 10 03:04:26.650156 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:26.650028    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1e0789-6595f797b5-wr8p8" event={"ID":"afdadca6-01cc-481a-bb19-425c3b10d50f","Type":"ContainerStarted","Data":"dd17690c763aafdd005323f552274d25e0bad4e4cd7877a150025b3f1273b6ba"}
Sep 10 03:04:26.650156 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:26.650066    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1e0789-6595f797b5-wr8p8"
Sep 10 03:04:26.662857 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:26.662803    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1e0789-77bbc99758-4fsxv" podStartSLOduration=1.6627858880000002 podStartE2EDuration="1.662785888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:04:26.661952746 +0000 UTC m=+1057.037038670" watchObservedRunningTime="2026-09-10 03:04:26.662785888 +0000 UTC m=+1057.037871813"
Sep 10 03:04:26.676122 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:26.676060    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1e0789-6595f797b5-wr8p8" podStartSLOduration=1.6760444460000001 podStartE2EDuration="1.676044446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:04:26.675646416 +0000 UTC m=+1057.050732338" watchObservedRunningTime="2026-09-10 03:04:26.676044446 +0000 UTC m=+1057.051130369"
Sep 10 03:04:31.644547 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:31.644509    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:31.745429 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:31.745401    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:37.659221 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:37.659188    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1e0789-77bbc99758-4fsxv"
Sep 10 03:04:37.659681 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:37.659266    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1e0789-6595f797b5-wr8p8"
Sep 10 03:04:48.841168 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:48.841065    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:49.167386 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:49.167296    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:49.516655 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:49.516621    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:49.631869 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:49.631839    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:52.983873 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:52.983832    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:53.077564 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:53.077532    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:53.961567 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:53.961535    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:54.684543 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:54.684508    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:54.792496 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:54.792454    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:54.886001 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:54.885972    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:55.237693 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:55.237657    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:55.414244 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:55.414204    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:58.231844 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:58.231811    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:58.500019 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:58.499937    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:58.971467 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:58.971441    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:59.289481 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:59.289379    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:04:59.737976 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:04:59.737945    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:05:00.368471 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:00.368441    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:05:00.468879 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:00.468857    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:05:00.576221 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:00.576191    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:05:17.990537 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:17.990500    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:05:22.946793 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:22.946762    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8012-9085c7"]
Sep 10 03:05:22.950324 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:22.950308    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8012-9085c7"
Sep 10 03:05:22.952526 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:22.952497    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mtn2b\""
Sep 10 03:05:22.956973 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:22.956944    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8012-9085c7"]
Sep 10 03:05:22.993123 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:22.993078    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c66s8\" (UniqueName: \"kubernetes.io/projected/5ecf6314-803a-4b07-bc9a-55cc05e7bd5d-kube-api-access-c66s8\") pod \"test-curl-8012-9085c7\" (UID: \"5ecf6314-803a-4b07-bc9a-55cc05e7bd5d\") " pod="opendatahub/test-curl-8012-9085c7"
Sep 10 03:05:23.093771 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:23.093745    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c66s8\" (UniqueName: \"kubernetes.io/projected/5ecf6314-803a-4b07-bc9a-55cc05e7bd5d-kube-api-access-c66s8\") pod \"test-curl-8012-9085c7\" (UID: \"5ecf6314-803a-4b07-bc9a-55cc05e7bd5d\") " pod="opendatahub/test-curl-8012-9085c7"
Sep 10 03:05:23.101076 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:23.101050    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c66s8\" (UniqueName: \"kubernetes.io/projected/5ecf6314-803a-4b07-bc9a-55cc05e7bd5d-kube-api-access-c66s8\") pod \"test-curl-8012-9085c7\" (UID: \"5ecf6314-803a-4b07-bc9a-55cc05e7bd5d\") " pod="opendatahub/test-curl-8012-9085c7"
Sep 10 03:05:23.265952 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:23.265928    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8012-9085c7"
Sep 10 03:05:23.533144 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:23.533119    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8012-9085c7"]
Sep 10 03:05:23.534920 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:05:23.534891    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ecf6314_803a_4b07_bc9a_55cc05e7bd5d.slice/crio-f4ec2182a0652fdbe1204721215314779c907a527471cf496ed7cc5fa193883f WatchSource:0}: Error finding container f4ec2182a0652fdbe1204721215314779c907a527471cf496ed7cc5fa193883f: Status 404 returned error can't find the container with id f4ec2182a0652fdbe1204721215314779c907a527471cf496ed7cc5fa193883f
Sep 10 03:05:23.536878 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:23.536857    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 03:05:23.998593 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:23.998561    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8012-9085c7" event={"ID":"5ecf6314-803a-4b07-bc9a-55cc05e7bd5d","Type":"ContainerStarted","Data":"f4ec2182a0652fdbe1204721215314779c907a527471cf496ed7cc5fa193883f"}
Sep 10 03:05:25.006334 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:25.006303    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8012-9085c7" event={"ID":"5ecf6314-803a-4b07-bc9a-55cc05e7bd5d","Type":"ContainerStarted","Data":"fef8f130f1308829e0ce7c858597a404ca8da668cf2071b4fc49bd544f5cc099"}
Sep 10 03:05:25.021349 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:25.021298    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8012-9085c7" podStartSLOduration=2.492295254 podStartE2EDuration="3.021283387s" totalImagesPullingTime="528.988133ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:05:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:05:25.018794304 +0000 UTC m=+1115.393880224" watchObservedRunningTime="2026-09-10 03:05:25.021283387 +0000 UTC m=+1115.396369307"
Sep 10 03:05:25.632384 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:25.632351    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8012-9085c7"]
Sep 10 03:05:25.635436 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:25.635413    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8012-9085c7"]
Sep 10 03:05:25.753498 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:25.753472    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8012-438d65"]
Sep 10 03:05:25.753870 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:25.753857    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ecf6314-803a-4b07-bc9a-55cc05e7bd5d" containerName="test-curl-8012-9085c7"
Sep 10 03:05:25.753914 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:25.753871    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ecf6314-803a-4b07-bc9a-55cc05e7bd5d" containerName="test-curl-8012-9085c7"
Sep 10 03:05:25.753950 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:25.753930    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ecf6314-803a-4b07-bc9a-55cc05e7bd5d" containerName="test-curl-8012-9085c7"
Sep 10 03:05:25.757686 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:25.757658    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8012-438d65"
Sep 10 03:05:25.761200 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:25.760818    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8012-438d65"]
Sep 10 03:05:25.815891 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:25.815868    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plgdt\" (UniqueName: \"kubernetes.io/projected/74c6e35d-6407-4259-96e0-9e5b06a9f331-kube-api-access-plgdt\") pod \"test-curl-8012-438d65\" (UID: \"74c6e35d-6407-4259-96e0-9e5b06a9f331\") " pod="opendatahub/test-curl-8012-438d65"
Sep 10 03:05:25.917331 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:25.917308    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plgdt\" (UniqueName: \"kubernetes.io/projected/74c6e35d-6407-4259-96e0-9e5b06a9f331-kube-api-access-plgdt\") pod \"test-curl-8012-438d65\" (UID: \"74c6e35d-6407-4259-96e0-9e5b06a9f331\") " pod="opendatahub/test-curl-8012-438d65"
Sep 10 03:05:25.924408 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:25.924390    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plgdt\" (UniqueName: \"kubernetes.io/projected/74c6e35d-6407-4259-96e0-9e5b06a9f331-kube-api-access-plgdt\") pod \"test-curl-8012-438d65\" (UID: \"74c6e35d-6407-4259-96e0-9e5b06a9f331\") " pod="opendatahub/test-curl-8012-438d65"
Sep 10 03:05:26.012480 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:05:26.012414    2558 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8012-9085c7" container="test-curl-8012-9085c7"
Sep 10 03:05:26.068486 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:26.068463    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8012-438d65"
Sep 10 03:05:26.334952 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:26.334928    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8012-438d65"]
Sep 10 03:05:26.337162 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:05:26.337134    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74c6e35d_6407_4259_96e0_9e5b06a9f331.slice/crio-2b27eed4ba1b2ab59c6697bb3531b0836af967d0f433430ff3faad8d59c6e8fb WatchSource:0}: Error finding container 2b27eed4ba1b2ab59c6697bb3531b0836af967d0f433430ff3faad8d59c6e8fb: Status 404 returned error can't find the container with id 2b27eed4ba1b2ab59c6697bb3531b0836af967d0f433430ff3faad8d59c6e8fb
Sep 10 03:05:27.019992 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:27.019956    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8012-438d65" event={"ID":"74c6e35d-6407-4259-96e0-9e5b06a9f331","Type":"ContainerStarted","Data":"32124bf98c47b4183bb8cd2d2cb4dd18ea331c387bd579ac28ebc7982e1caa48"}
Sep 10 03:05:27.020470 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:27.020005    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8012-438d65" event={"ID":"74c6e35d-6407-4259-96e0-9e5b06a9f331","Type":"ContainerStarted","Data":"2b27eed4ba1b2ab59c6697bb3531b0836af967d0f433430ff3faad8d59c6e8fb"}
Sep 10 03:05:27.020470 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:27.020038    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8012-9085c7" podUID="5ecf6314-803a-4b07-bc9a-55cc05e7bd5d" containerName="test-curl-8012-9085c7" containerID="cri-o://fef8f130f1308829e0ce7c858597a404ca8da668cf2071b4fc49bd544f5cc099" gracePeriod=2
Sep 10 03:05:27.033486 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:27.033442    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8012-438d65" podStartSLOduration=2.033427146 podStartE2EDuration="2.033427146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:05:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:05:27.031997057 +0000 UTC m=+1117.407082974" watchObservedRunningTime="2026-09-10 03:05:27.033427146 +0000 UTC m=+1117.408513067"
Sep 10 03:05:27.633671 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:27.633473    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8012-438d65"]
Sep 10 03:05:27.637441 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:27.637413    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8012-438d65"]
Sep 10 03:05:28.029615 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:05:28.029589    2558 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8012-438d65" container="test-curl-8012-438d65"
Sep 10 03:05:28.246120 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:28.246074    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:05:28.540425 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:28.540382    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:05:28.653001 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:28.652967    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:05:29.035563 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:29.035514    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8012-438d65" podUID="74c6e35d-6407-4259-96e0-9e5b06a9f331" containerName="test-curl-8012-438d65" containerID="cri-o://32124bf98c47b4183bb8cd2d2cb4dd18ea331c387bd579ac28ebc7982e1caa48" gracePeriod=2
Sep 10 03:05:29.196865 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:29.196838    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8012-9085c7"
Sep 10 03:05:29.245582 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:29.245558    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c66s8\" (UniqueName: \"kubernetes.io/projected/5ecf6314-803a-4b07-bc9a-55cc05e7bd5d-kube-api-access-c66s8\") pod \"5ecf6314-803a-4b07-bc9a-55cc05e7bd5d\" (UID: \"5ecf6314-803a-4b07-bc9a-55cc05e7bd5d\") "
Sep 10 03:05:29.247581 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:29.247550    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ecf6314-803a-4b07-bc9a-55cc05e7bd5d-kube-api-access-c66s8" (OuterVolumeSpecName: "kube-api-access-c66s8") pod "5ecf6314-803a-4b07-bc9a-55cc05e7bd5d" (UID: "5ecf6314-803a-4b07-bc9a-55cc05e7bd5d"). InnerVolumeSpecName "kube-api-access-c66s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:05:29.346599 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:29.346526    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c66s8\" (UniqueName: \"kubernetes.io/projected/5ecf6314-803a-4b07-bc9a-55cc05e7bd5d-kube-api-access-c66s8\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:05:30.043781 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:30.043749    2558 generic.go:356] "Generic (PLEG): container finished" podID="5ecf6314-803a-4b07-bc9a-55cc05e7bd5d" containerID="fef8f130f1308829e0ce7c858597a404ca8da668cf2071b4fc49bd544f5cc099" exitCode=137
Sep 10 03:05:30.044181 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:30.043791    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8012-9085c7"
Sep 10 03:05:30.044181 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:30.043825    2558 scope.go:117] "RemoveContainer" containerID="fef8f130f1308829e0ce7c858597a404ca8da668cf2071b4fc49bd544f5cc099"
Sep 10 03:05:30.053415 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:30.053396    2558 scope.go:117] "RemoveContainer" containerID="fef8f130f1308829e0ce7c858597a404ca8da668cf2071b4fc49bd544f5cc099"
Sep 10 03:05:30.053670 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:05:30.053652    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fef8f130f1308829e0ce7c858597a404ca8da668cf2071b4fc49bd544f5cc099\": container with ID starting with fef8f130f1308829e0ce7c858597a404ca8da668cf2071b4fc49bd544f5cc099 not found: ID does not exist" containerID="fef8f130f1308829e0ce7c858597a404ca8da668cf2071b4fc49bd544f5cc099"
Sep 10 03:05:30.053715 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:30.053678    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fef8f130f1308829e0ce7c858597a404ca8da668cf2071b4fc49bd544f5cc099"} err="failed to get container status \"fef8f130f1308829e0ce7c858597a404ca8da668cf2071b4fc49bd544f5cc099\": rpc error: code = NotFound desc = could not find container \"fef8f130f1308829e0ce7c858597a404ca8da668cf2071b4fc49bd544f5cc099\": container with ID starting with fef8f130f1308829e0ce7c858597a404ca8da668cf2071b4fc49bd544f5cc099 not found: ID does not exist"
Sep 10 03:05:30.054191 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:05:30.054173    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8012-9085c7\" is forbidden: User \"system:node:ip-10-0-128-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-101.ec2.internal' and this object" podUID="5ecf6314-803a-4b07-bc9a-55cc05e7bd5d" pod="opendatahub/test-curl-8012-9085c7"
Sep 10 03:05:30.066919 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:05:30.066891    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8012-9085c7\" is forbidden: User \"system:node:ip-10-0-128-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-101.ec2.internal' and this object" podUID="5ecf6314-803a-4b07-bc9a-55cc05e7bd5d" pod="opendatahub/test-curl-8012-9085c7"
Sep 10 03:05:30.069812 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:30.069785    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ecf6314-803a-4b07-bc9a-55cc05e7bd5d" path="/var/lib/kubelet/pods/5ecf6314-803a-4b07-bc9a-55cc05e7bd5d/volumes"
Sep 10 03:05:31.197963 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:31.197936    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:05:31.217651 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:31.217628    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8012-438d65"
Sep 10 03:05:31.262513 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:31.262487    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plgdt\" (UniqueName: \"kubernetes.io/projected/74c6e35d-6407-4259-96e0-9e5b06a9f331-kube-api-access-plgdt\") pod \"74c6e35d-6407-4259-96e0-9e5b06a9f331\" (UID: \"74c6e35d-6407-4259-96e0-9e5b06a9f331\") "
Sep 10 03:05:31.264433 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:31.264407    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74c6e35d-6407-4259-96e0-9e5b06a9f331-kube-api-access-plgdt" (OuterVolumeSpecName: "kube-api-access-plgdt") pod "74c6e35d-6407-4259-96e0-9e5b06a9f331" (UID: "74c6e35d-6407-4259-96e0-9e5b06a9f331"). InnerVolumeSpecName "kube-api-access-plgdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:05:31.278439 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:31.278417    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:05:31.364030 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:31.363974    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plgdt\" (UniqueName: \"kubernetes.io/projected/74c6e35d-6407-4259-96e0-9e5b06a9f331-kube-api-access-plgdt\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:05:32.062491 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:32.062457    2558 generic.go:356] "Generic (PLEG): container finished" podID="74c6e35d-6407-4259-96e0-9e5b06a9f331" containerID="32124bf98c47b4183bb8cd2d2cb4dd18ea331c387bd579ac28ebc7982e1caa48" exitCode=137
Sep 10 03:05:32.062670 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:32.062571    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8012-438d65"
Sep 10 03:05:32.069931 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:32.069900    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74c6e35d-6407-4259-96e0-9e5b06a9f331" path="/var/lib/kubelet/pods/74c6e35d-6407-4259-96e0-9e5b06a9f331/volumes"
Sep 10 03:05:32.070461 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:32.070403    2558 scope.go:117] "RemoveContainer" containerID="32124bf98c47b4183bb8cd2d2cb4dd18ea331c387bd579ac28ebc7982e1caa48"
Sep 10 03:05:32.079473 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:32.079456    2558 scope.go:117] "RemoveContainer" containerID="32124bf98c47b4183bb8cd2d2cb4dd18ea331c387bd579ac28ebc7982e1caa48"
Sep 10 03:05:32.079735 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:05:32.079715    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32124bf98c47b4183bb8cd2d2cb4dd18ea331c387bd579ac28ebc7982e1caa48\": container with ID starting with 32124bf98c47b4183bb8cd2d2cb4dd18ea331c387bd579ac28ebc7982e1caa48 not found: ID does not exist" containerID="32124bf98c47b4183bb8cd2d2cb4dd18ea331c387bd579ac28ebc7982e1caa48"
Sep 10 03:05:32.079802 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:32.079744    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32124bf98c47b4183bb8cd2d2cb4dd18ea331c387bd579ac28ebc7982e1caa48"} err="failed to get container status \"32124bf98c47b4183bb8cd2d2cb4dd18ea331c387bd579ac28ebc7982e1caa48\": rpc error: code = NotFound desc = could not find container \"32124bf98c47b4183bb8cd2d2cb4dd18ea331c387bd579ac28ebc7982e1caa48\": container with ID starting with 32124bf98c47b4183bb8cd2d2cb4dd18ea331c387bd579ac28ebc7982e1caa48 not found: ID does not exist"
Sep 10 03:05:32.362508 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:32.362435    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:05:32.459458 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:32.459417    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:05:32.826134 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:32.826085    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:05:32.921655 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:32.921627    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:05:32.923637 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:32.923618    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-61660fd5-77cccfc78b-r69c5"]
Sep 10 03:05:32.924048 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:32.924036    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74c6e35d-6407-4259-96e0-9e5b06a9f331" containerName="test-curl-8012-438d65"
Sep 10 03:05:32.924087 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:32.924049    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="74c6e35d-6407-4259-96e0-9e5b06a9f331" containerName="test-curl-8012-438d65"
Sep 10 03:05:32.924153 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:32.924142    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="74c6e35d-6407-4259-96e0-9e5b06a9f331" containerName="test-curl-8012-438d65"
Sep 10 03:05:32.929192 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:32.929172    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-61660fd5-77cccfc78b-r69c5"
Sep 10 03:05:32.931721 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:32.931700    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-61660fd5\""
Sep 10 03:05:32.931821 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:32.931723    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-61660fd5-dockercfg-k85bd\""
Sep 10 03:05:32.937233 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:32.937056    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-61660fd5-77cccfc78b-r69c5"]
Sep 10 03:05:32.959971 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:32.959943    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-61660fd5-5d75b5f79c-fxgbc"]
Sep 10 03:05:32.964445 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:32.964417    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-61660fd5-5d75b5f79c-fxgbc"
Sep 10 03:05:32.970341 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:32.970320    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-61660fd5-5d75b5f79c-fxgbc"]
Sep 10 03:05:32.978093 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:32.978070    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4e74c00-3d61-4c0f-be9e-520f22a69d7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-61660fd5-77cccfc78b-r69c5\" (UID: \"d4e74c00-3d61-4c0f-be9e-520f22a69d7c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-61660fd5-77cccfc78b-r69c5"
Sep 10 03:05:32.978228 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:32.978157    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drdmt\" (UniqueName: \"kubernetes.io/projected/d4e74c00-3d61-4c0f-be9e-520f22a69d7c-kube-api-access-drdmt\") pod \"payload-pre-processing-e2e-mt-61660fd5-77cccfc78b-r69c5\" (UID: \"d4e74c00-3d61-4c0f-be9e-520f22a69d7c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-61660fd5-77cccfc78b-r69c5"
Sep 10 03:05:33.079047 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:33.078973    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7cbe3da-8cad-4ce1-b97c-be9133355de9-plugins-config-volume\") pod \"payload-processing-e2e-mt-61660fd5-5d75b5f79c-fxgbc\" (UID: \"b7cbe3da-8cad-4ce1-b97c-be9133355de9\") " pod="openshift-ingress/payload-processing-e2e-mt-61660fd5-5d75b5f79c-fxgbc"
Sep 10 03:05:33.079047 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:33.079008    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmq2b\" (UniqueName: \"kubernetes.io/projected/b7cbe3da-8cad-4ce1-b97c-be9133355de9-kube-api-access-pmq2b\") pod \"payload-processing-e2e-mt-61660fd5-5d75b5f79c-fxgbc\" (UID: \"b7cbe3da-8cad-4ce1-b97c-be9133355de9\") " pod="openshift-ingress/payload-processing-e2e-mt-61660fd5-5d75b5f79c-fxgbc"
Sep 10 03:05:33.079235 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:33.079066    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4e74c00-3d61-4c0f-be9e-520f22a69d7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-61660fd5-77cccfc78b-r69c5\" (UID: \"d4e74c00-3d61-4c0f-be9e-520f22a69d7c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-61660fd5-77cccfc78b-r69c5"
Sep 10 03:05:33.079235 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:33.079133    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drdmt\" (UniqueName: \"kubernetes.io/projected/d4e74c00-3d61-4c0f-be9e-520f22a69d7c-kube-api-access-drdmt\") pod \"payload-pre-processing-e2e-mt-61660fd5-77cccfc78b-r69c5\" (UID: \"d4e74c00-3d61-4c0f-be9e-520f22a69d7c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-61660fd5-77cccfc78b-r69c5"
Sep 10 03:05:33.079794 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:33.079777    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4e74c00-3d61-4c0f-be9e-520f22a69d7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-61660fd5-77cccfc78b-r69c5\" (UID: \"d4e74c00-3d61-4c0f-be9e-520f22a69d7c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-61660fd5-77cccfc78b-r69c5"
Sep 10 03:05:33.086279 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:33.086252    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drdmt\" (UniqueName: \"kubernetes.io/projected/d4e74c00-3d61-4c0f-be9e-520f22a69d7c-kube-api-access-drdmt\") pod \"payload-pre-processing-e2e-mt-61660fd5-77cccfc78b-r69c5\" (UID: \"d4e74c00-3d61-4c0f-be9e-520f22a69d7c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-61660fd5-77cccfc78b-r69c5"
Sep 10 03:05:33.179988 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:33.179962    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7cbe3da-8cad-4ce1-b97c-be9133355de9-plugins-config-volume\") pod \"payload-processing-e2e-mt-61660fd5-5d75b5f79c-fxgbc\" (UID: \"b7cbe3da-8cad-4ce1-b97c-be9133355de9\") " pod="openshift-ingress/payload-processing-e2e-mt-61660fd5-5d75b5f79c-fxgbc"
Sep 10 03:05:33.180088 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:33.179995    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmq2b\" (UniqueName: \"kubernetes.io/projected/b7cbe3da-8cad-4ce1-b97c-be9133355de9-kube-api-access-pmq2b\") pod \"payload-processing-e2e-mt-61660fd5-5d75b5f79c-fxgbc\" (UID: \"b7cbe3da-8cad-4ce1-b97c-be9133355de9\") " pod="openshift-ingress/payload-processing-e2e-mt-61660fd5-5d75b5f79c-fxgbc"
Sep 10 03:05:33.180621 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:33.180602    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7cbe3da-8cad-4ce1-b97c-be9133355de9-plugins-config-volume\") pod \"payload-processing-e2e-mt-61660fd5-5d75b5f79c-fxgbc\" (UID: \"b7cbe3da-8cad-4ce1-b97c-be9133355de9\") " pod="openshift-ingress/payload-processing-e2e-mt-61660fd5-5d75b5f79c-fxgbc"
Sep 10 03:05:33.186967 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:33.186940    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmq2b\" (UniqueName: \"kubernetes.io/projected/b7cbe3da-8cad-4ce1-b97c-be9133355de9-kube-api-access-pmq2b\") pod \"payload-processing-e2e-mt-61660fd5-5d75b5f79c-fxgbc\" (UID: \"b7cbe3da-8cad-4ce1-b97c-be9133355de9\") " pod="openshift-ingress/payload-processing-e2e-mt-61660fd5-5d75b5f79c-fxgbc"
Sep 10 03:05:33.246366 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:33.246344    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-61660fd5-77cccfc78b-r69c5"
Sep 10 03:05:33.275768 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:33.275741    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-61660fd5-5d75b5f79c-fxgbc"
Sep 10 03:05:33.520569 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:33.520544    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-61660fd5-77cccfc78b-r69c5"]
Sep 10 03:05:33.522382 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:05:33.522357    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e74c00_3d61_4c0f_be9e_520f22a69d7c.slice/crio-76265e0e3907648e11edf8d4959382883a7a3ff4cc9feb8fdb2fa2b52dd3e418 WatchSource:0}: Error finding container 76265e0e3907648e11edf8d4959382883a7a3ff4cc9feb8fdb2fa2b52dd3e418: Status 404 returned error can't find the container with id 76265e0e3907648e11edf8d4959382883a7a3ff4cc9feb8fdb2fa2b52dd3e418
Sep 10 03:05:33.560316 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:33.560289    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-61660fd5-5d75b5f79c-fxgbc"]
Sep 10 03:05:33.565212 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:05:33.565084    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7cbe3da_8cad_4ce1_b97c_be9133355de9.slice/crio-7035809cfe8152e3f298849c9a96d79c537e66a9e4448b98c5727e8838aa4919 WatchSource:0}: Error finding container 7035809cfe8152e3f298849c9a96d79c537e66a9e4448b98c5727e8838aa4919: Status 404 returned error can't find the container with id 7035809cfe8152e3f298849c9a96d79c537e66a9e4448b98c5727e8838aa4919
Sep 10 03:05:34.078962 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:34.078759    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-61660fd5-5d75b5f79c-fxgbc" event={"ID":"b7cbe3da-8cad-4ce1-b97c-be9133355de9","Type":"ContainerStarted","Data":"4bc56684e8706a876970316a5022b56be157274649db79c7c7cd93531fc93cc7"}
Sep 10 03:05:34.078962 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:34.078817    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-61660fd5-5d75b5f79c-fxgbc" event={"ID":"b7cbe3da-8cad-4ce1-b97c-be9133355de9","Type":"ContainerStarted","Data":"7035809cfe8152e3f298849c9a96d79c537e66a9e4448b98c5727e8838aa4919"}
Sep 10 03:05:34.078962 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:34.078843    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-61660fd5-5d75b5f79c-fxgbc"
Sep 10 03:05:34.080465 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:34.080433    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-61660fd5-77cccfc78b-r69c5" event={"ID":"d4e74c00-3d61-4c0f-be9e-520f22a69d7c","Type":"ContainerStarted","Data":"9e564db807daea2ee8a6f0cdfc23b78afd1e1140cd3dcebb9fb8e82869bcb83a"}
Sep 10 03:05:34.080574 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:34.080483    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-61660fd5-77cccfc78b-r69c5" event={"ID":"d4e74c00-3d61-4c0f-be9e-520f22a69d7c","Type":"ContainerStarted","Data":"76265e0e3907648e11edf8d4959382883a7a3ff4cc9feb8fdb2fa2b52dd3e418"}
Sep 10 03:05:34.080574 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:34.080511    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-61660fd5-77cccfc78b-r69c5"
Sep 10 03:05:34.096170 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:34.096118    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-61660fd5-5d75b5f79c-fxgbc" podStartSLOduration=2.096091093 podStartE2EDuration="2.096091093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:05:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:05:34.094002727 +0000 UTC m=+1124.469088659" watchObservedRunningTime="2026-09-10 03:05:34.096091093 +0000 UTC m=+1124.471177080"
Sep 10 03:05:34.109782 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:34.109742    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-61660fd5-77cccfc78b-r69c5" podStartSLOduration=2.109729035 podStartE2EDuration="2.109729035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:05:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:05:34.108303075 +0000 UTC m=+1124.483388995" watchObservedRunningTime="2026-09-10 03:05:34.109729035 +0000 UTC m=+1124.484814955"
Sep 10 03:05:38.202292 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:38.202263    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:05:43.054419 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:43.054384    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:05:43.108361 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:43.108321    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:05:43.414789 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:43.414721    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:05:45.089201 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:45.089171    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-61660fd5-77cccfc78b-r69c5"
Sep 10 03:05:45.089592 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:45.089238    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-61660fd5-5d75b5f79c-fxgbc"
Sep 10 03:05:50.159903 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:50.159872    2558 scope.go:117] "RemoveContainer" containerID="45dd2bad9b2d1a381172f694f5f97f56e93c471b46374db2fe3947365c03ced3"
Sep 10 03:05:51.694710 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:51.694677    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0f4da-pz2rr"]
Sep 10 03:05:51.699173 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:51.699151    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0f4da-pz2rr"
Sep 10 03:05:51.701737 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:51.701714    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8jjl\""
Sep 10 03:05:51.704994 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:51.704970    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0f4da-pz2rr"]
Sep 10 03:05:51.813787 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:51.813761    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:05:51.843692 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:51.843665    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d488233-5f95-40ce-8a87-501524279409-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b0f4da-pz2rr\" (UID: \"4d488233-5f95-40ce-8a87-501524279409\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0f4da-pz2rr"
Sep 10 03:05:51.944565 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:51.944532    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d488233-5f95-40ce-8a87-501524279409-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b0f4da-pz2rr\" (UID: \"4d488233-5f95-40ce-8a87-501524279409\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0f4da-pz2rr"
Sep 10 03:05:51.945177 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:51.945097    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d488233-5f95-40ce-8a87-501524279409-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b0f4da-pz2rr\" (UID: \"4d488233-5f95-40ce-8a87-501524279409\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0f4da-pz2rr"
Sep 10 03:05:52.012224 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:52.012199    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0f4da-pz2rr"
Sep 10 03:05:52.283471 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:52.283406    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0f4da-pz2rr"]
Sep 10 03:05:52.286199 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:05:52.286172    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d488233_5f95_40ce_8a87_501524279409.slice/crio-86e9836fe11e0568a2fa1383ca52cc2bb7f96c03e7a68ac34c222807b8fa451c WatchSource:0}: Error finding container 86e9836fe11e0568a2fa1383ca52cc2bb7f96c03e7a68ac34c222807b8fa451c: Status 404 returned error can't find the container with id 86e9836fe11e0568a2fa1383ca52cc2bb7f96c03e7a68ac34c222807b8fa451c
Sep 10 03:05:52.310757 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:52.303566    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:05:53.196636 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:53.196577    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:05:53.213878 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:53.213848    2558 generic.go:356] "Generic (PLEG): container finished" podID="4d488233-5f95-40ce-8a87-501524279409" containerID="ba11117d2678afc68ece4105d3a802428b45d5939c43d9bc7da4af77255ad101" exitCode=0
Sep 10 03:05:53.213987 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:53.213934    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0f4da-pz2rr" event={"ID":"4d488233-5f95-40ce-8a87-501524279409","Type":"ContainerDied","Data":"ba11117d2678afc68ece4105d3a802428b45d5939c43d9bc7da4af77255ad101"}
Sep 10 03:05:53.213987 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:53.213979    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0f4da-pz2rr" event={"ID":"4d488233-5f95-40ce-8a87-501524279409","Type":"ContainerStarted","Data":"86e9836fe11e0568a2fa1383ca52cc2bb7f96c03e7a68ac34c222807b8fa451c"}
Sep 10 03:05:54.359287 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:54.359263    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0f4da-pz2rr"
Sep 10 03:05:54.468003 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:54.467966    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d488233-5f95-40ce-8a87-501524279409-maas-api-service-ca\") pod \"4d488233-5f95-40ce-8a87-501524279409\" (UID: \"4d488233-5f95-40ce-8a87-501524279409\") "
Sep 10 03:05:54.468332 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:54.468304    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d488233-5f95-40ce-8a87-501524279409-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4d488233-5f95-40ce-8a87-501524279409" (UID: "4d488233-5f95-40ce-8a87-501524279409"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:05:54.568885 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:54.568830    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d488233-5f95-40ce-8a87-501524279409-maas-api-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:05:55.227533 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:55.227508    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0f4da-pz2rr"
Sep 10 03:05:55.227702 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:55.227556    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0f4da-pz2rr" event={"ID":"4d488233-5f95-40ce-8a87-501524279409","Type":"ContainerDied","Data":"86e9836fe11e0568a2fa1383ca52cc2bb7f96c03e7a68ac34c222807b8fa451c"}
Sep 10 03:05:55.227702 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:55.227610    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86e9836fe11e0568a2fa1383ca52cc2bb7f96c03e7a68ac34c222807b8fa451c"
Sep 10 03:05:56.230818 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:05:56.230786    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:06:01.879636 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:01.879576    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:06:02.479616 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:02.479581    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:06:02.506613 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:02.506581    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b0f4da-77f559485f-2qpw7"]
Sep 10 03:06:02.506801 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:02.506779    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-b0f4da-77f559485f-2qpw7" podUID="7b487b15-49fd-4368-811a-f8df96477cf1" containerName="payload-processing" containerID="cri-o://af0ae470f0eda6bd2deef1bc77730b274c866d8cf925826c4165bdde69dfd61c" gracePeriod=30
Sep 10 03:06:02.515503 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:02.515477    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b0f4da-856f79778-8zd5g"]
Sep 10 03:06:02.515724 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:02.515696    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0f4da-856f79778-8zd5g" podUID="0d007f3d-7158-4d17-913b-be4d1f7015ef" containerName="payload-pre-processing" containerID="cri-o://9f1c798664477224ec155261416c99bd785d0c076c24801436dd0285aa122a46" gracePeriod=30
Sep 10 03:06:02.577827 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:02.577776    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:06:02.670688 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:02.670479    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0f4da-856f79778-8zd5g" podUID="0d007f3d-7158-4d17-913b-be4d1f7015ef" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.49:9004: connect: connection refused"
Sep 10 03:06:02.670688 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:02.670473    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-b0f4da-77f559485f-2qpw7" podUID="7b487b15-49fd-4368-811a-f8df96477cf1" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.50:9004: connect: connection refused"
Sep 10 03:06:02.678762 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:02.678736    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:06:02.686352 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:02.686326    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0f4da-pz2rr"]
Sep 10 03:06:02.689255 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:02.689231    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0f4da-pz2rr"]
Sep 10 03:06:02.736816 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:02.736754    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:06:02.797382 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:02.797363    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b0f4da-77f559485f-2qpw7"
Sep 10 03:06:02.800539 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:02.800520    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0f4da-856f79778-8zd5g"
Sep 10 03:06:02.934910 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:02.934879    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b487b15-49fd-4368-811a-f8df96477cf1-plugins-config-volume\") pod \"7b487b15-49fd-4368-811a-f8df96477cf1\" (UID: \"7b487b15-49fd-4368-811a-f8df96477cf1\") "
Sep 10 03:06:02.934910 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:02.934914    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prqbg\" (UniqueName: \"kubernetes.io/projected/7b487b15-49fd-4368-811a-f8df96477cf1-kube-api-access-prqbg\") pod \"7b487b15-49fd-4368-811a-f8df96477cf1\" (UID: \"7b487b15-49fd-4368-811a-f8df96477cf1\") "
Sep 10 03:06:02.935398 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:02.934936    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jjz6\" (UniqueName: \"kubernetes.io/projected/0d007f3d-7158-4d17-913b-be4d1f7015ef-kube-api-access-2jjz6\") pod \"0d007f3d-7158-4d17-913b-be4d1f7015ef\" (UID: \"0d007f3d-7158-4d17-913b-be4d1f7015ef\") "
Sep 10 03:06:02.935398 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:02.934968    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d007f3d-7158-4d17-913b-be4d1f7015ef-plugins-config-volume\") pod \"0d007f3d-7158-4d17-913b-be4d1f7015ef\" (UID: \"0d007f3d-7158-4d17-913b-be4d1f7015ef\") "
Sep 10 03:06:02.935398 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:02.935297    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b487b15-49fd-4368-811a-f8df96477cf1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7b487b15-49fd-4368-811a-f8df96477cf1" (UID: "7b487b15-49fd-4368-811a-f8df96477cf1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:06:02.935504 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:02.935409    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d007f3d-7158-4d17-913b-be4d1f7015ef-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0d007f3d-7158-4d17-913b-be4d1f7015ef" (UID: "0d007f3d-7158-4d17-913b-be4d1f7015ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:06:02.937003 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:02.936982    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d007f3d-7158-4d17-913b-be4d1f7015ef-kube-api-access-2jjz6" (OuterVolumeSpecName: "kube-api-access-2jjz6") pod "0d007f3d-7158-4d17-913b-be4d1f7015ef" (UID: "0d007f3d-7158-4d17-913b-be4d1f7015ef"). InnerVolumeSpecName "kube-api-access-2jjz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:06:02.937346 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:02.937322    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b487b15-49fd-4368-811a-f8df96477cf1-kube-api-access-prqbg" (OuterVolumeSpecName: "kube-api-access-prqbg") pod "7b487b15-49fd-4368-811a-f8df96477cf1" (UID: "7b487b15-49fd-4368-811a-f8df96477cf1"). InnerVolumeSpecName "kube-api-access-prqbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:06:03.035891 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:03.035821    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b487b15-49fd-4368-811a-f8df96477cf1-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:06:03.035891 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:03.035847    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prqbg\" (UniqueName: \"kubernetes.io/projected/7b487b15-49fd-4368-811a-f8df96477cf1-kube-api-access-prqbg\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:06:03.035891 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:03.035860    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jjz6\" (UniqueName: \"kubernetes.io/projected/0d007f3d-7158-4d17-913b-be4d1f7015ef-kube-api-access-2jjz6\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:06:03.035891 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:03.035872    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d007f3d-7158-4d17-913b-be4d1f7015ef-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:06:03.122532 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:03.122471    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:06:03.275705 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:03.275675    2558 generic.go:356] "Generic (PLEG): container finished" podID="0d007f3d-7158-4d17-913b-be4d1f7015ef" containerID="9f1c798664477224ec155261416c99bd785d0c076c24801436dd0285aa122a46" exitCode=0
Sep 10 03:06:03.275833 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:03.275729    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0f4da-856f79778-8zd5g"
Sep 10 03:06:03.275833 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:03.275755    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0f4da-856f79778-8zd5g" event={"ID":"0d007f3d-7158-4d17-913b-be4d1f7015ef","Type":"ContainerDied","Data":"9f1c798664477224ec155261416c99bd785d0c076c24801436dd0285aa122a46"}
Sep 10 03:06:03.275931 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:03.275834    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0f4da-856f79778-8zd5g" event={"ID":"0d007f3d-7158-4d17-913b-be4d1f7015ef","Type":"ContainerDied","Data":"e5671bc33ef539b809cf4da99636979524ee814de992d3a954e1981e8c9fdca2"}
Sep 10 03:06:03.275931 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:03.275857    2558 scope.go:117] "RemoveContainer" containerID="9f1c798664477224ec155261416c99bd785d0c076c24801436dd0285aa122a46"
Sep 10 03:06:03.277385 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:03.277326    2558 generic.go:356] "Generic (PLEG): container finished" podID="7b487b15-49fd-4368-811a-f8df96477cf1" containerID="af0ae470f0eda6bd2deef1bc77730b274c866d8cf925826c4165bdde69dfd61c" exitCode=0
Sep 10 03:06:03.277489 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:03.277356    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b0f4da-77f559485f-2qpw7" event={"ID":"7b487b15-49fd-4368-811a-f8df96477cf1","Type":"ContainerDied","Data":"af0ae470f0eda6bd2deef1bc77730b274c866d8cf925826c4165bdde69dfd61c"}
Sep 10 03:06:03.277489 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:03.277435    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b0f4da-77f559485f-2qpw7" event={"ID":"7b487b15-49fd-4368-811a-f8df96477cf1","Type":"ContainerDied","Data":"642d3ebbfe99a4b0f3ee8c225bfae5496013bdb1ac3e24cbaf21f68d3f154f67"}
Sep 10 03:06:03.277489 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:03.277384    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b0f4da-77f559485f-2qpw7"
Sep 10 03:06:03.285876 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:03.285859    2558 scope.go:117] "RemoveContainer" containerID="9f1c798664477224ec155261416c99bd785d0c076c24801436dd0285aa122a46"
Sep 10 03:06:03.286229 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:06:03.286174    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f1c798664477224ec155261416c99bd785d0c076c24801436dd0285aa122a46\": container with ID starting with 9f1c798664477224ec155261416c99bd785d0c076c24801436dd0285aa122a46 not found: ID does not exist" containerID="9f1c798664477224ec155261416c99bd785d0c076c24801436dd0285aa122a46"
Sep 10 03:06:03.286315 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:03.286216    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f1c798664477224ec155261416c99bd785d0c076c24801436dd0285aa122a46"} err="failed to get container status \"9f1c798664477224ec155261416c99bd785d0c076c24801436dd0285aa122a46\": rpc error: code = NotFound desc = could not find container \"9f1c798664477224ec155261416c99bd785d0c076c24801436dd0285aa122a46\": container with ID starting with 9f1c798664477224ec155261416c99bd785d0c076c24801436dd0285aa122a46 not found: ID does not exist"
Sep 10 03:06:03.286315 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:03.286241    2558 scope.go:117] "RemoveContainer" containerID="af0ae470f0eda6bd2deef1bc77730b274c866d8cf925826c4165bdde69dfd61c"
Sep 10 03:06:03.294705 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:03.294689    2558 scope.go:117] "RemoveContainer" containerID="af0ae470f0eda6bd2deef1bc77730b274c866d8cf925826c4165bdde69dfd61c"
Sep 10 03:06:03.294960 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:06:03.294938    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af0ae470f0eda6bd2deef1bc77730b274c866d8cf925826c4165bdde69dfd61c\": container with ID starting with af0ae470f0eda6bd2deef1bc77730b274c866d8cf925826c4165bdde69dfd61c not found: ID does not exist" containerID="af0ae470f0eda6bd2deef1bc77730b274c866d8cf925826c4165bdde69dfd61c"
Sep 10 03:06:03.295024 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:03.294967    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af0ae470f0eda6bd2deef1bc77730b274c866d8cf925826c4165bdde69dfd61c"} err="failed to get container status \"af0ae470f0eda6bd2deef1bc77730b274c866d8cf925826c4165bdde69dfd61c\": rpc error: code = NotFound desc = could not find container \"af0ae470f0eda6bd2deef1bc77730b274c866d8cf925826c4165bdde69dfd61c\": container with ID starting with af0ae470f0eda6bd2deef1bc77730b274c866d8cf925826c4165bdde69dfd61c not found: ID does not exist"
Sep 10 03:06:03.298405 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:03.298382    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b0f4da-856f79778-8zd5g"]
Sep 10 03:06:03.303900 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:03.303877    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b0f4da-856f79778-8zd5g"]
Sep 10 03:06:03.314328 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:03.314310    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b0f4da-77f559485f-2qpw7"]
Sep 10 03:06:03.327581 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:03.327554    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b0f4da-77f559485f-2qpw7"]
Sep 10 03:06:04.069603 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:04.069568    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d007f3d-7158-4d17-913b-be4d1f7015ef" path="/var/lib/kubelet/pods/0d007f3d-7158-4d17-913b-be4d1f7015ef/volumes"
Sep 10 03:06:04.069968 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:04.069884    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d488233-5f95-40ce-8a87-501524279409" path="/var/lib/kubelet/pods/4d488233-5f95-40ce-8a87-501524279409/volumes"
Sep 10 03:06:04.070206 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:04.070192    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b487b15-49fd-4368-811a-f8df96477cf1" path="/var/lib/kubelet/pods/7b487b15-49fd-4368-811a-f8df96477cf1/volumes"
Sep 10 03:06:08.696211 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:08.696178    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:06:08.757973 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:08.757943    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:06:08.960309 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:08.960271    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:06:09.340055 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:09.339975    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce5f9d-qmhff"]
Sep 10 03:06:09.340442 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:09.340428    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d007f3d-7158-4d17-913b-be4d1f7015ef" containerName="payload-pre-processing"
Sep 10 03:06:09.340493 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:09.340444    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d007f3d-7158-4d17-913b-be4d1f7015ef" containerName="payload-pre-processing"
Sep 10 03:06:09.340493 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:09.340457    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b487b15-49fd-4368-811a-f8df96477cf1" containerName="payload-processing"
Sep 10 03:06:09.340493 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:09.340463    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b487b15-49fd-4368-811a-f8df96477cf1" containerName="payload-processing"
Sep 10 03:06:09.340493 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:09.340477    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d488233-5f95-40ce-8a87-501524279409" containerName="revoke-keys"
Sep 10 03:06:09.340493 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:09.340484    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d488233-5f95-40ce-8a87-501524279409" containerName="revoke-keys"
Sep 10 03:06:09.340655 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:09.340554    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b487b15-49fd-4368-811a-f8df96477cf1" containerName="payload-processing"
Sep 10 03:06:09.340655 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:09.340561    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d488233-5f95-40ce-8a87-501524279409" containerName="revoke-keys"
Sep 10 03:06:09.340655 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:09.340570    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d007f3d-7158-4d17-913b-be4d1f7015ef" containerName="payload-pre-processing"
Sep 10 03:06:09.345117 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:09.345087    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce5f9d-qmhff"
Sep 10 03:06:09.347395 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:09.347368    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8jjl\""
Sep 10 03:06:09.349629 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:09.349607    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce5f9d-qmhff"]
Sep 10 03:06:09.459651 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:09.459603    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:06:09.488761 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:09.488737    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd11d03-3879-4879-9640-6b67d3972f6e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ce5f9d-qmhff\" (UID: \"4cd11d03-3879-4879-9640-6b67d3972f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce5f9d-qmhff"
Sep 10 03:06:09.589663 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:09.589639    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd11d03-3879-4879-9640-6b67d3972f6e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ce5f9d-qmhff\" (UID: \"4cd11d03-3879-4879-9640-6b67d3972f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce5f9d-qmhff"
Sep 10 03:06:09.590312 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:09.590268    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd11d03-3879-4879-9640-6b67d3972f6e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ce5f9d-qmhff\" (UID: \"4cd11d03-3879-4879-9640-6b67d3972f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce5f9d-qmhff"
Sep 10 03:06:09.661419 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:09.661396    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce5f9d-qmhff"
Sep 10 03:06:09.931610 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:09.931584    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce5f9d-qmhff"]
Sep 10 03:06:09.933163 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:06:09.933131    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd11d03_3879_4879_9640_6b67d3972f6e.slice/crio-74395ef8fb51625b9c1960b83d99f845114f4175ebc1b19ae5f17f088f261d58 WatchSource:0}: Error finding container 74395ef8fb51625b9c1960b83d99f845114f4175ebc1b19ae5f17f088f261d58: Status 404 returned error can't find the container with id 74395ef8fb51625b9c1960b83d99f845114f4175ebc1b19ae5f17f088f261d58
Sep 10 03:06:10.323025 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:10.322999    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce5f9d-qmhff" event={"ID":"4cd11d03-3879-4879-9640-6b67d3972f6e","Type":"ContainerStarted","Data":"feba1fbb71bdf347af9808ca9fc2141b5f722a52e64e54c59017132b3ff626c5"}
Sep 10 03:06:10.323149 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:10.323055    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce5f9d-qmhff" event={"ID":"4cd11d03-3879-4879-9640-6b67d3972f6e","Type":"ContainerStarted","Data":"74395ef8fb51625b9c1960b83d99f845114f4175ebc1b19ae5f17f088f261d58"}
Sep 10 03:06:10.337098 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:10.337054    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce5f9d-qmhff" podStartSLOduration=1.337039053 podStartE2EDuration="1.337039053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:06:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:06:10.335190457 +0000 UTC m=+1160.710276379" watchObservedRunningTime="2026-09-10 03:06:10.337039053 +0000 UTC m=+1160.712124999"
Sep 10 03:06:11.331442 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:11.331411    2558 generic.go:356] "Generic (PLEG): container finished" podID="4cd11d03-3879-4879-9640-6b67d3972f6e" containerID="feba1fbb71bdf347af9808ca9fc2141b5f722a52e64e54c59017132b3ff626c5" exitCode=0
Sep 10 03:06:11.331809 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:11.331507    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce5f9d-qmhff" event={"ID":"4cd11d03-3879-4879-9640-6b67d3972f6e","Type":"ContainerDied","Data":"feba1fbb71bdf347af9808ca9fc2141b5f722a52e64e54c59017132b3ff626c5"}
Sep 10 03:06:12.474803 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:12.474773    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce5f9d-qmhff"
Sep 10 03:06:12.615270 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:12.615208    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd11d03-3879-4879-9640-6b67d3972f6e-maas-api-service-ca\") pod \"4cd11d03-3879-4879-9640-6b67d3972f6e\" (UID: \"4cd11d03-3879-4879-9640-6b67d3972f6e\") "
Sep 10 03:06:12.615503 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:12.615482    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cd11d03-3879-4879-9640-6b67d3972f6e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4cd11d03-3879-4879-9640-6b67d3972f6e" (UID: "4cd11d03-3879-4879-9640-6b67d3972f6e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:06:12.716268 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:12.716214    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd11d03-3879-4879-9640-6b67d3972f6e-maas-api-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:06:13.344250 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:13.344221    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce5f9d-qmhff"
Sep 10 03:06:13.344416 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:13.344250    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce5f9d-qmhff" event={"ID":"4cd11d03-3879-4879-9640-6b67d3972f6e","Type":"ContainerDied","Data":"74395ef8fb51625b9c1960b83d99f845114f4175ebc1b19ae5f17f088f261d58"}
Sep 10 03:06:13.344416 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:13.344292    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74395ef8fb51625b9c1960b83d99f845114f4175ebc1b19ae5f17f088f261d58"
Sep 10 03:06:15.862680 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:15.862643    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:06:19.787596 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:19.787558    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ce5f9d-29816820-79v9c"]
Sep 10 03:06:19.789911 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:19.789889    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ce5f9d-29816820-79v9c"]
Sep 10 03:06:19.799050 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:19.799014    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:06:19.873322 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:19.873295    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:06:19.961166 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:19.961077    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce5f9d-qmhff"]
Sep 10 03:06:19.962997 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:19.962970    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:06:19.963776 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:19.963754    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce5f9d-qmhff"]
Sep 10 03:06:20.028596 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:20.028570    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:06:20.071169 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:20.071097    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cd11d03-3879-4879-9640-6b67d3972f6e" path="/var/lib/kubelet/pods/4cd11d03-3879-4879-9640-6b67d3972f6e/volumes"
Sep 10 03:06:20.071416 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:20.071403    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bae36d3f-86b7-4695-9604-68ce874095d1" path="/var/lib/kubelet/pods/bae36d3f-86b7-4695-9604-68ce874095d1/volumes"
Sep 10 03:06:20.354602 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:20.354527    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:06:22.478598 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:22.478555    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:06:25.338324 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:25.338293    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h"]
Sep 10 03:06:25.434046 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:25.434017    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:06:30.041018 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:30.040980    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:06:33.112057 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:33.112019    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:06:40.358239 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:40.358177    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h" podUID="a35c892b-01e1-402c-804c-71301f820d81" containerName="main" containerID="cri-o://16ac3e95047f8fc292dd28ce2b56ea7ce652262b1208bd995155a6c6bb1c8521" gracePeriod=45
Sep 10 03:06:40.518249 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:40.518221    2558 generic.go:356] "Generic (PLEG): container finished" podID="a35c892b-01e1-402c-804c-71301f820d81" containerID="16ac3e95047f8fc292dd28ce2b56ea7ce652262b1208bd995155a6c6bb1c8521" exitCode=0
Sep 10 03:06:40.518395 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:40.518302    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h" event={"ID":"a35c892b-01e1-402c-804c-71301f820d81","Type":"ContainerDied","Data":"16ac3e95047f8fc292dd28ce2b56ea7ce652262b1208bd995155a6c6bb1c8521"}
Sep 10 03:06:40.614695 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:40.614645    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h"
Sep 10 03:06:40.741142 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:40.741090    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a35c892b-01e1-402c-804c-71301f820d81-tmp-dir\") pod \"a35c892b-01e1-402c-804c-71301f820d81\" (UID: \"a35c892b-01e1-402c-804c-71301f820d81\") "
Sep 10 03:06:40.741269 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:40.741175    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxxg7\" (UniqueName: \"kubernetes.io/projected/a35c892b-01e1-402c-804c-71301f820d81-kube-api-access-fxxg7\") pod \"a35c892b-01e1-402c-804c-71301f820d81\" (UID: \"a35c892b-01e1-402c-804c-71301f820d81\") "
Sep 10 03:06:40.741269 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:40.741197    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a35c892b-01e1-402c-804c-71301f820d81-home\") pod \"a35c892b-01e1-402c-804c-71301f820d81\" (UID: \"a35c892b-01e1-402c-804c-71301f820d81\") "
Sep 10 03:06:40.741269 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:40.741212    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a35c892b-01e1-402c-804c-71301f820d81-dshm\") pod \"a35c892b-01e1-402c-804c-71301f820d81\" (UID: \"a35c892b-01e1-402c-804c-71301f820d81\") "
Sep 10 03:06:40.741269 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:40.741264    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a35c892b-01e1-402c-804c-71301f820d81-model-cache\") pod \"a35c892b-01e1-402c-804c-71301f820d81\" (UID: \"a35c892b-01e1-402c-804c-71301f820d81\") "
Sep 10 03:06:40.741473 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:40.741303    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a35c892b-01e1-402c-804c-71301f820d81-tls-certs\") pod \"a35c892b-01e1-402c-804c-71301f820d81\" (UID: \"a35c892b-01e1-402c-804c-71301f820d81\") "
Sep 10 03:06:40.741473 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:40.741380    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a35c892b-01e1-402c-804c-71301f820d81-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a35c892b-01e1-402c-804c-71301f820d81" (UID: "a35c892b-01e1-402c-804c-71301f820d81"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:06:40.741473 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:40.741413    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a35c892b-01e1-402c-804c-71301f820d81-home" (OuterVolumeSpecName: "home") pod "a35c892b-01e1-402c-804c-71301f820d81" (UID: "a35c892b-01e1-402c-804c-71301f820d81"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:06:40.741632 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:40.741514    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a35c892b-01e1-402c-804c-71301f820d81-tmp-dir\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:06:40.741632 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:40.741558    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a35c892b-01e1-402c-804c-71301f820d81-home\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:06:40.741632 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:40.741587    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a35c892b-01e1-402c-804c-71301f820d81-model-cache" (OuterVolumeSpecName: "model-cache") pod "a35c892b-01e1-402c-804c-71301f820d81" (UID: "a35c892b-01e1-402c-804c-71301f820d81"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:06:40.743481 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:40.743450    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a35c892b-01e1-402c-804c-71301f820d81-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a35c892b-01e1-402c-804c-71301f820d81" (UID: "a35c892b-01e1-402c-804c-71301f820d81"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:06:40.743481 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:40.743471    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a35c892b-01e1-402c-804c-71301f820d81-dshm" (OuterVolumeSpecName: "dshm") pod "a35c892b-01e1-402c-804c-71301f820d81" (UID: "a35c892b-01e1-402c-804c-71301f820d81"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:06:40.743630 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:40.743528    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a35c892b-01e1-402c-804c-71301f820d81-kube-api-access-fxxg7" (OuterVolumeSpecName: "kube-api-access-fxxg7") pod "a35c892b-01e1-402c-804c-71301f820d81" (UID: "a35c892b-01e1-402c-804c-71301f820d81"). InnerVolumeSpecName "kube-api-access-fxxg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:06:40.842714 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:40.842690    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a35c892b-01e1-402c-804c-71301f820d81-model-cache\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:06:40.842714 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:40.842710    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a35c892b-01e1-402c-804c-71301f820d81-tls-certs\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:06:40.842849 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:40.842719    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxxg7\" (UniqueName: \"kubernetes.io/projected/a35c892b-01e1-402c-804c-71301f820d81-kube-api-access-fxxg7\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:06:40.842849 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:40.842729    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a35c892b-01e1-402c-804c-71301f820d81-dshm\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:06:41.531148 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:41.531098    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h"
Sep 10 03:06:41.531148 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:41.531131    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h" event={"ID":"a35c892b-01e1-402c-804c-71301f820d81","Type":"ContainerDied","Data":"4763df444281538600fe7146c1c93e9b7269a5857b5b5c6d8ef24b8c1594ef22"}
Sep 10 03:06:41.531616 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:41.531183    2558 scope.go:117] "RemoveContainer" containerID="16ac3e95047f8fc292dd28ce2b56ea7ce652262b1208bd995155a6c6bb1c8521"
Sep 10 03:06:41.553051 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:41.553018    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h"]
Sep 10 03:06:41.558602 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:41.558580    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ce5f9d/test-model-ce5f9d-kserve-57d6fbcdbf-n7w2h"]
Sep 10 03:06:42.071347 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:42.071314    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a35c892b-01e1-402c-804c-71301f820d81" path="/var/lib/kubelet/pods/a35c892b-01e1-402c-804c-71301f820d81/volumes"
Sep 10 03:06:50.192341 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:50.192311    2558 scope.go:117] "RemoveContainer" containerID="5a7d018081a638f280292662ab479d1a49e77ffe248d0233ebb6ef7026b448f5"
Sep 10 03:06:50.200904 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:06:50.200886    2558 scope.go:117] "RemoveContainer" containerID="8771d712c5a08e126d54836e54c2c8fc732ce8d9b9015b9d56a09716809ca584"
Sep 10 03:07:02.327480 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:02.327431    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:07:02.885625 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:02.885595    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf"]
Sep 10 03:07:02.885992 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:02.885979    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a35c892b-01e1-402c-804c-71301f820d81" containerName="main"
Sep 10 03:07:02.886046 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:02.885994    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a35c892b-01e1-402c-804c-71301f820d81" containerName="main"
Sep 10 03:07:02.886046 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:02.886014    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cd11d03-3879-4879-9640-6b67d3972f6e" containerName="revoke-keys"
Sep 10 03:07:02.886046 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:02.886020    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cd11d03-3879-4879-9640-6b67d3972f6e" containerName="revoke-keys"
Sep 10 03:07:02.886195 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:02.886084    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cd11d03-3879-4879-9640-6b67d3972f6e" containerName="revoke-keys"
Sep 10 03:07:02.886195 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:02.886093    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a35c892b-01e1-402c-804c-71301f820d81" containerName="main"
Sep 10 03:07:02.889089 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:02.889072    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf"
Sep 10 03:07:02.891466 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:02.891447    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e6e6b8\"/\"distinct-2-w4-e6e6b8-kserve-self-signed-certs\""
Sep 10 03:07:02.897603 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:02.897579    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf"]
Sep 10 03:07:03.013702 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:03.013676    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-home\") pod \"distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf\" (UID: \"90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf"
Sep 10 03:07:03.013803 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:03.013704    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-model-cache\") pod \"distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf\" (UID: \"90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf"
Sep 10 03:07:03.013803 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:03.013725    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-tls-certs\") pod \"distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf\" (UID: \"90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf"
Sep 10 03:07:03.013881 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:03.013824    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-tmp-dir\") pod \"distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf\" (UID: \"90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf"
Sep 10 03:07:03.013881 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:03.013855    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-dshm\") pod \"distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf\" (UID: \"90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf"
Sep 10 03:07:03.013946 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:03.013876    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stmjp\" (UniqueName: \"kubernetes.io/projected/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-kube-api-access-stmjp\") pod \"distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf\" (UID: \"90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf"
Sep 10 03:07:03.114314 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:03.114285    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-dshm\") pod \"distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf\" (UID: \"90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf"
Sep 10 03:07:03.114433 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:03.114331    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stmjp\" (UniqueName: \"kubernetes.io/projected/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-kube-api-access-stmjp\") pod \"distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf\" (UID: \"90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf"
Sep 10 03:07:03.114433 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:03.114407    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-home\") pod \"distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf\" (UID: \"90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf"
Sep 10 03:07:03.114555 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:03.114527    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-model-cache\") pod \"distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf\" (UID: \"90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf"
Sep 10 03:07:03.114608 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:03.114585    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-tls-certs\") pod \"distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf\" (UID: \"90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf"
Sep 10 03:07:03.114719 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:03.114683    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-tmp-dir\") pod \"distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf\" (UID: \"90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf"
Sep 10 03:07:03.114789 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:03.114716    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-home\") pod \"distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf\" (UID: \"90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf"
Sep 10 03:07:03.114943 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:03.114918    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-model-cache\") pod \"distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf\" (UID: \"90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf"
Sep 10 03:07:03.115015 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:03.114997    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-tmp-dir\") pod \"distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf\" (UID: \"90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf"
Sep 10 03:07:03.116629 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:03.116600    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-dshm\") pod \"distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf\" (UID: \"90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf"
Sep 10 03:07:03.116938 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:03.116918    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-tls-certs\") pod \"distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf\" (UID: \"90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf"
Sep 10 03:07:03.121391 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:03.121371    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stmjp\" (UniqueName: \"kubernetes.io/projected/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-kube-api-access-stmjp\") pod \"distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf\" (UID: \"90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4\") " pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf"
Sep 10 03:07:03.198883 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:03.198863    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf"
Sep 10 03:07:03.470695 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:03.470669    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf"]
Sep 10 03:07:03.473933 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:07:03.473902    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90cdbd86_1f59_4cf7_81c7_d04bf29cc0c4.slice/crio-5ccd6bb1024cdcc0d19922661ac701f44fda5fc27183e08ec72c73d182c6ee95 WatchSource:0}: Error finding container 5ccd6bb1024cdcc0d19922661ac701f44fda5fc27183e08ec72c73d182c6ee95: Status 404 returned error can't find the container with id 5ccd6bb1024cdcc0d19922661ac701f44fda5fc27183e08ec72c73d182c6ee95
Sep 10 03:07:03.630795 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:03.630754    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:07:03.668521 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:03.668488    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf" event={"ID":"90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4","Type":"ContainerStarted","Data":"fc28b82c0b9adfdd4963bbc542f6fd800a2537e06d35142962332789937f1214"}
Sep 10 03:07:03.668670 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:03.668535    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf" event={"ID":"90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4","Type":"ContainerStarted","Data":"5ccd6bb1024cdcc0d19922661ac701f44fda5fc27183e08ec72c73d182c6ee95"}
Sep 10 03:07:03.684030 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:03.683984    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf" podStartSLOduration=1.683935023 podStartE2EDuration="1.683935023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:07:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:07:03.682772229 +0000 UTC m=+1214.057858151" watchObservedRunningTime="2026-09-10 03:07:03.683935023 +0000 UTC m=+1214.059020944"
Sep 10 03:07:03.730801 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:03.730744    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:07:04.473699 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:04.473653    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:07:04.718854 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:04.718822    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:07:04.837326 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:04.837256    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:07:06.823507 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:06.823474    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:07:06.910198 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:06.910161    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:07:07.976706 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:07.976673    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:07:08.077359 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:08.077314    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:07:08.532741 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:08.532707    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:07:08.650777 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:08.650748    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:07:08.746531 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:08.746479    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:07:08.751273 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:08.751243    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7727-7dbbddd5d-mh874"]
Sep 10 03:07:08.754866 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:08.754851    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7727-7dbbddd5d-mh874"
Sep 10 03:07:08.757981 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:08.757943    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-af7727\""
Sep 10 03:07:08.769156 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:08.758501    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-af7727-dockercfg-6n8ts\""
Sep 10 03:07:08.774275 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:08.774247    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7727-7dbbddd5d-mh874"]
Sep 10 03:07:08.864388 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:08.864316    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7590551e-6d05-4757-bf5e-7ef219553510-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-af7727-7dbbddd5d-mh874\" (UID: \"7590551e-6d05-4757-bf5e-7ef219553510\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7727-7dbbddd5d-mh874"
Sep 10 03:07:08.864388 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:08.864363    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55bj2\" (UniqueName: \"kubernetes.io/projected/7590551e-6d05-4757-bf5e-7ef219553510-kube-api-access-55bj2\") pod \"payload-pre-processing-e2e-shared-a-w3-af7727-7dbbddd5d-mh874\" (UID: \"7590551e-6d05-4757-bf5e-7ef219553510\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7727-7dbbddd5d-mh874"
Sep 10 03:07:08.965392 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:08.965361    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55bj2\" (UniqueName: \"kubernetes.io/projected/7590551e-6d05-4757-bf5e-7ef219553510-kube-api-access-55bj2\") pod \"payload-pre-processing-e2e-shared-a-w3-af7727-7dbbddd5d-mh874\" (UID: \"7590551e-6d05-4757-bf5e-7ef219553510\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7727-7dbbddd5d-mh874"
Sep 10 03:07:08.965526 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:08.965453    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7590551e-6d05-4757-bf5e-7ef219553510-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-af7727-7dbbddd5d-mh874\" (UID: \"7590551e-6d05-4757-bf5e-7ef219553510\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7727-7dbbddd5d-mh874"
Sep 10 03:07:08.966079 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:08.966060    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7590551e-6d05-4757-bf5e-7ef219553510-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-af7727-7dbbddd5d-mh874\" (UID: \"7590551e-6d05-4757-bf5e-7ef219553510\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7727-7dbbddd5d-mh874"
Sep 10 03:07:08.974419 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:08.974393    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55bj2\" (UniqueName: \"kubernetes.io/projected/7590551e-6d05-4757-bf5e-7ef219553510-kube-api-access-55bj2\") pod \"payload-pre-processing-e2e-shared-a-w3-af7727-7dbbddd5d-mh874\" (UID: \"7590551e-6d05-4757-bf5e-7ef219553510\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7727-7dbbddd5d-mh874"
Sep 10 03:07:09.072586 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:09.072560    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7727-7dbbddd5d-mh874"
Sep 10 03:07:09.345239 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:09.345206    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7727-7dbbddd5d-mh874"]
Sep 10 03:07:09.346663 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:07:09.346637    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7590551e_6d05_4757_bf5e_7ef219553510.slice/crio-b8fea9efa24624b92e20bf7189fdb362ea8da5a2af1c5c45c813cbd693d61028 WatchSource:0}: Error finding container b8fea9efa24624b92e20bf7189fdb362ea8da5a2af1c5c45c813cbd693d61028: Status 404 returned error can't find the container with id b8fea9efa24624b92e20bf7189fdb362ea8da5a2af1c5c45c813cbd693d61028
Sep 10 03:07:09.707204 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:09.707163    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7727-7dbbddd5d-mh874" event={"ID":"7590551e-6d05-4757-bf5e-7ef219553510","Type":"ContainerStarted","Data":"f8de70542e7af396c759073cd9f02ddf64e68ea4066461d7c61309a107dc7cd5"}
Sep 10 03:07:09.707388 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:09.707220    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7727-7dbbddd5d-mh874" event={"ID":"7590551e-6d05-4757-bf5e-7ef219553510","Type":"ContainerStarted","Data":"b8fea9efa24624b92e20bf7189fdb362ea8da5a2af1c5c45c813cbd693d61028"}
Sep 10 03:07:09.707388 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:09.707246    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7727-7dbbddd5d-mh874"
Sep 10 03:07:09.725152 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:09.725089    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7727-7dbbddd5d-mh874" podStartSLOduration=1.7250724050000001 podStartE2EDuration="1.725072405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:07:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:07:09.722987407 +0000 UTC m=+1220.098073327" watchObservedRunningTime="2026-09-10 03:07:09.725072405 +0000 UTC m=+1220.100158342"
Sep 10 03:07:09.889492 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:09.889450    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:07:12.931789 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:12.931749    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:07:13.199291 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:13.199268    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf"
Sep 10 03:07:13.199436 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:13.199308    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf"
Sep 10 03:07:13.211809 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:13.211790    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf"
Sep 10 03:07:13.741334 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:13.741308    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf"
Sep 10 03:07:20.715464 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:20.715427    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7727-7dbbddd5d-mh874"
Sep 10 03:07:31.845462 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:31.845427    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:07:32.231178 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:32.231137    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:07:32.539923 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:32.539824    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:07:32.639493 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:32.639458    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:07:36.017801 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:36.017764    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:07:36.119940 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:36.119906    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:07:37.199877 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:37.199819    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:07:37.316661 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:37.316627    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:07:37.472998 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:37.472923    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:07:37.748758 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:37.748682    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:07:37.846948 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:37.846920    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:07:46.087375 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:46.087293    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:07:47.241614 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:47.241582    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:07:50.237600 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:07:50.237569    2558 scope.go:117] "RemoveContainer" containerID="47dd7c745f067fa436d78d79cdc6101557c1a72f5bf10037c7833506198555f7"
Sep 10 03:08:00.944328 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:00.944296    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:08:01.330122 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:01.330081    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft"]
Sep 10 03:08:01.335179 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:01.335155    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft"
Sep 10 03:08:01.338295 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:01.338273    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-61660fd5\"/\"kube-root-ca.crt\""
Sep 10 03:08:01.338434 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:01.338292    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-61660fd5\"/\"openshift-service-ca.crt\""
Sep 10 03:08:01.338947 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:01.338927    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-61660fd5\"/\"default-dockercfg-rzxp8\""
Sep 10 03:08:01.339002 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:01.338981    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-61660fd5\"/\"e2e-lifecycle-model-61660fd5-kserve-self-signed-certs\""
Sep 10 03:08:01.347352 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:01.347327    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft"]
Sep 10 03:08:01.361547 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:01.361496    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:08:01.382962 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:01.382941    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c17d129-0beb-4070-ae05-8d27c10e6bec-home\") pod \"e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft\" (UID: \"3c17d129-0beb-4070-ae05-8d27c10e6bec\") " pod="ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft"
Sep 10 03:08:01.383071 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:01.382972    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c17d129-0beb-4070-ae05-8d27c10e6bec-tmp-dir\") pod \"e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft\" (UID: \"3c17d129-0beb-4070-ae05-8d27c10e6bec\") " pod="ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft"
Sep 10 03:08:01.383071 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:01.383002    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c17d129-0beb-4070-ae05-8d27c10e6bec-dshm\") pod \"e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft\" (UID: \"3c17d129-0beb-4070-ae05-8d27c10e6bec\") " pod="ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft"
Sep 10 03:08:01.383184 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:01.383091    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c17d129-0beb-4070-ae05-8d27c10e6bec-model-cache\") pod \"e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft\" (UID: \"3c17d129-0beb-4070-ae05-8d27c10e6bec\") " pod="ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft"
Sep 10 03:08:01.383184 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:01.383145    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwjx9\" (UniqueName: \"kubernetes.io/projected/3c17d129-0beb-4070-ae05-8d27c10e6bec-kube-api-access-pwjx9\") pod \"e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft\" (UID: \"3c17d129-0beb-4070-ae05-8d27c10e6bec\") " pod="ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft"
Sep 10 03:08:01.383184 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:01.383173    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c17d129-0beb-4070-ae05-8d27c10e6bec-tls-certs\") pod \"e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft\" (UID: \"3c17d129-0beb-4070-ae05-8d27c10e6bec\") " pod="ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft"
Sep 10 03:08:01.483684 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:01.483663    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwjx9\" (UniqueName: \"kubernetes.io/projected/3c17d129-0beb-4070-ae05-8d27c10e6bec-kube-api-access-pwjx9\") pod \"e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft\" (UID: \"3c17d129-0beb-4070-ae05-8d27c10e6bec\") " pod="ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft"
Sep 10 03:08:01.483787 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:01.483698    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c17d129-0beb-4070-ae05-8d27c10e6bec-tls-certs\") pod \"e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft\" (UID: \"3c17d129-0beb-4070-ae05-8d27c10e6bec\") " pod="ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft"
Sep 10 03:08:01.483787 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:01.483734    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c17d129-0beb-4070-ae05-8d27c10e6bec-home\") pod \"e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft\" (UID: \"3c17d129-0beb-4070-ae05-8d27c10e6bec\") " pod="ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft"
Sep 10 03:08:01.483787 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:01.483764    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c17d129-0beb-4070-ae05-8d27c10e6bec-tmp-dir\") pod \"e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft\" (UID: \"3c17d129-0beb-4070-ae05-8d27c10e6bec\") " pod="ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft"
Sep 10 03:08:01.483921 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:01.483797    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c17d129-0beb-4070-ae05-8d27c10e6bec-dshm\") pod \"e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft\" (UID: \"3c17d129-0beb-4070-ae05-8d27c10e6bec\") " pod="ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft"
Sep 10 03:08:01.483921 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:01.483844    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c17d129-0beb-4070-ae05-8d27c10e6bec-model-cache\") pod \"e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft\" (UID: \"3c17d129-0beb-4070-ae05-8d27c10e6bec\") " pod="ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft"
Sep 10 03:08:01.484235 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:01.484212    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c17d129-0beb-4070-ae05-8d27c10e6bec-tmp-dir\") pod \"e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft\" (UID: \"3c17d129-0beb-4070-ae05-8d27c10e6bec\") " pod="ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft"
Sep 10 03:08:01.484336 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:01.484253    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c17d129-0beb-4070-ae05-8d27c10e6bec-model-cache\") pod \"e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft\" (UID: \"3c17d129-0beb-4070-ae05-8d27c10e6bec\") " pod="ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft"
Sep 10 03:08:01.484336 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:01.484272    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c17d129-0beb-4070-ae05-8d27c10e6bec-home\") pod \"e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft\" (UID: \"3c17d129-0beb-4070-ae05-8d27c10e6bec\") " pod="ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft"
Sep 10 03:08:01.485997 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:01.485977    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c17d129-0beb-4070-ae05-8d27c10e6bec-dshm\") pod \"e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft\" (UID: \"3c17d129-0beb-4070-ae05-8d27c10e6bec\") " pod="ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft"
Sep 10 03:08:01.486304 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:01.486284    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c17d129-0beb-4070-ae05-8d27c10e6bec-tls-certs\") pod \"e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft\" (UID: \"3c17d129-0beb-4070-ae05-8d27c10e6bec\") " pod="ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft"
Sep 10 03:08:01.492318 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:01.492297    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwjx9\" (UniqueName: \"kubernetes.io/projected/3c17d129-0beb-4070-ae05-8d27c10e6bec-kube-api-access-pwjx9\") pod \"e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft\" (UID: \"3c17d129-0beb-4070-ae05-8d27c10e6bec\") " pod="ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft"
Sep 10 03:08:01.645451 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:01.645390    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft"
Sep 10 03:08:01.924688 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:01.924477    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft"]
Sep 10 03:08:01.927015 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:08:01.926986    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c17d129_0beb_4070_ae05_8d27c10e6bec.slice/crio-b33e9f5588e7cf8e411382a617477eaea74981e4493bf706094688489268a0e0 WatchSource:0}: Error finding container b33e9f5588e7cf8e411382a617477eaea74981e4493bf706094688489268a0e0: Status 404 returned error can't find the container with id b33e9f5588e7cf8e411382a617477eaea74981e4493bf706094688489268a0e0
Sep 10 03:08:02.024735 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:02.024704    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft" event={"ID":"3c17d129-0beb-4070-ae05-8d27c10e6bec","Type":"ContainerStarted","Data":"56760e94ecc00f9d6fa028f6cfb0f932ead68ce3e74dbe734595f07e0aa785f6"}
Sep 10 03:08:02.025058 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:02.024750    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft" event={"ID":"3c17d129-0beb-4070-ae05-8d27c10e6bec","Type":"ContainerStarted","Data":"b33e9f5588e7cf8e411382a617477eaea74981e4493bf706094688489268a0e0"}
Sep 10 03:08:02.044239 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:02.044198    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft" podStartSLOduration=1.04418319 podStartE2EDuration="1.04418319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:08:02.042163114 +0000 UTC m=+1272.417249031" watchObservedRunningTime="2026-09-10 03:08:02.04418319 +0000 UTC m=+1272.419269110"
Sep 10 03:08:11.336792 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:11.336755    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:08:11.436538 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:11.436502    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:08:11.645726 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:11.645642    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft"
Sep 10 03:08:11.645726 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:11.645694    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft"
Sep 10 03:08:11.659455 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:11.659429    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft"
Sep 10 03:08:12.095166 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:12.095136    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft"
Sep 10 03:08:29.820789 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:29.820756    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c"]
Sep 10 03:08:29.824981 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:29.824959    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c"
Sep 10 03:08:29.827146 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:29.827114    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-097e16\"/\"unconfigured-w0-097e16-kserve-self-signed-certs\""
Sep 10 03:08:29.832818 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:29.832773    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c"]
Sep 10 03:08:29.840881 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:29.840852    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:08:29.902302 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:29.902273    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34443357-d4bd-4934-9910-dc5ebaf64125-home\") pod \"unconfigured-w0-097e16-kserve-6fd84696-v2v9c\" (UID: \"34443357-d4bd-4934-9910-dc5ebaf64125\") " pod="e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c"
Sep 10 03:08:29.902415 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:29.902316    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34443357-d4bd-4934-9910-dc5ebaf64125-tls-certs\") pod \"unconfigured-w0-097e16-kserve-6fd84696-v2v9c\" (UID: \"34443357-d4bd-4934-9910-dc5ebaf64125\") " pod="e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c"
Sep 10 03:08:29.902496 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:29.902473    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34443357-d4bd-4934-9910-dc5ebaf64125-tmp-dir\") pod \"unconfigured-w0-097e16-kserve-6fd84696-v2v9c\" (UID: \"34443357-d4bd-4934-9910-dc5ebaf64125\") " pod="e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c"
Sep 10 03:08:29.902546 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:29.902513    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34443357-d4bd-4934-9910-dc5ebaf64125-dshm\") pod \"unconfigured-w0-097e16-kserve-6fd84696-v2v9c\" (UID: \"34443357-d4bd-4934-9910-dc5ebaf64125\") " pod="e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c"
Sep 10 03:08:29.902546 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:29.902531    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34443357-d4bd-4934-9910-dc5ebaf64125-model-cache\") pod \"unconfigured-w0-097e16-kserve-6fd84696-v2v9c\" (UID: \"34443357-d4bd-4934-9910-dc5ebaf64125\") " pod="e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c"
Sep 10 03:08:29.902618 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:29.902562    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8fcr\" (UniqueName: \"kubernetes.io/projected/34443357-d4bd-4934-9910-dc5ebaf64125-kube-api-access-s8fcr\") pod \"unconfigured-w0-097e16-kserve-6fd84696-v2v9c\" (UID: \"34443357-d4bd-4934-9910-dc5ebaf64125\") " pod="e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c"
Sep 10 03:08:30.003157 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:30.003131    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34443357-d4bd-4934-9910-dc5ebaf64125-tmp-dir\") pod \"unconfigured-w0-097e16-kserve-6fd84696-v2v9c\" (UID: \"34443357-d4bd-4934-9910-dc5ebaf64125\") " pod="e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c"
Sep 10 03:08:30.003261 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:30.003165    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34443357-d4bd-4934-9910-dc5ebaf64125-dshm\") pod \"unconfigured-w0-097e16-kserve-6fd84696-v2v9c\" (UID: \"34443357-d4bd-4934-9910-dc5ebaf64125\") " pod="e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c"
Sep 10 03:08:30.003261 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:30.003181    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34443357-d4bd-4934-9910-dc5ebaf64125-model-cache\") pod \"unconfigured-w0-097e16-kserve-6fd84696-v2v9c\" (UID: \"34443357-d4bd-4934-9910-dc5ebaf64125\") " pod="e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c"
Sep 10 03:08:30.003261 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:30.003200    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8fcr\" (UniqueName: \"kubernetes.io/projected/34443357-d4bd-4934-9910-dc5ebaf64125-kube-api-access-s8fcr\") pod \"unconfigured-w0-097e16-kserve-6fd84696-v2v9c\" (UID: \"34443357-d4bd-4934-9910-dc5ebaf64125\") " pod="e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c"
Sep 10 03:08:30.003261 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:30.003220    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34443357-d4bd-4934-9910-dc5ebaf64125-home\") pod \"unconfigured-w0-097e16-kserve-6fd84696-v2v9c\" (UID: \"34443357-d4bd-4934-9910-dc5ebaf64125\") " pod="e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c"
Sep 10 03:08:30.003261 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:30.003252    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34443357-d4bd-4934-9910-dc5ebaf64125-tls-certs\") pod \"unconfigured-w0-097e16-kserve-6fd84696-v2v9c\" (UID: \"34443357-d4bd-4934-9910-dc5ebaf64125\") " pod="e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c"
Sep 10 03:08:30.003570 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:30.003550    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34443357-d4bd-4934-9910-dc5ebaf64125-tmp-dir\") pod \"unconfigured-w0-097e16-kserve-6fd84696-v2v9c\" (UID: \"34443357-d4bd-4934-9910-dc5ebaf64125\") " pod="e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c"
Sep 10 03:08:30.003629 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:30.003586    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34443357-d4bd-4934-9910-dc5ebaf64125-model-cache\") pod \"unconfigured-w0-097e16-kserve-6fd84696-v2v9c\" (UID: \"34443357-d4bd-4934-9910-dc5ebaf64125\") " pod="e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c"
Sep 10 03:08:30.003629 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:30.003621    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34443357-d4bd-4934-9910-dc5ebaf64125-home\") pod \"unconfigured-w0-097e16-kserve-6fd84696-v2v9c\" (UID: \"34443357-d4bd-4934-9910-dc5ebaf64125\") " pod="e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c"
Sep 10 03:08:30.005462 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:30.005437    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34443357-d4bd-4934-9910-dc5ebaf64125-dshm\") pod \"unconfigured-w0-097e16-kserve-6fd84696-v2v9c\" (UID: \"34443357-d4bd-4934-9910-dc5ebaf64125\") " pod="e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c"
Sep 10 03:08:30.005744 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:30.005727    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34443357-d4bd-4934-9910-dc5ebaf64125-tls-certs\") pod \"unconfigured-w0-097e16-kserve-6fd84696-v2v9c\" (UID: \"34443357-d4bd-4934-9910-dc5ebaf64125\") " pod="e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c"
Sep 10 03:08:30.011876 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:30.011853    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8fcr\" (UniqueName: \"kubernetes.io/projected/34443357-d4bd-4934-9910-dc5ebaf64125-kube-api-access-s8fcr\") pod \"unconfigured-w0-097e16-kserve-6fd84696-v2v9c\" (UID: \"34443357-d4bd-4934-9910-dc5ebaf64125\") " pod="e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c"
Sep 10 03:08:30.137961 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:30.137910    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c"
Sep 10 03:08:30.413154 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:30.413130    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c"]
Sep 10 03:08:30.414189 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:08:30.414162    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34443357_d4bd_4934_9910_dc5ebaf64125.slice/crio-54d6509a6493f95a13b3a7034164068773871bf46188c32dbf158c9dfc4ce5b2 WatchSource:0}: Error finding container 54d6509a6493f95a13b3a7034164068773871bf46188c32dbf158c9dfc4ce5b2: Status 404 returned error can't find the container with id 54d6509a6493f95a13b3a7034164068773871bf46188c32dbf158c9dfc4ce5b2
Sep 10 03:08:31.195558 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:31.195525    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c" event={"ID":"34443357-d4bd-4934-9910-dc5ebaf64125","Type":"ContainerStarted","Data":"9dff8e338f3f50d5797581f3557a571f50f022eeed20bcea4fe88b8350775957"}
Sep 10 03:08:31.195968 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:31.195568    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c" event={"ID":"34443357-d4bd-4934-9910-dc5ebaf64125","Type":"ContainerStarted","Data":"54d6509a6493f95a13b3a7034164068773871bf46188c32dbf158c9dfc4ce5b2"}
Sep 10 03:08:31.220061 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:31.220014    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c" podStartSLOduration=2.219998567 podStartE2EDuration="2.219998567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:08:31.217398 +0000 UTC m=+1301.592483921" watchObservedRunningTime="2026-09-10 03:08:31.219998567 +0000 UTC m=+1301.595084487"
Sep 10 03:08:38.427261 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:38.427232    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:08:39.575904 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:39.575875    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:08:39.634576 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:39.634543    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:08:40.138721 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:40.138684    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c"
Sep 10 03:08:40.138920 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:40.138731    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c"
Sep 10 03:08:40.150665 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:40.150634    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c"
Sep 10 03:08:40.259983 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:40.259955    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c"
Sep 10 03:08:41.973584 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:41.973544    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:08:54.709663 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:08:54.709626    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:09:20.394164 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:20.394131    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61660fd5-btldl"]
Sep 10 03:09:20.397894 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:20.397874    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61660fd5-btldl"
Sep 10 03:09:20.399967 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:20.399944    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8jjl\""
Sep 10 03:09:20.404784 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:20.404760    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61660fd5-btldl"]
Sep 10 03:09:20.503041 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:20.503014    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:09:20.510943 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:20.510918    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68216720-74ec-426f-b38f-d3250a3a1c13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-61660fd5-btldl\" (UID: \"68216720-74ec-426f-b38f-d3250a3a1c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61660fd5-btldl"
Sep 10 03:09:20.611529 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:20.611503    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68216720-74ec-426f-b38f-d3250a3a1c13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-61660fd5-btldl\" (UID: \"68216720-74ec-426f-b38f-d3250a3a1c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61660fd5-btldl"
Sep 10 03:09:20.612070 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:20.612045    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68216720-74ec-426f-b38f-d3250a3a1c13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-61660fd5-btldl\" (UID: \"68216720-74ec-426f-b38f-d3250a3a1c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61660fd5-btldl"
Sep 10 03:09:20.711919 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:20.711894    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61660fd5-btldl"
Sep 10 03:09:20.989182 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:20.989157    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61660fd5-btldl"]
Sep 10 03:09:20.990990 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:09:20.990961    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68216720_74ec_426f_b38f_d3250a3a1c13.slice/crio-11c9e7b920e228d8e724eb7c60cef77dcfaa443a1b38451eb04852d37f11c2bf WatchSource:0}: Error finding container 11c9e7b920e228d8e724eb7c60cef77dcfaa443a1b38451eb04852d37f11c2bf: Status 404 returned error can't find the container with id 11c9e7b920e228d8e724eb7c60cef77dcfaa443a1b38451eb04852d37f11c2bf
Sep 10 03:09:21.219383 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:21.219353    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:09:21.507311 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:21.507277    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61660fd5-btldl" event={"ID":"68216720-74ec-426f-b38f-d3250a3a1c13","Type":"ContainerStarted","Data":"d83240527519750e6bdcfdc15aa8e40cdb04827de1ff7b5dc6f41715805af3a8"}
Sep 10 03:09:21.507636 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:21.507337    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61660fd5-btldl" event={"ID":"68216720-74ec-426f-b38f-d3250a3a1c13","Type":"ContainerStarted","Data":"11c9e7b920e228d8e724eb7c60cef77dcfaa443a1b38451eb04852d37f11c2bf"}
Sep 10 03:09:21.524882 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:21.524798    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61660fd5-btldl" podStartSLOduration=1.524783438 podStartE2EDuration="1.524783438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:09:21.521854443 +0000 UTC m=+1351.896940365" watchObservedRunningTime="2026-09-10 03:09:21.524783438 +0000 UTC m=+1351.899869359"
Sep 10 03:09:21.985391 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:21.985364    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:09:22.515022 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:22.514984    2558 generic.go:356] "Generic (PLEG): container finished" podID="68216720-74ec-426f-b38f-d3250a3a1c13" containerID="d83240527519750e6bdcfdc15aa8e40cdb04827de1ff7b5dc6f41715805af3a8" exitCode=0
Sep 10 03:09:22.515434 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:22.515073    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61660fd5-btldl" event={"ID":"68216720-74ec-426f-b38f-d3250a3a1c13","Type":"ContainerDied","Data":"d83240527519750e6bdcfdc15aa8e40cdb04827de1ff7b5dc6f41715805af3a8"}
Sep 10 03:09:22.675682 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:22.675654    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:09:23.663435 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:23.663407    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61660fd5-btldl"
Sep 10 03:09:23.737011 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:23.736980    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68216720-74ec-426f-b38f-d3250a3a1c13-maas-api-service-ca\") pod \"68216720-74ec-426f-b38f-d3250a3a1c13\" (UID: \"68216720-74ec-426f-b38f-d3250a3a1c13\") "
Sep 10 03:09:23.737318 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:23.737297    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68216720-74ec-426f-b38f-d3250a3a1c13-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "68216720-74ec-426f-b38f-d3250a3a1c13" (UID: "68216720-74ec-426f-b38f-d3250a3a1c13"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:09:23.838233 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:23.838177    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68216720-74ec-426f-b38f-d3250a3a1c13-maas-api-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:09:24.527783 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:24.527751    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61660fd5-btldl"
Sep 10 03:09:24.527956 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:24.527792    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61660fd5-btldl" event={"ID":"68216720-74ec-426f-b38f-d3250a3a1c13","Type":"ContainerDied","Data":"11c9e7b920e228d8e724eb7c60cef77dcfaa443a1b38451eb04852d37f11c2bf"}
Sep 10 03:09:24.527956 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:24.527840    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11c9e7b920e228d8e724eb7c60cef77dcfaa443a1b38451eb04852d37f11c2bf"
Sep 10 03:09:30.656667 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:30.656630    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:09:36.308275 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:36.308242    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:09:36.332201 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:36.332171    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-61660fd5-5d75b5f79c-fxgbc"]
Sep 10 03:09:36.332402 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:36.332371    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-61660fd5-5d75b5f79c-fxgbc" podUID="b7cbe3da-8cad-4ce1-b97c-be9133355de9" containerName="payload-processing" containerID="cri-o://4bc56684e8706a876970316a5022b56be157274649db79c7c7cd93531fc93cc7" gracePeriod=30
Sep 10 03:09:36.343145 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:36.343119    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-61660fd5-77cccfc78b-r69c5"]
Sep 10 03:09:36.343395 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:36.343352    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-61660fd5-77cccfc78b-r69c5" podUID="d4e74c00-3d61-4c0f-be9e-520f22a69d7c" containerName="payload-pre-processing" containerID="cri-o://9e564db807daea2ee8a6f0cdfc23b78afd1e1140cd3dcebb9fb8e82869bcb83a" gracePeriod=30
Sep 10 03:09:36.386019 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:36.385971    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:09:36.457378 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:36.457312    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:09:36.522370 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:36.522338    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61660fd5-btldl"]
Sep 10 03:09:36.524623 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:36.524582    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:09:36.525190 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:36.525167    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61660fd5-btldl"]
Sep 10 03:09:36.603367 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:36.603340    2558 generic.go:356] "Generic (PLEG): container finished" podID="b7cbe3da-8cad-4ce1-b97c-be9133355de9" containerID="4bc56684e8706a876970316a5022b56be157274649db79c7c7cd93531fc93cc7" exitCode=0
Sep 10 03:09:36.603462 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:36.603411    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-61660fd5-5d75b5f79c-fxgbc" event={"ID":"b7cbe3da-8cad-4ce1-b97c-be9133355de9","Type":"ContainerDied","Data":"4bc56684e8706a876970316a5022b56be157274649db79c7c7cd93531fc93cc7"}
Sep 10 03:09:36.605438 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:36.605414    2558 generic.go:356] "Generic (PLEG): container finished" podID="d4e74c00-3d61-4c0f-be9e-520f22a69d7c" containerID="9e564db807daea2ee8a6f0cdfc23b78afd1e1140cd3dcebb9fb8e82869bcb83a" exitCode=0
Sep 10 03:09:36.605547 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:36.605491    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-61660fd5-77cccfc78b-r69c5" event={"ID":"d4e74c00-3d61-4c0f-be9e-520f22a69d7c","Type":"ContainerDied","Data":"9e564db807daea2ee8a6f0cdfc23b78afd1e1140cd3dcebb9fb8e82869bcb83a"}
Sep 10 03:09:36.628367 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:36.628348    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-61660fd5-5d75b5f79c-fxgbc"
Sep 10 03:09:36.631545 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:36.631528    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-61660fd5-77cccfc78b-r69c5"
Sep 10 03:09:36.743469 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:36.743442    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drdmt\" (UniqueName: \"kubernetes.io/projected/d4e74c00-3d61-4c0f-be9e-520f22a69d7c-kube-api-access-drdmt\") pod \"d4e74c00-3d61-4c0f-be9e-520f22a69d7c\" (UID: \"d4e74c00-3d61-4c0f-be9e-520f22a69d7c\") "
Sep 10 03:09:36.743588 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:36.743490    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmq2b\" (UniqueName: \"kubernetes.io/projected/b7cbe3da-8cad-4ce1-b97c-be9133355de9-kube-api-access-pmq2b\") pod \"b7cbe3da-8cad-4ce1-b97c-be9133355de9\" (UID: \"b7cbe3da-8cad-4ce1-b97c-be9133355de9\") "
Sep 10 03:09:36.743634 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:36.743613    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4e74c00-3d61-4c0f-be9e-520f22a69d7c-plugins-config-volume\") pod \"d4e74c00-3d61-4c0f-be9e-520f22a69d7c\" (UID: \"d4e74c00-3d61-4c0f-be9e-520f22a69d7c\") "
Sep 10 03:09:36.743726 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:36.743710    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7cbe3da-8cad-4ce1-b97c-be9133355de9-plugins-config-volume\") pod \"b7cbe3da-8cad-4ce1-b97c-be9133355de9\" (UID: \"b7cbe3da-8cad-4ce1-b97c-be9133355de9\") "
Sep 10 03:09:36.743988 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:36.743962    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4e74c00-3d61-4c0f-be9e-520f22a69d7c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d4e74c00-3d61-4c0f-be9e-520f22a69d7c" (UID: "d4e74c00-3d61-4c0f-be9e-520f22a69d7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:09:36.744090 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:36.744050    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4e74c00-3d61-4c0f-be9e-520f22a69d7c-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:09:36.744090 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:36.744046    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7cbe3da-8cad-4ce1-b97c-be9133355de9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b7cbe3da-8cad-4ce1-b97c-be9133355de9" (UID: "b7cbe3da-8cad-4ce1-b97c-be9133355de9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:09:36.745566 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:36.745537    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7cbe3da-8cad-4ce1-b97c-be9133355de9-kube-api-access-pmq2b" (OuterVolumeSpecName: "kube-api-access-pmq2b") pod "b7cbe3da-8cad-4ce1-b97c-be9133355de9" (UID: "b7cbe3da-8cad-4ce1-b97c-be9133355de9"). InnerVolumeSpecName "kube-api-access-pmq2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:09:36.745652 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:36.745627    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e74c00-3d61-4c0f-be9e-520f22a69d7c-kube-api-access-drdmt" (OuterVolumeSpecName: "kube-api-access-drdmt") pod "d4e74c00-3d61-4c0f-be9e-520f22a69d7c" (UID: "d4e74c00-3d61-4c0f-be9e-520f22a69d7c"). InnerVolumeSpecName "kube-api-access-drdmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:09:36.844452 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:36.844428    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7cbe3da-8cad-4ce1-b97c-be9133355de9-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:09:36.844452 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:36.844451    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drdmt\" (UniqueName: \"kubernetes.io/projected/d4e74c00-3d61-4c0f-be9e-520f22a69d7c-kube-api-access-drdmt\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:09:36.844577 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:36.844460    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmq2b\" (UniqueName: \"kubernetes.io/projected/b7cbe3da-8cad-4ce1-b97c-be9133355de9-kube-api-access-pmq2b\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:09:37.612957 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:37.612925    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-61660fd5-5d75b5f79c-fxgbc"
Sep 10 03:09:37.613409 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:37.612963    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-61660fd5-5d75b5f79c-fxgbc" event={"ID":"b7cbe3da-8cad-4ce1-b97c-be9133355de9","Type":"ContainerDied","Data":"7035809cfe8152e3f298849c9a96d79c537e66a9e4448b98c5727e8838aa4919"}
Sep 10 03:09:37.613409 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:37.613028    2558 scope.go:117] "RemoveContainer" containerID="4bc56684e8706a876970316a5022b56be157274649db79c7c7cd93531fc93cc7"
Sep 10 03:09:37.614476 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:37.614455    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-61660fd5-77cccfc78b-r69c5" event={"ID":"d4e74c00-3d61-4c0f-be9e-520f22a69d7c","Type":"ContainerDied","Data":"76265e0e3907648e11edf8d4959382883a7a3ff4cc9feb8fdb2fa2b52dd3e418"}
Sep 10 03:09:37.614548 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:37.614474    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-61660fd5-77cccfc78b-r69c5"
Sep 10 03:09:37.623559 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:37.623539    2558 scope.go:117] "RemoveContainer" containerID="9e564db807daea2ee8a6f0cdfc23b78afd1e1140cd3dcebb9fb8e82869bcb83a"
Sep 10 03:09:37.638969 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:37.638947    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-61660fd5-77cccfc78b-r69c5"]
Sep 10 03:09:37.642620 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:37.642602    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-61660fd5-77cccfc78b-r69c5"]
Sep 10 03:09:37.653985 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:37.653964    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-61660fd5-5d75b5f79c-fxgbc"]
Sep 10 03:09:37.659744 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:37.659722    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-61660fd5-5d75b5f79c-fxgbc"]
Sep 10 03:09:38.069331 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:38.069305    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68216720-74ec-426f-b38f-d3250a3a1c13" path="/var/lib/kubelet/pods/68216720-74ec-426f-b38f-d3250a3a1c13/volumes"
Sep 10 03:09:38.069652 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:38.069639    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7cbe3da-8cad-4ce1-b97c-be9133355de9" path="/var/lib/kubelet/pods/b7cbe3da-8cad-4ce1-b97c-be9133355de9/volumes"
Sep 10 03:09:38.069953 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:38.069941    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4e74c00-3d61-4c0f-be9e-520f22a69d7c" path="/var/lib/kubelet/pods/d4e74c00-3d61-4c0f-be9e-520f22a69d7c/volumes"
Sep 10 03:09:39.559484 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:39.559445    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:09:44.540486 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:44.540449    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft"]
Sep 10 03:09:44.587290 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:44.587259    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:09:47.308842 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:47.308780    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:09:50.276806 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:50.276776    2558 scope.go:117] "RemoveContainer" containerID="91df2f9a93a19607c68852ff59c34584db3f3d89b4a5432b8d388ccb800b6028"
Sep 10 03:09:52.421085 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:52.421052    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9"]
Sep 10 03:09:52.421704 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:52.421681    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7cbe3da-8cad-4ce1-b97c-be9133355de9" containerName="payload-processing"
Sep 10 03:09:52.421756 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:52.421708    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7cbe3da-8cad-4ce1-b97c-be9133355de9" containerName="payload-processing"
Sep 10 03:09:52.421756 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:52.421733    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68216720-74ec-426f-b38f-d3250a3a1c13" containerName="revoke-keys"
Sep 10 03:09:52.421756 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:52.421745    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="68216720-74ec-426f-b38f-d3250a3a1c13" containerName="revoke-keys"
Sep 10 03:09:52.421854 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:52.421776    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4e74c00-3d61-4c0f-be9e-520f22a69d7c" containerName="payload-pre-processing"
Sep 10 03:09:52.421854 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:52.421785    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4e74c00-3d61-4c0f-be9e-520f22a69d7c" containerName="payload-pre-processing"
Sep 10 03:09:52.421919 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:52.421887    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4e74c00-3d61-4c0f-be9e-520f22a69d7c" containerName="payload-pre-processing"
Sep 10 03:09:52.421919 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:52.421899    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7cbe3da-8cad-4ce1-b97c-be9133355de9" containerName="payload-processing"
Sep 10 03:09:52.421919 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:52.421909    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="68216720-74ec-426f-b38f-d3250a3a1c13" containerName="revoke-keys"
Sep 10 03:09:52.426841 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:52.426823    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9"
Sep 10 03:09:52.429231 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:52.429198    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-097e16\"/\"embedding-w0-097e16-kserve-self-signed-certs\""
Sep 10 03:09:52.433918 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:52.433836    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9"]
Sep 10 03:09:52.452563 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:52.452543    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:09:52.562231 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:52.562206    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4716-eaa6-450e-aa7f-5f88437939a0-dshm\") pod \"embedding-w0-097e16-kserve-56b66d4dcb-65qf9\" (UID: \"1d3f4716-eaa6-450e-aa7f-5f88437939a0\") " pod="e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9"
Sep 10 03:09:52.562346 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:52.562232    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4716-eaa6-450e-aa7f-5f88437939a0-tmp-dir\") pod \"embedding-w0-097e16-kserve-56b66d4dcb-65qf9\" (UID: \"1d3f4716-eaa6-450e-aa7f-5f88437939a0\") " pod="e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9"
Sep 10 03:09:52.562346 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:52.562250    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d3f4716-eaa6-450e-aa7f-5f88437939a0-tls-certs\") pod \"embedding-w0-097e16-kserve-56b66d4dcb-65qf9\" (UID: \"1d3f4716-eaa6-450e-aa7f-5f88437939a0\") " pod="e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9"
Sep 10 03:09:52.562346 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:52.562265    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4716-eaa6-450e-aa7f-5f88437939a0-home\") pod \"embedding-w0-097e16-kserve-56b66d4dcb-65qf9\" (UID: \"1d3f4716-eaa6-450e-aa7f-5f88437939a0\") " pod="e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9"
Sep 10 03:09:52.562474 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:52.562417    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4716-eaa6-450e-aa7f-5f88437939a0-model-cache\") pod \"embedding-w0-097e16-kserve-56b66d4dcb-65qf9\" (UID: \"1d3f4716-eaa6-450e-aa7f-5f88437939a0\") " pod="e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9"
Sep 10 03:09:52.562474 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:52.562457    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2zzw\" (UniqueName: \"kubernetes.io/projected/1d3f4716-eaa6-450e-aa7f-5f88437939a0-kube-api-access-x2zzw\") pod \"embedding-w0-097e16-kserve-56b66d4dcb-65qf9\" (UID: \"1d3f4716-eaa6-450e-aa7f-5f88437939a0\") " pod="e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9"
Sep 10 03:09:52.663922 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:52.663896    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4716-eaa6-450e-aa7f-5f88437939a0-model-cache\") pod \"embedding-w0-097e16-kserve-56b66d4dcb-65qf9\" (UID: \"1d3f4716-eaa6-450e-aa7f-5f88437939a0\") " pod="e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9"
Sep 10 03:09:52.664021 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:52.663933    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2zzw\" (UniqueName: \"kubernetes.io/projected/1d3f4716-eaa6-450e-aa7f-5f88437939a0-kube-api-access-x2zzw\") pod \"embedding-w0-097e16-kserve-56b66d4dcb-65qf9\" (UID: \"1d3f4716-eaa6-450e-aa7f-5f88437939a0\") " pod="e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9"
Sep 10 03:09:52.664021 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:52.664004    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4716-eaa6-450e-aa7f-5f88437939a0-dshm\") pod \"embedding-w0-097e16-kserve-56b66d4dcb-65qf9\" (UID: \"1d3f4716-eaa6-450e-aa7f-5f88437939a0\") " pod="e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9"
Sep 10 03:09:52.664162 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:52.664030    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4716-eaa6-450e-aa7f-5f88437939a0-tmp-dir\") pod \"embedding-w0-097e16-kserve-56b66d4dcb-65qf9\" (UID: \"1d3f4716-eaa6-450e-aa7f-5f88437939a0\") " pod="e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9"
Sep 10 03:09:52.664219 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:52.664159    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d3f4716-eaa6-450e-aa7f-5f88437939a0-tls-certs\") pod \"embedding-w0-097e16-kserve-56b66d4dcb-65qf9\" (UID: \"1d3f4716-eaa6-450e-aa7f-5f88437939a0\") " pod="e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9"
Sep 10 03:09:52.664219 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:52.664205    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4716-eaa6-450e-aa7f-5f88437939a0-home\") pod \"embedding-w0-097e16-kserve-56b66d4dcb-65qf9\" (UID: \"1d3f4716-eaa6-450e-aa7f-5f88437939a0\") " pod="e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9"
Sep 10 03:09:52.664353 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:52.664332    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4716-eaa6-450e-aa7f-5f88437939a0-model-cache\") pod \"embedding-w0-097e16-kserve-56b66d4dcb-65qf9\" (UID: \"1d3f4716-eaa6-450e-aa7f-5f88437939a0\") " pod="e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9"
Sep 10 03:09:52.664417 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:52.664353    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4716-eaa6-450e-aa7f-5f88437939a0-tmp-dir\") pod \"embedding-w0-097e16-kserve-56b66d4dcb-65qf9\" (UID: \"1d3f4716-eaa6-450e-aa7f-5f88437939a0\") " pod="e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9"
Sep 10 03:09:52.664617 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:52.664584    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4716-eaa6-450e-aa7f-5f88437939a0-home\") pod \"embedding-w0-097e16-kserve-56b66d4dcb-65qf9\" (UID: \"1d3f4716-eaa6-450e-aa7f-5f88437939a0\") " pod="e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9"
Sep 10 03:09:52.666358 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:52.666333    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4716-eaa6-450e-aa7f-5f88437939a0-dshm\") pod \"embedding-w0-097e16-kserve-56b66d4dcb-65qf9\" (UID: \"1d3f4716-eaa6-450e-aa7f-5f88437939a0\") " pod="e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9"
Sep 10 03:09:52.666813 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:52.666793    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d3f4716-eaa6-450e-aa7f-5f88437939a0-tls-certs\") pod \"embedding-w0-097e16-kserve-56b66d4dcb-65qf9\" (UID: \"1d3f4716-eaa6-450e-aa7f-5f88437939a0\") " pod="e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9"
Sep 10 03:09:52.671362 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:52.671307    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2zzw\" (UniqueName: \"kubernetes.io/projected/1d3f4716-eaa6-450e-aa7f-5f88437939a0-kube-api-access-x2zzw\") pod \"embedding-w0-097e16-kserve-56b66d4dcb-65qf9\" (UID: \"1d3f4716-eaa6-450e-aa7f-5f88437939a0\") " pod="e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9"
Sep 10 03:09:52.739301 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:52.739279    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9"
Sep 10 03:09:53.016371 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:53.016348    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9"]
Sep 10 03:09:53.017647 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:09:53.017623    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d3f4716_eaa6_450e_aa7f_5f88437939a0.slice/crio-b59e72e9d0705d082588870296a12eeb4c27480678c0d80339252193c4d8183c WatchSource:0}: Error finding container b59e72e9d0705d082588870296a12eeb4c27480678c0d80339252193c4d8183c: Status 404 returned error can't find the container with id b59e72e9d0705d082588870296a12eeb4c27480678c0d80339252193c4d8183c
Sep 10 03:09:53.714322 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:53.714289    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9" event={"ID":"1d3f4716-eaa6-450e-aa7f-5f88437939a0","Type":"ContainerStarted","Data":"540e9e09f0089dc87ef2096e9c8608ee98f58dac06c682bc0bd3501d06ff3b81"}
Sep 10 03:09:53.714322 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:53.714326    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9" event={"ID":"1d3f4716-eaa6-450e-aa7f-5f88437939a0","Type":"ContainerStarted","Data":"b59e72e9d0705d082588870296a12eeb4c27480678c0d80339252193c4d8183c"}
Sep 10 03:09:53.731759 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:53.731711    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9" podStartSLOduration=1.731696366 podStartE2EDuration="1.731696366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:09:53.731002707 +0000 UTC m=+1384.106088628" watchObservedRunningTime="2026-09-10 03:09:53.731696366 +0000 UTC m=+1384.106782287"
Sep 10 03:09:59.487096 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:59.487062    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:09:59.563732 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:59.563682    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft" podUID="3c17d129-0beb-4070-ae05-8d27c10e6bec" containerName="main" containerID="cri-o://56760e94ecc00f9d6fa028f6cfb0f932ead68ce3e74dbe734595f07e0aa785f6" gracePeriod=45
Sep 10 03:09:59.752315 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:59.752239    2558 generic.go:356] "Generic (PLEG): container finished" podID="3c17d129-0beb-4070-ae05-8d27c10e6bec" containerID="56760e94ecc00f9d6fa028f6cfb0f932ead68ce3e74dbe734595f07e0aa785f6" exitCode=0
Sep 10 03:09:59.752459 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:59.752326    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft" event={"ID":"3c17d129-0beb-4070-ae05-8d27c10e6bec","Type":"ContainerDied","Data":"56760e94ecc00f9d6fa028f6cfb0f932ead68ce3e74dbe734595f07e0aa785f6"}
Sep 10 03:09:59.816641 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:59.816609    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft"
Sep 10 03:09:59.915671 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:59.915625    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c17d129-0beb-4070-ae05-8d27c10e6bec-tls-certs\") pod \"3c17d129-0beb-4070-ae05-8d27c10e6bec\" (UID: \"3c17d129-0beb-4070-ae05-8d27c10e6bec\") "
Sep 10 03:09:59.915840 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:59.915715    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c17d129-0beb-4070-ae05-8d27c10e6bec-home\") pod \"3c17d129-0beb-4070-ae05-8d27c10e6bec\" (UID: \"3c17d129-0beb-4070-ae05-8d27c10e6bec\") "
Sep 10 03:09:59.915840 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:59.915740    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c17d129-0beb-4070-ae05-8d27c10e6bec-model-cache\") pod \"3c17d129-0beb-4070-ae05-8d27c10e6bec\" (UID: \"3c17d129-0beb-4070-ae05-8d27c10e6bec\") "
Sep 10 03:09:59.915840 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:59.915773    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c17d129-0beb-4070-ae05-8d27c10e6bec-dshm\") pod \"3c17d129-0beb-4070-ae05-8d27c10e6bec\" (UID: \"3c17d129-0beb-4070-ae05-8d27c10e6bec\") "
Sep 10 03:09:59.915840 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:59.915806    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwjx9\" (UniqueName: \"kubernetes.io/projected/3c17d129-0beb-4070-ae05-8d27c10e6bec-kube-api-access-pwjx9\") pod \"3c17d129-0beb-4070-ae05-8d27c10e6bec\" (UID: \"3c17d129-0beb-4070-ae05-8d27c10e6bec\") "
Sep 10 03:09:59.916043 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:59.915843    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c17d129-0beb-4070-ae05-8d27c10e6bec-tmp-dir\") pod \"3c17d129-0beb-4070-ae05-8d27c10e6bec\" (UID: \"3c17d129-0beb-4070-ae05-8d27c10e6bec\") "
Sep 10 03:09:59.916043 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:59.915954    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c17d129-0beb-4070-ae05-8d27c10e6bec-model-cache" (OuterVolumeSpecName: "model-cache") pod "3c17d129-0beb-4070-ae05-8d27c10e6bec" (UID: "3c17d129-0beb-4070-ae05-8d27c10e6bec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:09:59.916043 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:59.915985    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c17d129-0beb-4070-ae05-8d27c10e6bec-home" (OuterVolumeSpecName: "home") pod "3c17d129-0beb-4070-ae05-8d27c10e6bec" (UID: "3c17d129-0beb-4070-ae05-8d27c10e6bec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:09:59.916193 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:59.916147    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c17d129-0beb-4070-ae05-8d27c10e6bec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3c17d129-0beb-4070-ae05-8d27c10e6bec" (UID: "3c17d129-0beb-4070-ae05-8d27c10e6bec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:09:59.916282 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:59.916267    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c17d129-0beb-4070-ae05-8d27c10e6bec-home\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:09:59.916319 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:59.916284    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c17d129-0beb-4070-ae05-8d27c10e6bec-model-cache\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:09:59.916319 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:59.916295    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c17d129-0beb-4070-ae05-8d27c10e6bec-tmp-dir\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:09:59.917961 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:59.917937    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c17d129-0beb-4070-ae05-8d27c10e6bec-kube-api-access-pwjx9" (OuterVolumeSpecName: "kube-api-access-pwjx9") pod "3c17d129-0beb-4070-ae05-8d27c10e6bec" (UID: "3c17d129-0beb-4070-ae05-8d27c10e6bec"). InnerVolumeSpecName "kube-api-access-pwjx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:09:59.918222 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:59.918202    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c17d129-0beb-4070-ae05-8d27c10e6bec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3c17d129-0beb-4070-ae05-8d27c10e6bec" (UID: "3c17d129-0beb-4070-ae05-8d27c10e6bec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:09:59.918317 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:09:59.918220    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c17d129-0beb-4070-ae05-8d27c10e6bec-dshm" (OuterVolumeSpecName: "dshm") pod "3c17d129-0beb-4070-ae05-8d27c10e6bec" (UID: "3c17d129-0beb-4070-ae05-8d27c10e6bec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:10:00.017538 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:00.017483    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c17d129-0beb-4070-ae05-8d27c10e6bec-dshm\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:10:00.017538 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:00.017503    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwjx9\" (UniqueName: \"kubernetes.io/projected/3c17d129-0beb-4070-ae05-8d27c10e6bec-kube-api-access-pwjx9\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:10:00.017538 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:00.017512    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c17d129-0beb-4070-ae05-8d27c10e6bec-tls-certs\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:10:00.760731 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:00.760696    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft"
Sep 10 03:10:00.761205 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:00.760728    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft" event={"ID":"3c17d129-0beb-4070-ae05-8d27c10e6bec","Type":"ContainerDied","Data":"b33e9f5588e7cf8e411382a617477eaea74981e4493bf706094688489268a0e0"}
Sep 10 03:10:00.761205 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:00.760779    2558 scope.go:117] "RemoveContainer" containerID="56760e94ecc00f9d6fa028f6cfb0f932ead68ce3e74dbe734595f07e0aa785f6"
Sep 10 03:10:00.781264 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:00.781232    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft"]
Sep 10 03:10:00.787317 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:00.787284    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-61660fd5/e2e-lifecycle-model-61660fd5-kserve-7779fc9-ml7ft"]
Sep 10 03:10:02.069766 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:02.069731    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c17d129-0beb-4070-ae05-8d27c10e6bec" path="/var/lib/kubelet/pods/3c17d129-0beb-4070-ae05-8d27c10e6bec/volumes"
Sep 10 03:10:02.739979 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:02.739947    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9"
Sep 10 03:10:02.739979 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:02.739988    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9"
Sep 10 03:10:02.752150 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:02.752125    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9"
Sep 10 03:10:02.785583 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:02.785552    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9"
Sep 10 03:10:07.121201 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:07.121161    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:10:15.104279 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:15.104247    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2"]
Sep 10 03:10:15.104723 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:15.104655    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c17d129-0beb-4070-ae05-8d27c10e6bec" containerName="main"
Sep 10 03:10:15.104723 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:15.104666    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c17d129-0beb-4070-ae05-8d27c10e6bec" containerName="main"
Sep 10 03:10:15.104801 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:15.104727    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c17d129-0beb-4070-ae05-8d27c10e6bec" containerName="main"
Sep 10 03:10:15.107914 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:15.107896    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2"
Sep 10 03:10:15.110405 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:15.110382    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e6e6b8\"/\"embedding-w4-e6e6b8-kserve-self-signed-certs\""
Sep 10 03:10:15.117039 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:15.117012    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2"]
Sep 10 03:10:15.138628 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:15.138590    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:10:15.247253 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:15.247228    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b74f9d2-9e8c-478f-9f38-1fef910c943c-home\") pod \"embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2\" (UID: \"5b74f9d2-9e8c-478f-9f38-1fef910c943c\") " pod="e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2"
Sep 10 03:10:15.247349 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:15.247260    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b74f9d2-9e8c-478f-9f38-1fef910c943c-model-cache\") pod \"embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2\" (UID: \"5b74f9d2-9e8c-478f-9f38-1fef910c943c\") " pod="e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2"
Sep 10 03:10:15.247349 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:15.247281    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b74f9d2-9e8c-478f-9f38-1fef910c943c-tmp-dir\") pod \"embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2\" (UID: \"5b74f9d2-9e8c-478f-9f38-1fef910c943c\") " pod="e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2"
Sep 10 03:10:15.247349 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:15.247295    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwjb8\" (UniqueName: \"kubernetes.io/projected/5b74f9d2-9e8c-478f-9f38-1fef910c943c-kube-api-access-jwjb8\") pod \"embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2\" (UID: \"5b74f9d2-9e8c-478f-9f38-1fef910c943c\") " pod="e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2"
Sep 10 03:10:15.247459 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:15.247373    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b74f9d2-9e8c-478f-9f38-1fef910c943c-dshm\") pod \"embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2\" (UID: \"5b74f9d2-9e8c-478f-9f38-1fef910c943c\") " pod="e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2"
Sep 10 03:10:15.247499 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:15.247457    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b74f9d2-9e8c-478f-9f38-1fef910c943c-tls-certs\") pod \"embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2\" (UID: \"5b74f9d2-9e8c-478f-9f38-1fef910c943c\") " pod="e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2"
Sep 10 03:10:15.348808 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:15.348784    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b74f9d2-9e8c-478f-9f38-1fef910c943c-dshm\") pod \"embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2\" (UID: \"5b74f9d2-9e8c-478f-9f38-1fef910c943c\") " pod="e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2"
Sep 10 03:10:15.348902 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:15.348825    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b74f9d2-9e8c-478f-9f38-1fef910c943c-tls-certs\") pod \"embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2\" (UID: \"5b74f9d2-9e8c-478f-9f38-1fef910c943c\") " pod="e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2"
Sep 10 03:10:15.348902 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:15.348870    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b74f9d2-9e8c-478f-9f38-1fef910c943c-home\") pod \"embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2\" (UID: \"5b74f9d2-9e8c-478f-9f38-1fef910c943c\") " pod="e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2"
Sep 10 03:10:15.349000 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:15.348962    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b74f9d2-9e8c-478f-9f38-1fef910c943c-model-cache\") pod \"embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2\" (UID: \"5b74f9d2-9e8c-478f-9f38-1fef910c943c\") " pod="e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2"
Sep 10 03:10:15.349000 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:15.348994    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b74f9d2-9e8c-478f-9f38-1fef910c943c-tmp-dir\") pod \"embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2\" (UID: \"5b74f9d2-9e8c-478f-9f38-1fef910c943c\") " pod="e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2"
Sep 10 03:10:15.349083 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:15.349011    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwjb8\" (UniqueName: \"kubernetes.io/projected/5b74f9d2-9e8c-478f-9f38-1fef910c943c-kube-api-access-jwjb8\") pod \"embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2\" (UID: \"5b74f9d2-9e8c-478f-9f38-1fef910c943c\") " pod="e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2"
Sep 10 03:10:15.349287 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:15.349258    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b74f9d2-9e8c-478f-9f38-1fef910c943c-home\") pod \"embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2\" (UID: \"5b74f9d2-9e8c-478f-9f38-1fef910c943c\") " pod="e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2"
Sep 10 03:10:15.349287 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:15.349272    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b74f9d2-9e8c-478f-9f38-1fef910c943c-model-cache\") pod \"embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2\" (UID: \"5b74f9d2-9e8c-478f-9f38-1fef910c943c\") " pod="e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2"
Sep 10 03:10:15.349418 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:15.349354    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b74f9d2-9e8c-478f-9f38-1fef910c943c-tmp-dir\") pod \"embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2\" (UID: \"5b74f9d2-9e8c-478f-9f38-1fef910c943c\") " pod="e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2"
Sep 10 03:10:15.351590 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:15.351572    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b74f9d2-9e8c-478f-9f38-1fef910c943c-dshm\") pod \"embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2\" (UID: \"5b74f9d2-9e8c-478f-9f38-1fef910c943c\") " pod="e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2"
Sep 10 03:10:15.351755 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:15.351739    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b74f9d2-9e8c-478f-9f38-1fef910c943c-tls-certs\") pod \"embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2\" (UID: \"5b74f9d2-9e8c-478f-9f38-1fef910c943c\") " pod="e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2"
Sep 10 03:10:15.358414 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:15.358350    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwjb8\" (UniqueName: \"kubernetes.io/projected/5b74f9d2-9e8c-478f-9f38-1fef910c943c-kube-api-access-jwjb8\") pod \"embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2\" (UID: \"5b74f9d2-9e8c-478f-9f38-1fef910c943c\") " pod="e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2"
Sep 10 03:10:15.418919 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:15.418893    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2"
Sep 10 03:10:15.699498 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:15.699458    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2"]
Sep 10 03:10:15.700862 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:10:15.700830    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b74f9d2_9e8c_478f_9f38_1fef910c943c.slice/crio-827fddedfda15bf924b1ae6f99acdcdb3d6e8c07ac51bba6df793aad4f8486be WatchSource:0}: Error finding container 827fddedfda15bf924b1ae6f99acdcdb3d6e8c07ac51bba6df793aad4f8486be: Status 404 returned error can't find the container with id 827fddedfda15bf924b1ae6f99acdcdb3d6e8c07ac51bba6df793aad4f8486be
Sep 10 03:10:15.854446 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:15.854413    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2" event={"ID":"5b74f9d2-9e8c-478f-9f38-1fef910c943c","Type":"ContainerStarted","Data":"79ba55ebcfba69e042beefd8f8d74ec5beacfcd1facbf5734f47bb19f0d50d17"}
Sep 10 03:10:15.854600 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:15.854456    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2" event={"ID":"5b74f9d2-9e8c-478f-9f38-1fef910c943c","Type":"ContainerStarted","Data":"827fddedfda15bf924b1ae6f99acdcdb3d6e8c07ac51bba6df793aad4f8486be"}
Sep 10 03:10:15.871941 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:15.871893    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2" podStartSLOduration=0.871877722 podStartE2EDuration="871.877722ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:10:15.870562896 +0000 UTC m=+1406.245648841" watchObservedRunningTime="2026-09-10 03:10:15.871877722 +0000 UTC m=+1406.246963670"
Sep 10 03:10:16.445510 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:16.445474    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:10:21.049607 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:21.049572    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:10:22.202489 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:22.202445    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:10:25.419425 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:25.419393    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2"
Sep 10 03:10:25.419818 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:25.419441    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2"
Sep 10 03:10:25.431719 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:25.431693    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2"
Sep 10 03:10:25.929578 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:25.929544    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2"
Sep 10 03:10:27.714749 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:27.714722    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:10:33.637991 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:33.637950    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:10:33.739729 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:33.739704    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:10:33.958691 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:33.958657    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:10:35.134802 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:35.134770    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:10:35.233644 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:35.233614    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:10:35.334555 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:35.334523    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:10:35.912361 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:35.912308    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:10:36.013329 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:36.013278    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:10:36.112294 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:36.112238    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:10:37.312803 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:37.312753    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:10:43.249249 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:43.249164    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:10:43.350417 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:43.350381    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:10:43.904604 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:43.904569    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:10:49.538248 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:49.538214    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:10:49.606126 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:49.606073    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:10:49.688370 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:49.688337    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:10:49.787274 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:49.787242    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:10:50.306710 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:10:50.306682    2558 scope.go:117] "RemoveContainer" containerID="cc42b57a93366c59e033e17f77e9e634e0ca1c8d11a331fd0068438158377793"
Sep 10 03:11:00.237582 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:00.237542    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:00.478449 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:00.478415    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:16.764364 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:16.764268    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:16.842782 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:16.842753    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-097e16-x9nbh"]
Sep 10 03:11:16.847814 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:16.847798    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-097e16-x9nbh"
Sep 10 03:11:16.850155 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:16.850132    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8jjl\""
Sep 10 03:11:16.854039 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:16.854019    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-097e16-x9nbh"]
Sep 10 03:11:16.945737 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:16.945709    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b4bef4f-2b97-4307-a797-66dbbe75099c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-097e16-x9nbh\" (UID: \"4b4bef4f-2b97-4307-a797-66dbbe75099c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-097e16-x9nbh"
Sep 10 03:11:17.012280 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:17.012251    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:17.046279 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:17.046217    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b4bef4f-2b97-4307-a797-66dbbe75099c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-097e16-x9nbh\" (UID: \"4b4bef4f-2b97-4307-a797-66dbbe75099c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-097e16-x9nbh"
Sep 10 03:11:17.046830 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:17.046805    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b4bef4f-2b97-4307-a797-66dbbe75099c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-097e16-x9nbh\" (UID: \"4b4bef4f-2b97-4307-a797-66dbbe75099c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-097e16-x9nbh"
Sep 10 03:11:17.159732 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:17.159685    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-097e16-x9nbh"
Sep 10 03:11:17.439514 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:17.439488    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-097e16-x9nbh"]
Sep 10 03:11:17.440742 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:11:17.440713    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b4bef4f_2b97_4307_a797_66dbbe75099c.slice/crio-39dd02569a03528768873fc1b93bd29cb7200a3bec525964e3b38ce6352b84ca WatchSource:0}: Error finding container 39dd02569a03528768873fc1b93bd29cb7200a3bec525964e3b38ce6352b84ca: Status 404 returned error can't find the container with id 39dd02569a03528768873fc1b93bd29cb7200a3bec525964e3b38ce6352b84ca
Sep 10 03:11:18.237198 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:18.237170    2558 generic.go:356] "Generic (PLEG): container finished" podID="4b4bef4f-2b97-4307-a797-66dbbe75099c" containerID="327495bcc2ac3e7c0ea27c2e87a7c5c77f85e5b309bc1866f66fe54b7d136323" exitCode=0
Sep 10 03:11:18.237567 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:18.237262    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-097e16-x9nbh" event={"ID":"4b4bef4f-2b97-4307-a797-66dbbe75099c","Type":"ContainerDied","Data":"327495bcc2ac3e7c0ea27c2e87a7c5c77f85e5b309bc1866f66fe54b7d136323"}
Sep 10 03:11:18.237567 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:18.237323    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-097e16-x9nbh" event={"ID":"4b4bef4f-2b97-4307-a797-66dbbe75099c","Type":"ContainerStarted","Data":"39dd02569a03528768873fc1b93bd29cb7200a3bec525964e3b38ce6352b84ca"}
Sep 10 03:11:19.389291 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:19.389262    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-097e16-x9nbh"
Sep 10 03:11:19.467363 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:19.467333    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b4bef4f-2b97-4307-a797-66dbbe75099c-maas-api-service-ca\") pod \"4b4bef4f-2b97-4307-a797-66dbbe75099c\" (UID: \"4b4bef4f-2b97-4307-a797-66dbbe75099c\") "
Sep 10 03:11:19.467682 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:19.467655    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b4bef4f-2b97-4307-a797-66dbbe75099c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4b4bef4f-2b97-4307-a797-66dbbe75099c" (UID: "4b4bef4f-2b97-4307-a797-66dbbe75099c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:11:19.568822 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:19.568768    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b4bef4f-2b97-4307-a797-66dbbe75099c-maas-api-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:11:20.250728 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:20.250703    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-097e16-x9nbh"
Sep 10 03:11:20.250728 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:20.250708    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-097e16-x9nbh" event={"ID":"4b4bef4f-2b97-4307-a797-66dbbe75099c","Type":"ContainerDied","Data":"39dd02569a03528768873fc1b93bd29cb7200a3bec525964e3b38ce6352b84ca"}
Sep 10 03:11:20.250917 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:20.250751    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39dd02569a03528768873fc1b93bd29cb7200a3bec525964e3b38ce6352b84ca"
Sep 10 03:11:21.386493 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:21.386456    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:21.735373 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:21.735343    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9"]
Sep 10 03:11:21.743487 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:21.743460    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m"]
Sep 10 03:11:21.764898 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:21.764870    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c"]
Sep 10 03:11:21.830975 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:21.830907    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:24.109209 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:24.109175    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:27.131621 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:27.131588    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:27.231836 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:27.231807    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:27.464045 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:27.464012    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:27.774176 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:27.774087    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:27.875291 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:27.875258    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:31.330140 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:31.330083    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:31.415675 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:31.415647    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:32.467024 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:32.466985    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:32.569598 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:32.569568    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:32.713875 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:32.713849    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:32.951273 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:32.951246    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:33.023221 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.023193    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:33.051634 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.051608    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f0a342-cfbfbb758-2m28m"]
Sep 10 03:11:33.052014 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.052001    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b4bef4f-2b97-4307-a797-66dbbe75099c" containerName="revoke-keys"
Sep 10 03:11:33.052071 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.052015    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b4bef4f-2b97-4307-a797-66dbbe75099c" containerName="revoke-keys"
Sep 10 03:11:33.052123 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.052085    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b4bef4f-2b97-4307-a797-66dbbe75099c" containerName="revoke-keys"
Sep 10 03:11:33.054710 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.054693    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0a342-cfbfbb758-2m28m"
Sep 10 03:11:33.057332 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.057311    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f0a342\""
Sep 10 03:11:33.065280 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.065262    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f0a342-cfbfbb758-2m28m"]
Sep 10 03:11:33.084820 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.084792    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f0a342-79c5677d67-bjtwm"]
Sep 10 03:11:33.087272 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.087253    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f0a342-79c5677d67-bjtwm"
Sep 10 03:11:33.089507 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.089490    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f0a342-dockercfg-gm6m5\""
Sep 10 03:11:33.103851 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.103827    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f0a342-79c5677d67-bjtwm"]
Sep 10 03:11:33.180626 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.180598    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c73fe4f-e350-418f-bdc9-8f2f83825c9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f0a342-cfbfbb758-2m28m\" (UID: \"3c73fe4f-e350-418f-bdc9-8f2f83825c9c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0a342-cfbfbb758-2m28m"
Sep 10 03:11:33.180748 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.180640    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f628aac-05cc-4933-8887-1524945a483a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f0a342-79c5677d67-bjtwm\" (UID: \"3f628aac-05cc-4933-8887-1524945a483a\") " pod="openshift-ingress/payload-processing-e2e-api-a-f0a342-79c5677d67-bjtwm"
Sep 10 03:11:33.180827 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.180740    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mmdj\" (UniqueName: \"kubernetes.io/projected/3c73fe4f-e350-418f-bdc9-8f2f83825c9c-kube-api-access-2mmdj\") pod \"payload-pre-processing-e2e-api-a-f0a342-cfbfbb758-2m28m\" (UID: \"3c73fe4f-e350-418f-bdc9-8f2f83825c9c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0a342-cfbfbb758-2m28m"
Sep 10 03:11:33.180896 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.180819    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w26z8\" (UniqueName: \"kubernetes.io/projected/3f628aac-05cc-4933-8887-1524945a483a-kube-api-access-w26z8\") pod \"payload-processing-e2e-api-a-f0a342-79c5677d67-bjtwm\" (UID: \"3f628aac-05cc-4933-8887-1524945a483a\") " pod="openshift-ingress/payload-processing-e2e-api-a-f0a342-79c5677d67-bjtwm"
Sep 10 03:11:33.271025 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.270953    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:33.281911 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.281882    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c73fe4f-e350-418f-bdc9-8f2f83825c9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f0a342-cfbfbb758-2m28m\" (UID: \"3c73fe4f-e350-418f-bdc9-8f2f83825c9c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0a342-cfbfbb758-2m28m"
Sep 10 03:11:33.282052 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.281969    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f628aac-05cc-4933-8887-1524945a483a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f0a342-79c5677d67-bjtwm\" (UID: \"3f628aac-05cc-4933-8887-1524945a483a\") " pod="openshift-ingress/payload-processing-e2e-api-a-f0a342-79c5677d67-bjtwm"
Sep 10 03:11:33.282052 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.282035    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mmdj\" (UniqueName: \"kubernetes.io/projected/3c73fe4f-e350-418f-bdc9-8f2f83825c9c-kube-api-access-2mmdj\") pod \"payload-pre-processing-e2e-api-a-f0a342-cfbfbb758-2m28m\" (UID: \"3c73fe4f-e350-418f-bdc9-8f2f83825c9c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0a342-cfbfbb758-2m28m"
Sep 10 03:11:33.282199 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.282066    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w26z8\" (UniqueName: \"kubernetes.io/projected/3f628aac-05cc-4933-8887-1524945a483a-kube-api-access-w26z8\") pod \"payload-processing-e2e-api-a-f0a342-79c5677d67-bjtwm\" (UID: \"3f628aac-05cc-4933-8887-1524945a483a\") " pod="openshift-ingress/payload-processing-e2e-api-a-f0a342-79c5677d67-bjtwm"
Sep 10 03:11:33.282903 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.282866    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c73fe4f-e350-418f-bdc9-8f2f83825c9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f0a342-cfbfbb758-2m28m\" (UID: \"3c73fe4f-e350-418f-bdc9-8f2f83825c9c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0a342-cfbfbb758-2m28m"
Sep 10 03:11:33.283023 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.282866    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f628aac-05cc-4933-8887-1524945a483a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f0a342-79c5677d67-bjtwm\" (UID: \"3f628aac-05cc-4933-8887-1524945a483a\") " pod="openshift-ingress/payload-processing-e2e-api-a-f0a342-79c5677d67-bjtwm"
Sep 10 03:11:33.292523 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.292494    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-097e16-65ddd76cdc-nrwtx"]
Sep 10 03:11:33.292760 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.292729    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-097e16-65ddd76cdc-nrwtx" podUID="a1792e8c-6841-4193-9ca0-53ecbe580290" containerName="payload-processing" containerID="cri-o://3b9ab0504c679f5b32edb381d00600aed5c563bbbba9b14279710ba4e49e8997" gracePeriod=30
Sep 10 03:11:33.297542 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.297507    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w26z8\" (UniqueName: \"kubernetes.io/projected/3f628aac-05cc-4933-8887-1524945a483a-kube-api-access-w26z8\") pod \"payload-processing-e2e-api-a-f0a342-79c5677d67-bjtwm\" (UID: \"3f628aac-05cc-4933-8887-1524945a483a\") " pod="openshift-ingress/payload-processing-e2e-api-a-f0a342-79c5677d67-bjtwm"
Sep 10 03:11:33.298536 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.298507    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mmdj\" (UniqueName: \"kubernetes.io/projected/3c73fe4f-e350-418f-bdc9-8f2f83825c9c-kube-api-access-2mmdj\") pod \"payload-pre-processing-e2e-api-a-f0a342-cfbfbb758-2m28m\" (UID: \"3c73fe4f-e350-418f-bdc9-8f2f83825c9c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0a342-cfbfbb758-2m28m"
Sep 10 03:11:33.306026 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.306002    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-097e16-546949c98-csmxk"]
Sep 10 03:11:33.306286 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.306258    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-097e16-546949c98-csmxk" podUID="a4255849-3d3c-4e26-b766-6d1474682843" containerName="payload-pre-processing" containerID="cri-o://5d8296c90ea9f4938e466d1b282417091299b9e72ea088066757fc3a30725875" gracePeriod=30
Sep 10 03:11:33.366906 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.366875    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:33.368844 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.368817    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0a342-cfbfbb758-2m28m"
Sep 10 03:11:33.396946 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.396919    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f0a342-79c5677d67-bjtwm"
Sep 10 03:11:33.468306 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.468249    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:33.566946 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.566867    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:33.598863 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.598839    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-097e16-65ddd76cdc-nrwtx"
Sep 10 03:11:33.611309 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.611287    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-097e16-546949c98-csmxk"
Sep 10 03:11:33.642080 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.642050    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:33.643472 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.643448    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f0a342-cfbfbb758-2m28m"]
Sep 10 03:11:33.644686 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:11:33.644664    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c73fe4f_e350_418f_bdc9_8f2f83825c9c.slice/crio-a315bee18d16db8919880836e9e6f30bc914596e94ee5679dbdfdf873f64126a WatchSource:0}: Error finding container a315bee18d16db8919880836e9e6f30bc914596e94ee5679dbdfdf873f64126a: Status 404 returned error can't find the container with id a315bee18d16db8919880836e9e6f30bc914596e94ee5679dbdfdf873f64126a
Sep 10 03:11:33.686356 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.686335    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9pn5\" (UniqueName: \"kubernetes.io/projected/a1792e8c-6841-4193-9ca0-53ecbe580290-kube-api-access-k9pn5\") pod \"a1792e8c-6841-4193-9ca0-53ecbe580290\" (UID: \"a1792e8c-6841-4193-9ca0-53ecbe580290\") "
Sep 10 03:11:33.686449 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.686365    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4255849-3d3c-4e26-b766-6d1474682843-plugins-config-volume\") pod \"a4255849-3d3c-4e26-b766-6d1474682843\" (UID: \"a4255849-3d3c-4e26-b766-6d1474682843\") "
Sep 10 03:11:33.686449 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.686412    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47788\" (UniqueName: \"kubernetes.io/projected/a4255849-3d3c-4e26-b766-6d1474682843-kube-api-access-47788\") pod \"a4255849-3d3c-4e26-b766-6d1474682843\" (UID: \"a4255849-3d3c-4e26-b766-6d1474682843\") "
Sep 10 03:11:33.686522 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.686483    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1792e8c-6841-4193-9ca0-53ecbe580290-plugins-config-volume\") pod \"a1792e8c-6841-4193-9ca0-53ecbe580290\" (UID: \"a1792e8c-6841-4193-9ca0-53ecbe580290\") "
Sep 10 03:11:33.686904 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.686874    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4255849-3d3c-4e26-b766-6d1474682843-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a4255849-3d3c-4e26-b766-6d1474682843" (UID: "a4255849-3d3c-4e26-b766-6d1474682843"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:11:33.687003 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.686899    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1792e8c-6841-4193-9ca0-53ecbe580290-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a1792e8c-6841-4193-9ca0-53ecbe580290" (UID: "a1792e8c-6841-4193-9ca0-53ecbe580290"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:11:33.688437 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.688416    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1792e8c-6841-4193-9ca0-53ecbe580290-kube-api-access-k9pn5" (OuterVolumeSpecName: "kube-api-access-k9pn5") pod "a1792e8c-6841-4193-9ca0-53ecbe580290" (UID: "a1792e8c-6841-4193-9ca0-53ecbe580290"). InnerVolumeSpecName "kube-api-access-k9pn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:11:33.688574 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.688553    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4255849-3d3c-4e26-b766-6d1474682843-kube-api-access-47788" (OuterVolumeSpecName: "kube-api-access-47788") pod "a4255849-3d3c-4e26-b766-6d1474682843" (UID: "a4255849-3d3c-4e26-b766-6d1474682843"). InnerVolumeSpecName "kube-api-access-47788". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:11:33.694488 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:11:33.694468    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f628aac_05cc_4933_8887_1524945a483a.slice/crio-f26090c23284dcc2a27de10dd7e578ef3befaca405532ccbf4b0d425218d1f3c WatchSource:0}: Error finding container f26090c23284dcc2a27de10dd7e578ef3befaca405532ccbf4b0d425218d1f3c: Status 404 returned error can't find the container with id f26090c23284dcc2a27de10dd7e578ef3befaca405532ccbf4b0d425218d1f3c
Sep 10 03:11:33.694836 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.694802    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f0a342-79c5677d67-bjtwm"]
Sep 10 03:11:33.716295 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.716086    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-097e16-x9nbh"]
Sep 10 03:11:33.720955 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.720899    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-097e16-x9nbh"]
Sep 10 03:11:33.787943 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.787910    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47788\" (UniqueName: \"kubernetes.io/projected/a4255849-3d3c-4e26-b766-6d1474682843-kube-api-access-47788\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:11:33.787943 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.787936    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1792e8c-6841-4193-9ca0-53ecbe580290-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:11:33.787943 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.787947    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9pn5\" (UniqueName: \"kubernetes.io/projected/a1792e8c-6841-4193-9ca0-53ecbe580290-kube-api-access-k9pn5\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:11:33.788256 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:33.787955    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4255849-3d3c-4e26-b766-6d1474682843-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:11:34.021542 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:34.021501    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:34.070520 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:34.070488    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b4bef4f-2b97-4307-a797-66dbbe75099c" path="/var/lib/kubelet/pods/4b4bef4f-2b97-4307-a797-66dbbe75099c/volumes"
Sep 10 03:11:34.338838 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:34.338811    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f0a342-79c5677d67-bjtwm" event={"ID":"3f628aac-05cc-4933-8887-1524945a483a","Type":"ContainerStarted","Data":"6422ac4918a26f97a8f34d633d0bb8974281da3df7cedf6c2ff241fa624d5b6f"}
Sep 10 03:11:34.338984 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:34.338869    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f0a342-79c5677d67-bjtwm" event={"ID":"3f628aac-05cc-4933-8887-1524945a483a","Type":"ContainerStarted","Data":"f26090c23284dcc2a27de10dd7e578ef3befaca405532ccbf4b0d425218d1f3c"}
Sep 10 03:11:34.338984 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:34.338935    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f0a342-79c5677d67-bjtwm"
Sep 10 03:11:34.340171 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:34.340149    2558 generic.go:356] "Generic (PLEG): container finished" podID="a4255849-3d3c-4e26-b766-6d1474682843" containerID="5d8296c90ea9f4938e466d1b282417091299b9e72ea088066757fc3a30725875" exitCode=0
Sep 10 03:11:34.340265 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:34.340198    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-097e16-546949c98-csmxk"
Sep 10 03:11:34.340265 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:34.340234    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-097e16-546949c98-csmxk" event={"ID":"a4255849-3d3c-4e26-b766-6d1474682843","Type":"ContainerDied","Data":"5d8296c90ea9f4938e466d1b282417091299b9e72ea088066757fc3a30725875"}
Sep 10 03:11:34.340349 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:34.340275    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-097e16-546949c98-csmxk" event={"ID":"a4255849-3d3c-4e26-b766-6d1474682843","Type":"ContainerDied","Data":"9fe20a5ef60edab0343d45df28b18c493dd3f40cd90d9f3b0ccc7f57ad1d002e"}
Sep 10 03:11:34.340349 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:34.340298    2558 scope.go:117] "RemoveContainer" containerID="5d8296c90ea9f4938e466d1b282417091299b9e72ea088066757fc3a30725875"
Sep 10 03:11:34.342036 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:34.342014    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0a342-cfbfbb758-2m28m" event={"ID":"3c73fe4f-e350-418f-bdc9-8f2f83825c9c","Type":"ContainerStarted","Data":"ffaf2ef28c0bdf43baff47c580f6a506c94f2aaeefd304972fb5edb2faa86dd4"}
Sep 10 03:11:34.342160 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:34.342053    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0a342-cfbfbb758-2m28m" event={"ID":"3c73fe4f-e350-418f-bdc9-8f2f83825c9c","Type":"ContainerStarted","Data":"a315bee18d16db8919880836e9e6f30bc914596e94ee5679dbdfdf873f64126a"}
Sep 10 03:11:34.342225 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:34.342171    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0a342-cfbfbb758-2m28m"
Sep 10 03:11:34.343615 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:34.343593    2558 generic.go:356] "Generic (PLEG): container finished" podID="a1792e8c-6841-4193-9ca0-53ecbe580290" containerID="3b9ab0504c679f5b32edb381d00600aed5c563bbbba9b14279710ba4e49e8997" exitCode=0
Sep 10 03:11:34.343705 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:34.343631    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-097e16-65ddd76cdc-nrwtx"
Sep 10 03:11:34.343705 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:34.343632    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-097e16-65ddd76cdc-nrwtx" event={"ID":"a1792e8c-6841-4193-9ca0-53ecbe580290","Type":"ContainerDied","Data":"3b9ab0504c679f5b32edb381d00600aed5c563bbbba9b14279710ba4e49e8997"}
Sep 10 03:11:34.343786 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:34.343738    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-097e16-65ddd76cdc-nrwtx" event={"ID":"a1792e8c-6841-4193-9ca0-53ecbe580290","Type":"ContainerDied","Data":"f54fd4f4ae90757abd1e4a009b70707a38ed07e5ed5cfd5e388155e4661df592"}
Sep 10 03:11:34.349983 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:34.349963    2558 scope.go:117] "RemoveContainer" containerID="5d8296c90ea9f4938e466d1b282417091299b9e72ea088066757fc3a30725875"
Sep 10 03:11:34.350262 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:11:34.350244    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d8296c90ea9f4938e466d1b282417091299b9e72ea088066757fc3a30725875\": container with ID starting with 5d8296c90ea9f4938e466d1b282417091299b9e72ea088066757fc3a30725875 not found: ID does not exist" containerID="5d8296c90ea9f4938e466d1b282417091299b9e72ea088066757fc3a30725875"
Sep 10 03:11:34.350326 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:34.350270    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d8296c90ea9f4938e466d1b282417091299b9e72ea088066757fc3a30725875"} err="failed to get container status \"5d8296c90ea9f4938e466d1b282417091299b9e72ea088066757fc3a30725875\": rpc error: code = NotFound desc = could not find container \"5d8296c90ea9f4938e466d1b282417091299b9e72ea088066757fc3a30725875\": container with ID starting with 5d8296c90ea9f4938e466d1b282417091299b9e72ea088066757fc3a30725875 not found: ID does not exist"
Sep 10 03:11:34.350326 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:34.350287    2558 scope.go:117] "RemoveContainer" containerID="3b9ab0504c679f5b32edb381d00600aed5c563bbbba9b14279710ba4e49e8997"
Sep 10 03:11:34.359147 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:34.359129    2558 scope.go:117] "RemoveContainer" containerID="3b9ab0504c679f5b32edb381d00600aed5c563bbbba9b14279710ba4e49e8997"
Sep 10 03:11:34.359441 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:11:34.359420    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b9ab0504c679f5b32edb381d00600aed5c563bbbba9b14279710ba4e49e8997\": container with ID starting with 3b9ab0504c679f5b32edb381d00600aed5c563bbbba9b14279710ba4e49e8997 not found: ID does not exist" containerID="3b9ab0504c679f5b32edb381d00600aed5c563bbbba9b14279710ba4e49e8997"
Sep 10 03:11:34.359496 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:34.359457    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b9ab0504c679f5b32edb381d00600aed5c563bbbba9b14279710ba4e49e8997"} err="failed to get container status \"3b9ab0504c679f5b32edb381d00600aed5c563bbbba9b14279710ba4e49e8997\": rpc error: code = NotFound desc = could not find container \"3b9ab0504c679f5b32edb381d00600aed5c563bbbba9b14279710ba4e49e8997\": container with ID starting with 3b9ab0504c679f5b32edb381d00600aed5c563bbbba9b14279710ba4e49e8997 not found: ID does not exist"
Sep 10 03:11:34.408735 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:34.408699    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f0a342-79c5677d67-bjtwm" podStartSLOduration=1.40868728 podStartE2EDuration="1.40868728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:11:34.379098858 +0000 UTC m=+1484.754184779" watchObservedRunningTime="2026-09-10 03:11:34.40868728 +0000 UTC m=+1484.783773201"
Sep 10 03:11:34.409462 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:34.409435    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0a342-cfbfbb758-2m28m" podStartSLOduration=1.409428488 podStartE2EDuration="1.409428488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:11:34.408560289 +0000 UTC m=+1484.783646211" watchObservedRunningTime="2026-09-10 03:11:34.409428488 +0000 UTC m=+1484.784514418"
Sep 10 03:11:34.452261 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:34.452238    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-097e16-65ddd76cdc-nrwtx"]
Sep 10 03:11:34.461291 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:34.461270    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-097e16-65ddd76cdc-nrwtx"]
Sep 10 03:11:34.509653 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:34.509632    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-097e16-546949c98-csmxk"]
Sep 10 03:11:34.525136 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:34.525112    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-097e16-546949c98-csmxk"]
Sep 10 03:11:36.071183 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:36.071146    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1792e8c-6841-4193-9ca0-53ecbe580290" path="/var/lib/kubelet/pods/a1792e8c-6841-4193-9ca0-53ecbe580290/volumes"
Sep 10 03:11:36.071629 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:36.071614    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4255849-3d3c-4e26-b766-6d1474682843" path="/var/lib/kubelet/pods/a4255849-3d3c-4e26-b766-6d1474682843/volumes"
Sep 10 03:11:36.756290 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:36.756247    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9" podUID="1d3f4716-eaa6-450e-aa7f-5f88437939a0" containerName="main" containerID="cri-o://540e9e09f0089dc87ef2096e9c8608ee98f58dac06c682bc0bd3501d06ff3b81" gracePeriod=45
Sep 10 03:11:36.768539 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:36.768496    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m" podUID="fc8b91c3-7d8d-4379-87e6-4d02d59401b8" containerName="main" containerID="cri-o://b92115c1f6a983cf015fc62df89762c110982f5ca4cf796bad0ffaffa6e75f37" gracePeriod=45
Sep 10 03:11:36.784943 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:36.784914    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c" podUID="34443357-d4bd-4934-9910-dc5ebaf64125" containerName="main" containerID="cri-o://9dff8e338f3f50d5797581f3557a571f50f022eeed20bcea4fe88b8350775957" gracePeriod=45
Sep 10 03:11:37.040507 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.040479    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9"
Sep 10 03:11:37.094400 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.094376    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m"
Sep 10 03:11:37.098582 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.098560    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c"
Sep 10 03:11:37.119083 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.118657    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d3f4716-eaa6-450e-aa7f-5f88437939a0-tls-certs\") pod \"1d3f4716-eaa6-450e-aa7f-5f88437939a0\" (UID: \"1d3f4716-eaa6-450e-aa7f-5f88437939a0\") "
Sep 10 03:11:37.119083 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.118718    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4716-eaa6-450e-aa7f-5f88437939a0-dshm\") pod \"1d3f4716-eaa6-450e-aa7f-5f88437939a0\" (UID: \"1d3f4716-eaa6-450e-aa7f-5f88437939a0\") "
Sep 10 03:11:37.119083 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.118805    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2zzw\" (UniqueName: \"kubernetes.io/projected/1d3f4716-eaa6-450e-aa7f-5f88437939a0-kube-api-access-x2zzw\") pod \"1d3f4716-eaa6-450e-aa7f-5f88437939a0\" (UID: \"1d3f4716-eaa6-450e-aa7f-5f88437939a0\") "
Sep 10 03:11:37.119083 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.118864    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4716-eaa6-450e-aa7f-5f88437939a0-home\") pod \"1d3f4716-eaa6-450e-aa7f-5f88437939a0\" (UID: \"1d3f4716-eaa6-450e-aa7f-5f88437939a0\") "
Sep 10 03:11:37.119083 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.118901    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4716-eaa6-450e-aa7f-5f88437939a0-tmp-dir\") pod \"1d3f4716-eaa6-450e-aa7f-5f88437939a0\" (UID: \"1d3f4716-eaa6-450e-aa7f-5f88437939a0\") "
Sep 10 03:11:37.119083 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.118952    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4716-eaa6-450e-aa7f-5f88437939a0-model-cache\") pod \"1d3f4716-eaa6-450e-aa7f-5f88437939a0\" (UID: \"1d3f4716-eaa6-450e-aa7f-5f88437939a0\") "
Sep 10 03:11:37.119610 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.119381    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d3f4716-eaa6-450e-aa7f-5f88437939a0-home" (OuterVolumeSpecName: "home") pod "1d3f4716-eaa6-450e-aa7f-5f88437939a0" (UID: "1d3f4716-eaa6-450e-aa7f-5f88437939a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:11:37.119610 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.119472    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d3f4716-eaa6-450e-aa7f-5f88437939a0-model-cache" (OuterVolumeSpecName: "model-cache") pod "1d3f4716-eaa6-450e-aa7f-5f88437939a0" (UID: "1d3f4716-eaa6-450e-aa7f-5f88437939a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:11:37.119776 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.119680    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d3f4716-eaa6-450e-aa7f-5f88437939a0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1d3f4716-eaa6-450e-aa7f-5f88437939a0" (UID: "1d3f4716-eaa6-450e-aa7f-5f88437939a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:11:37.121742 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.121714    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d3f4716-eaa6-450e-aa7f-5f88437939a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1d3f4716-eaa6-450e-aa7f-5f88437939a0" (UID: "1d3f4716-eaa6-450e-aa7f-5f88437939a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:11:37.121742 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.121725    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d3f4716-eaa6-450e-aa7f-5f88437939a0-dshm" (OuterVolumeSpecName: "dshm") pod "1d3f4716-eaa6-450e-aa7f-5f88437939a0" (UID: "1d3f4716-eaa6-450e-aa7f-5f88437939a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:11:37.121873 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.121858    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d3f4716-eaa6-450e-aa7f-5f88437939a0-kube-api-access-x2zzw" (OuterVolumeSpecName: "kube-api-access-x2zzw") pod "1d3f4716-eaa6-450e-aa7f-5f88437939a0" (UID: "1d3f4716-eaa6-450e-aa7f-5f88437939a0"). InnerVolumeSpecName "kube-api-access-x2zzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:11:37.220321 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.220286    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-tmp-dir\") pod \"fc8b91c3-7d8d-4379-87e6-4d02d59401b8\" (UID: \"fc8b91c3-7d8d-4379-87e6-4d02d59401b8\") "
Sep 10 03:11:37.220491 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.220352    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34443357-d4bd-4934-9910-dc5ebaf64125-tmp-dir\") pod \"34443357-d4bd-4934-9910-dc5ebaf64125\" (UID: \"34443357-d4bd-4934-9910-dc5ebaf64125\") "
Sep 10 03:11:37.220491 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.220387    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bndlk\" (UniqueName: \"kubernetes.io/projected/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-kube-api-access-bndlk\") pod \"fc8b91c3-7d8d-4379-87e6-4d02d59401b8\" (UID: \"fc8b91c3-7d8d-4379-87e6-4d02d59401b8\") "
Sep 10 03:11:37.220491 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.220447    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34443357-d4bd-4934-9910-dc5ebaf64125-model-cache\") pod \"34443357-d4bd-4934-9910-dc5ebaf64125\" (UID: \"34443357-d4bd-4934-9910-dc5ebaf64125\") "
Sep 10 03:11:37.220491 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.220470    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-model-cache\") pod \"fc8b91c3-7d8d-4379-87e6-4d02d59401b8\" (UID: \"fc8b91c3-7d8d-4379-87e6-4d02d59401b8\") "
Sep 10 03:11:37.220709 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.220571    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34443357-d4bd-4934-9910-dc5ebaf64125-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "34443357-d4bd-4934-9910-dc5ebaf64125" (UID: "34443357-d4bd-4934-9910-dc5ebaf64125"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:11:37.220709 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.220575    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fc8b91c3-7d8d-4379-87e6-4d02d59401b8" (UID: "fc8b91c3-7d8d-4379-87e6-4d02d59401b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:11:37.220709 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.220584    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8fcr\" (UniqueName: \"kubernetes.io/projected/34443357-d4bd-4934-9910-dc5ebaf64125-kube-api-access-s8fcr\") pod \"34443357-d4bd-4934-9910-dc5ebaf64125\" (UID: \"34443357-d4bd-4934-9910-dc5ebaf64125\") "
Sep 10 03:11:37.220861 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.220708    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-model-cache" (OuterVolumeSpecName: "model-cache") pod "fc8b91c3-7d8d-4379-87e6-4d02d59401b8" (UID: "fc8b91c3-7d8d-4379-87e6-4d02d59401b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:11:37.220861 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.220716    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34443357-d4bd-4934-9910-dc5ebaf64125-model-cache" (OuterVolumeSpecName: "model-cache") pod "34443357-d4bd-4934-9910-dc5ebaf64125" (UID: "34443357-d4bd-4934-9910-dc5ebaf64125"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:11:37.220861 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.220719    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34443357-d4bd-4934-9910-dc5ebaf64125-dshm\") pod \"34443357-d4bd-4934-9910-dc5ebaf64125\" (UID: \"34443357-d4bd-4934-9910-dc5ebaf64125\") "
Sep 10 03:11:37.220861 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.220779    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-dshm\") pod \"fc8b91c3-7d8d-4379-87e6-4d02d59401b8\" (UID: \"fc8b91c3-7d8d-4379-87e6-4d02d59401b8\") "
Sep 10 03:11:37.220861 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.220799    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-home\") pod \"fc8b91c3-7d8d-4379-87e6-4d02d59401b8\" (UID: \"fc8b91c3-7d8d-4379-87e6-4d02d59401b8\") "
Sep 10 03:11:37.220861 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.220826    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34443357-d4bd-4934-9910-dc5ebaf64125-home\") pod \"34443357-d4bd-4934-9910-dc5ebaf64125\" (UID: \"34443357-d4bd-4934-9910-dc5ebaf64125\") "
Sep 10 03:11:37.220861 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.220850    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34443357-d4bd-4934-9910-dc5ebaf64125-tls-certs\") pod \"34443357-d4bd-4934-9910-dc5ebaf64125\" (UID: \"34443357-d4bd-4934-9910-dc5ebaf64125\") "
Sep 10 03:11:37.221266 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.220884    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-tls-certs\") pod \"fc8b91c3-7d8d-4379-87e6-4d02d59401b8\" (UID: \"fc8b91c3-7d8d-4379-87e6-4d02d59401b8\") "
Sep 10 03:11:37.221266 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.221036    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-home" (OuterVolumeSpecName: "home") pod "fc8b91c3-7d8d-4379-87e6-4d02d59401b8" (UID: "fc8b91c3-7d8d-4379-87e6-4d02d59401b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:11:37.221375 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.221279    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4716-eaa6-450e-aa7f-5f88437939a0-model-cache\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.221375 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.221300    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d3f4716-eaa6-450e-aa7f-5f88437939a0-tls-certs\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.221375 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.221314    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34443357-d4bd-4934-9910-dc5ebaf64125-model-cache\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.221375 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.221326    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-model-cache\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.221375 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.221337    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4716-eaa6-450e-aa7f-5f88437939a0-dshm\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.221375 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.221349    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-home\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.221375 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.221362    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2zzw\" (UniqueName: \"kubernetes.io/projected/1d3f4716-eaa6-450e-aa7f-5f88437939a0-kube-api-access-x2zzw\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.221735 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.221379    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-tmp-dir\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.221735 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.221411    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4716-eaa6-450e-aa7f-5f88437939a0-home\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.221735 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.221423    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34443357-d4bd-4934-9910-dc5ebaf64125-tmp-dir\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.221735 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.221435    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4716-eaa6-450e-aa7f-5f88437939a0-tmp-dir\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.221735 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.221539    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34443357-d4bd-4934-9910-dc5ebaf64125-home" (OuterVolumeSpecName: "home") pod "34443357-d4bd-4934-9910-dc5ebaf64125" (UID: "34443357-d4bd-4934-9910-dc5ebaf64125"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:11:37.222828 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.222783    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-dshm" (OuterVolumeSpecName: "dshm") pod "fc8b91c3-7d8d-4379-87e6-4d02d59401b8" (UID: "fc8b91c3-7d8d-4379-87e6-4d02d59401b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:11:37.223055 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.223033    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34443357-d4bd-4934-9910-dc5ebaf64125-dshm" (OuterVolumeSpecName: "dshm") pod "34443357-d4bd-4934-9910-dc5ebaf64125" (UID: "34443357-d4bd-4934-9910-dc5ebaf64125"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:11:37.223055 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.223042    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-kube-api-access-bndlk" (OuterVolumeSpecName: "kube-api-access-bndlk") pod "fc8b91c3-7d8d-4379-87e6-4d02d59401b8" (UID: "fc8b91c3-7d8d-4379-87e6-4d02d59401b8"). InnerVolumeSpecName "kube-api-access-bndlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:11:37.223194 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.223058    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34443357-d4bd-4934-9910-dc5ebaf64125-kube-api-access-s8fcr" (OuterVolumeSpecName: "kube-api-access-s8fcr") pod "34443357-d4bd-4934-9910-dc5ebaf64125" (UID: "34443357-d4bd-4934-9910-dc5ebaf64125"). InnerVolumeSpecName "kube-api-access-s8fcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:11:37.223358 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.223330    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34443357-d4bd-4934-9910-dc5ebaf64125-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "34443357-d4bd-4934-9910-dc5ebaf64125" (UID: "34443357-d4bd-4934-9910-dc5ebaf64125"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:11:37.223569 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.223548    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fc8b91c3-7d8d-4379-87e6-4d02d59401b8" (UID: "fc8b91c3-7d8d-4379-87e6-4d02d59401b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:11:37.322971 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.322908    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-tls-certs\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.322971 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.322939    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bndlk\" (UniqueName: \"kubernetes.io/projected/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-kube-api-access-bndlk\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.322971 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.322956    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8fcr\" (UniqueName: \"kubernetes.io/projected/34443357-d4bd-4934-9910-dc5ebaf64125-kube-api-access-s8fcr\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.323173 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.322971    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34443357-d4bd-4934-9910-dc5ebaf64125-dshm\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.323173 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.322985    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc8b91c3-7d8d-4379-87e6-4d02d59401b8-dshm\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.323173 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.322998    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34443357-d4bd-4934-9910-dc5ebaf64125-home\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.323173 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.323012    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34443357-d4bd-4934-9910-dc5ebaf64125-tls-certs\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:11:37.365544 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.365522    2558 generic.go:356] "Generic (PLEG): container finished" podID="fc8b91c3-7d8d-4379-87e6-4d02d59401b8" containerID="b92115c1f6a983cf015fc62df89762c110982f5ca4cf796bad0ffaffa6e75f37" exitCode=0
Sep 10 03:11:37.365646 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.365566    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m"
Sep 10 03:11:37.365646 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.365607    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m" event={"ID":"fc8b91c3-7d8d-4379-87e6-4d02d59401b8","Type":"ContainerDied","Data":"b92115c1f6a983cf015fc62df89762c110982f5ca4cf796bad0ffaffa6e75f37"}
Sep 10 03:11:37.365741 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.365648    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m" event={"ID":"fc8b91c3-7d8d-4379-87e6-4d02d59401b8","Type":"ContainerDied","Data":"7033dd931ca0ae507ffaf7d857e2528678bedfe98a8aecb85d20c9557eac2cdb"}
Sep 10 03:11:37.365741 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.365670    2558 scope.go:117] "RemoveContainer" containerID="b92115c1f6a983cf015fc62df89762c110982f5ca4cf796bad0ffaffa6e75f37"
Sep 10 03:11:37.367207 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.367185    2558 generic.go:356] "Generic (PLEG): container finished" podID="1d3f4716-eaa6-450e-aa7f-5f88437939a0" containerID="540e9e09f0089dc87ef2096e9c8608ee98f58dac06c682bc0bd3501d06ff3b81" exitCode=0
Sep 10 03:11:37.367961 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.367381    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9"
Sep 10 03:11:37.367961 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.367391    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9" event={"ID":"1d3f4716-eaa6-450e-aa7f-5f88437939a0","Type":"ContainerDied","Data":"540e9e09f0089dc87ef2096e9c8608ee98f58dac06c682bc0bd3501d06ff3b81"}
Sep 10 03:11:37.367961 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.367869    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9" event={"ID":"1d3f4716-eaa6-450e-aa7f-5f88437939a0","Type":"ContainerDied","Data":"b59e72e9d0705d082588870296a12eeb4c27480678c0d80339252193c4d8183c"}
Sep 10 03:11:37.369151 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.369126    2558 generic.go:356] "Generic (PLEG): container finished" podID="34443357-d4bd-4934-9910-dc5ebaf64125" containerID="9dff8e338f3f50d5797581f3557a571f50f022eeed20bcea4fe88b8350775957" exitCode=0
Sep 10 03:11:37.369272 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.369156    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c"
Sep 10 03:11:37.369272 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.369163    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c" event={"ID":"34443357-d4bd-4934-9910-dc5ebaf64125","Type":"ContainerDied","Data":"9dff8e338f3f50d5797581f3557a571f50f022eeed20bcea4fe88b8350775957"}
Sep 10 03:11:37.369272 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.369193    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c" event={"ID":"34443357-d4bd-4934-9910-dc5ebaf64125","Type":"ContainerDied","Data":"54d6509a6493f95a13b3a7034164068773871bf46188c32dbf158c9dfc4ce5b2"}
Sep 10 03:11:37.377354 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.377339    2558 scope.go:117] "RemoveContainer" containerID="b92115c1f6a983cf015fc62df89762c110982f5ca4cf796bad0ffaffa6e75f37"
Sep 10 03:11:37.377631 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:11:37.377612    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b92115c1f6a983cf015fc62df89762c110982f5ca4cf796bad0ffaffa6e75f37\": container with ID starting with b92115c1f6a983cf015fc62df89762c110982f5ca4cf796bad0ffaffa6e75f37 not found: ID does not exist" containerID="b92115c1f6a983cf015fc62df89762c110982f5ca4cf796bad0ffaffa6e75f37"
Sep 10 03:11:37.377698 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.377638    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b92115c1f6a983cf015fc62df89762c110982f5ca4cf796bad0ffaffa6e75f37"} err="failed to get container status \"b92115c1f6a983cf015fc62df89762c110982f5ca4cf796bad0ffaffa6e75f37\": rpc error: code = NotFound desc = could not find container \"b92115c1f6a983cf015fc62df89762c110982f5ca4cf796bad0ffaffa6e75f37\": container with ID starting with b92115c1f6a983cf015fc62df89762c110982f5ca4cf796bad0ffaffa6e75f37 not found: ID does not exist"
Sep 10 03:11:37.377698 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.377652    2558 scope.go:117] "RemoveContainer" containerID="540e9e09f0089dc87ef2096e9c8608ee98f58dac06c682bc0bd3501d06ff3b81"
Sep 10 03:11:37.386374 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.386353    2558 scope.go:117] "RemoveContainer" containerID="540e9e09f0089dc87ef2096e9c8608ee98f58dac06c682bc0bd3501d06ff3b81"
Sep 10 03:11:37.386616 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:11:37.386601    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"540e9e09f0089dc87ef2096e9c8608ee98f58dac06c682bc0bd3501d06ff3b81\": container with ID starting with 540e9e09f0089dc87ef2096e9c8608ee98f58dac06c682bc0bd3501d06ff3b81 not found: ID does not exist" containerID="540e9e09f0089dc87ef2096e9c8608ee98f58dac06c682bc0bd3501d06ff3b81"
Sep 10 03:11:37.386672 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.386622    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"540e9e09f0089dc87ef2096e9c8608ee98f58dac06c682bc0bd3501d06ff3b81"} err="failed to get container status \"540e9e09f0089dc87ef2096e9c8608ee98f58dac06c682bc0bd3501d06ff3b81\": rpc error: code = NotFound desc = could not find container \"540e9e09f0089dc87ef2096e9c8608ee98f58dac06c682bc0bd3501d06ff3b81\": container with ID starting with 540e9e09f0089dc87ef2096e9c8608ee98f58dac06c682bc0bd3501d06ff3b81 not found: ID does not exist"
Sep 10 03:11:37.386672 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.386638    2558 scope.go:117] "RemoveContainer" containerID="9dff8e338f3f50d5797581f3557a571f50f022eeed20bcea4fe88b8350775957"
Sep 10 03:11:37.393434 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.393415    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m"]
Sep 10 03:11:37.396149 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.396130    2558 scope.go:117] "RemoveContainer" containerID="9dff8e338f3f50d5797581f3557a571f50f022eeed20bcea4fe88b8350775957"
Sep 10 03:11:37.396450 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:11:37.396433    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dff8e338f3f50d5797581f3557a571f50f022eeed20bcea4fe88b8350775957\": container with ID starting with 9dff8e338f3f50d5797581f3557a571f50f022eeed20bcea4fe88b8350775957 not found: ID does not exist" containerID="9dff8e338f3f50d5797581f3557a571f50f022eeed20bcea4fe88b8350775957"
Sep 10 03:11:37.396513 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.396455    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dff8e338f3f50d5797581f3557a571f50f022eeed20bcea4fe88b8350775957"} err="failed to get container status \"9dff8e338f3f50d5797581f3557a571f50f022eeed20bcea4fe88b8350775957\": rpc error: code = NotFound desc = could not find container \"9dff8e338f3f50d5797581f3557a571f50f022eeed20bcea4fe88b8350775957\": container with ID starting with 9dff8e338f3f50d5797581f3557a571f50f022eeed20bcea4fe88b8350775957 not found: ID does not exist"
Sep 10 03:11:37.397036 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.397019    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-097e16/premium-w0-097e16-kserve-747bf48f7b-rv44m"]
Sep 10 03:11:37.407379 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.407357    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9"]
Sep 10 03:11:37.413819 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.413797    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-097e16/embedding-w0-097e16-kserve-56b66d4dcb-65qf9"]
Sep 10 03:11:37.423375 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.423348    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c"]
Sep 10 03:11:37.427957 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:37.427933    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-097e16/unconfigured-w0-097e16-kserve-6fd84696-v2v9c"]
Sep 10 03:11:38.069433 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:38.069356    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d3f4716-eaa6-450e-aa7f-5f88437939a0" path="/var/lib/kubelet/pods/1d3f4716-eaa6-450e-aa7f-5f88437939a0/volumes"
Sep 10 03:11:38.069736 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:38.069721    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34443357-d4bd-4934-9910-dc5ebaf64125" path="/var/lib/kubelet/pods/34443357-d4bd-4934-9910-dc5ebaf64125/volumes"
Sep 10 03:11:38.070065 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:38.070052    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc8b91c3-7d8d-4379-87e6-4d02d59401b8" path="/var/lib/kubelet/pods/fc8b91c3-7d8d-4379-87e6-4d02d59401b8/volumes"
Sep 10 03:11:39.821550 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:39.821510    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:39.922940 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:39.922908    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:40.119231 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:40.119150    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:44.281869 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:44.281837    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:45.353714 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:45.353686    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0a342-cfbfbb758-2m28m"
Sep 10 03:11:45.354159 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:45.353808    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f0a342-79c5677d67-bjtwm"
Sep 10 03:11:56.276918 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:56.276886    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:56.710747 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:56.710719    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:57.050129 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:57.050023    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:57.363838 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:57.363749    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:59.120930 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:59.120898    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:59.222328 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:59.222299    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:11:59.632226 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:11:59.632199    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:12:00.435353 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:00.435319    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:12:00.527969 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:00.527934    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:12:01.714035 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.714004    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f23388-c9dc64984-gqgjr"]
Sep 10 03:12:01.714607 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.714585    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1792e8c-6841-4193-9ca0-53ecbe580290" containerName="payload-processing"
Sep 10 03:12:01.714698 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.714609    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1792e8c-6841-4193-9ca0-53ecbe580290" containerName="payload-processing"
Sep 10 03:12:01.714698 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.714639    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34443357-d4bd-4934-9910-dc5ebaf64125" containerName="main"
Sep 10 03:12:01.714698 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.714648    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="34443357-d4bd-4934-9910-dc5ebaf64125" containerName="main"
Sep 10 03:12:01.714698 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.714659    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4255849-3d3c-4e26-b766-6d1474682843" containerName="payload-pre-processing"
Sep 10 03:12:01.714698 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.714668    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4255849-3d3c-4e26-b766-6d1474682843" containerName="payload-pre-processing"
Sep 10 03:12:01.714698 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.714679    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d3f4716-eaa6-450e-aa7f-5f88437939a0" containerName="main"
Sep 10 03:12:01.714698 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.714687    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d3f4716-eaa6-450e-aa7f-5f88437939a0" containerName="main"
Sep 10 03:12:01.714698 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.714699    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc8b91c3-7d8d-4379-87e6-4d02d59401b8" containerName="main"
Sep 10 03:12:01.715095 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.714707    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc8b91c3-7d8d-4379-87e6-4d02d59401b8" containerName="main"
Sep 10 03:12:01.715095 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.714812    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc8b91c3-7d8d-4379-87e6-4d02d59401b8" containerName="main"
Sep 10 03:12:01.715095 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.714823    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1792e8c-6841-4193-9ca0-53ecbe580290" containerName="payload-processing"
Sep 10 03:12:01.715095 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.714836    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d3f4716-eaa6-450e-aa7f-5f88437939a0" containerName="main"
Sep 10 03:12:01.715095 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.714846    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="34443357-d4bd-4934-9910-dc5ebaf64125" containerName="main"
Sep 10 03:12:01.715095 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.714855    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4255849-3d3c-4e26-b766-6d1474682843" containerName="payload-pre-processing"
Sep 10 03:12:01.718175 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.718153    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f23388-c9dc64984-gqgjr"
Sep 10 03:12:01.721076 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.721056    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fqv2s\""
Sep 10 03:12:01.721696 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.721680    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 03:12:01.721696 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.721690    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f23388-serving-cert\""
Sep 10 03:12:01.737472 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.737452    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f23388-c9dc64984-gqgjr"]
Sep 10 03:12:01.796866 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.796841    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:12:01.819196 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.819166    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hfvf\" (UniqueName: \"kubernetes.io/projected/737f6d01-3d1f-49a1-9b5e-e249399b639f-kube-api-access-7hfvf\") pod \"maas-api-e2e-api-b-f23388-c9dc64984-gqgjr\" (UID: \"737f6d01-3d1f-49a1-9b5e-e249399b639f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f23388-c9dc64984-gqgjr"
Sep 10 03:12:01.819295 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.819229    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/737f6d01-3d1f-49a1-9b5e-e249399b639f-maas-api-tls\") pod \"maas-api-e2e-api-b-f23388-c9dc64984-gqgjr\" (UID: \"737f6d01-3d1f-49a1-9b5e-e249399b639f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f23388-c9dc64984-gqgjr"
Sep 10 03:12:01.819295 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.819276    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/737f6d01-3d1f-49a1-9b5e-e249399b639f-metrics-certs\") pod \"maas-api-e2e-api-b-f23388-c9dc64984-gqgjr\" (UID: \"737f6d01-3d1f-49a1-9b5e-e249399b639f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f23388-c9dc64984-gqgjr"
Sep 10 03:12:01.819382 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.819320    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/737f6d01-3d1f-49a1-9b5e-e249399b639f-openshift-service-ca\") pod \"maas-api-e2e-api-b-f23388-c9dc64984-gqgjr\" (UID: \"737f6d01-3d1f-49a1-9b5e-e249399b639f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f23388-c9dc64984-gqgjr"
Sep 10 03:12:01.878486 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.878458    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:12:01.920519 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.920495    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/737f6d01-3d1f-49a1-9b5e-e249399b639f-maas-api-tls\") pod \"maas-api-e2e-api-b-f23388-c9dc64984-gqgjr\" (UID: \"737f6d01-3d1f-49a1-9b5e-e249399b639f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f23388-c9dc64984-gqgjr"
Sep 10 03:12:01.920587 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.920528    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/737f6d01-3d1f-49a1-9b5e-e249399b639f-metrics-certs\") pod \"maas-api-e2e-api-b-f23388-c9dc64984-gqgjr\" (UID: \"737f6d01-3d1f-49a1-9b5e-e249399b639f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f23388-c9dc64984-gqgjr"
Sep 10 03:12:01.920587 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.920556    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/737f6d01-3d1f-49a1-9b5e-e249399b639f-openshift-service-ca\") pod \"maas-api-e2e-api-b-f23388-c9dc64984-gqgjr\" (UID: \"737f6d01-3d1f-49a1-9b5e-e249399b639f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f23388-c9dc64984-gqgjr"
Sep 10 03:12:01.920658 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.920592    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hfvf\" (UniqueName: \"kubernetes.io/projected/737f6d01-3d1f-49a1-9b5e-e249399b639f-kube-api-access-7hfvf\") pod \"maas-api-e2e-api-b-f23388-c9dc64984-gqgjr\" (UID: \"737f6d01-3d1f-49a1-9b5e-e249399b639f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f23388-c9dc64984-gqgjr"
Sep 10 03:12:01.921301 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.921282    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/737f6d01-3d1f-49a1-9b5e-e249399b639f-openshift-service-ca\") pod \"maas-api-e2e-api-b-f23388-c9dc64984-gqgjr\" (UID: \"737f6d01-3d1f-49a1-9b5e-e249399b639f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f23388-c9dc64984-gqgjr"
Sep 10 03:12:01.922918 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.922902    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/737f6d01-3d1f-49a1-9b5e-e249399b639f-metrics-certs\") pod \"maas-api-e2e-api-b-f23388-c9dc64984-gqgjr\" (UID: \"737f6d01-3d1f-49a1-9b5e-e249399b639f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f23388-c9dc64984-gqgjr"
Sep 10 03:12:01.923067 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.923041    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/737f6d01-3d1f-49a1-9b5e-e249399b639f-maas-api-tls\") pod \"maas-api-e2e-api-b-f23388-c9dc64984-gqgjr\" (UID: \"737f6d01-3d1f-49a1-9b5e-e249399b639f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f23388-c9dc64984-gqgjr"
Sep 10 03:12:01.928368 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:01.928343    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hfvf\" (UniqueName: \"kubernetes.io/projected/737f6d01-3d1f-49a1-9b5e-e249399b639f-kube-api-access-7hfvf\") pod \"maas-api-e2e-api-b-f23388-c9dc64984-gqgjr\" (UID: \"737f6d01-3d1f-49a1-9b5e-e249399b639f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f23388-c9dc64984-gqgjr"
Sep 10 03:12:02.028678 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:02.028622    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f23388-c9dc64984-gqgjr"
Sep 10 03:12:02.309649 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:02.309618    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f23388-c9dc64984-gqgjr"]
Sep 10 03:12:02.311363 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:12:02.311329    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod737f6d01_3d1f_49a1_9b5e_e249399b639f.slice/crio-29c87bb3ea110ee25fe51ef13f10c373ca1b7a530cce4e163946d826f916ba18 WatchSource:0}: Error finding container 29c87bb3ea110ee25fe51ef13f10c373ca1b7a530cce4e163946d826f916ba18: Status 404 returned error can't find the container with id 29c87bb3ea110ee25fe51ef13f10c373ca1b7a530cce4e163946d826f916ba18
Sep 10 03:12:02.312743 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:02.312723    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 03:12:02.331603 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:02.331520    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:12:02.429949 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:02.429926    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:12:02.469935 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:02.469905    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f23388-6f684696d5-4phh8"]
Sep 10 03:12:02.472710 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:02.472695    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f23388-6f684696d5-4phh8"
Sep 10 03:12:02.475078 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:02.475045    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f23388\""
Sep 10 03:12:02.475215 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:02.475155    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f23388-dockercfg-9shqj\""
Sep 10 03:12:02.484115 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:02.484078    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f23388-6f684696d5-4phh8"]
Sep 10 03:12:02.525909 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:02.525825    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4ns8\" (UniqueName: \"kubernetes.io/projected/8aa9fa77-1b97-4ff0-83d5-ad0addcafc33-kube-api-access-d4ns8\") pod \"payload-processing-e2e-api-b-f23388-6f684696d5-4phh8\" (UID: \"8aa9fa77-1b97-4ff0-83d5-ad0addcafc33\") " pod="openshift-ingress/payload-processing-e2e-api-b-f23388-6f684696d5-4phh8"
Sep 10 03:12:02.526051 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:02.525941    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f23388-c9dc64984-gqgjr" event={"ID":"737f6d01-3d1f-49a1-9b5e-e249399b639f","Type":"ContainerStarted","Data":"29c87bb3ea110ee25fe51ef13f10c373ca1b7a530cce4e163946d826f916ba18"}
Sep 10 03:12:02.526051 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:02.525949    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aa9fa77-1b97-4ff0-83d5-ad0addcafc33-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f23388-6f684696d5-4phh8\" (UID: \"8aa9fa77-1b97-4ff0-83d5-ad0addcafc33\") " pod="openshift-ingress/payload-processing-e2e-api-b-f23388-6f684696d5-4phh8"
Sep 10 03:12:02.626891 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:02.626821    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aa9fa77-1b97-4ff0-83d5-ad0addcafc33-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f23388-6f684696d5-4phh8\" (UID: \"8aa9fa77-1b97-4ff0-83d5-ad0addcafc33\") " pod="openshift-ingress/payload-processing-e2e-api-b-f23388-6f684696d5-4phh8"
Sep 10 03:12:02.626891 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:02.626867    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4ns8\" (UniqueName: \"kubernetes.io/projected/8aa9fa77-1b97-4ff0-83d5-ad0addcafc33-kube-api-access-d4ns8\") pod \"payload-processing-e2e-api-b-f23388-6f684696d5-4phh8\" (UID: \"8aa9fa77-1b97-4ff0-83d5-ad0addcafc33\") " pod="openshift-ingress/payload-processing-e2e-api-b-f23388-6f684696d5-4phh8"
Sep 10 03:12:02.627530 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:02.627512    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aa9fa77-1b97-4ff0-83d5-ad0addcafc33-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f23388-6f684696d5-4phh8\" (UID: \"8aa9fa77-1b97-4ff0-83d5-ad0addcafc33\") " pod="openshift-ingress/payload-processing-e2e-api-b-f23388-6f684696d5-4phh8"
Sep 10 03:12:02.637015 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:02.636990    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4ns8\" (UniqueName: \"kubernetes.io/projected/8aa9fa77-1b97-4ff0-83d5-ad0addcafc33-kube-api-access-d4ns8\") pod \"payload-processing-e2e-api-b-f23388-6f684696d5-4phh8\" (UID: \"8aa9fa77-1b97-4ff0-83d5-ad0addcafc33\") " pod="openshift-ingress/payload-processing-e2e-api-b-f23388-6f684696d5-4phh8"
Sep 10 03:12:02.785017 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:02.784994    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f23388-6f684696d5-4phh8"
Sep 10 03:12:03.063037 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:03.063007    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f23388-6f684696d5-4phh8"]
Sep 10 03:12:03.065740 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:12:03.065698    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa9fa77_1b97_4ff0_83d5_ad0addcafc33.slice/crio-e589b85b642bbd0a3deacff8f59eb9df707f03ab10c288f1ef35cb2aea1d31bb WatchSource:0}: Error finding container e589b85b642bbd0a3deacff8f59eb9df707f03ab10c288f1ef35cb2aea1d31bb: Status 404 returned error can't find the container with id e589b85b642bbd0a3deacff8f59eb9df707f03ab10c288f1ef35cb2aea1d31bb
Sep 10 03:12:03.536210 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:03.536173    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f23388-6f684696d5-4phh8" event={"ID":"8aa9fa77-1b97-4ff0-83d5-ad0addcafc33","Type":"ContainerStarted","Data":"a6a933da7ef6bf465b3593df4b86a8d84ef33f828ca9d8c1d5af29d158e40a67"}
Sep 10 03:12:03.536388 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:03.536238    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f23388-6f684696d5-4phh8" event={"ID":"8aa9fa77-1b97-4ff0-83d5-ad0addcafc33","Type":"ContainerStarted","Data":"e589b85b642bbd0a3deacff8f59eb9df707f03ab10c288f1ef35cb2aea1d31bb"}
Sep 10 03:12:03.536388 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:03.536340    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f23388-6f684696d5-4phh8"
Sep 10 03:12:03.554446 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:03.554398    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f23388-6f684696d5-4phh8" podStartSLOduration=1.554384866 podStartE2EDuration="1.554384866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:12:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:12:03.551449043 +0000 UTC m=+1513.926534987" watchObservedRunningTime="2026-09-10 03:12:03.554384866 +0000 UTC m=+1513.929470788"
Sep 10 03:12:05.552516 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:05.552480    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f23388-c9dc64984-gqgjr" event={"ID":"737f6d01-3d1f-49a1-9b5e-e249399b639f","Type":"ContainerStarted","Data":"2aad509bfaf783df7b8da1a8c1fe52a7b30dc1be889ebda0b67858f54920ad26"}
Sep 10 03:12:05.552878 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:05.552624    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f23388-c9dc64984-gqgjr"
Sep 10 03:12:05.571521 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:05.571472    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f23388-c9dc64984-gqgjr" podStartSLOduration=2.127726842 podStartE2EDuration="4.571456452s" totalImagesPullingTime="2.44372961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:12:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:12:05.569389652 +0000 UTC m=+1515.944475590" watchObservedRunningTime="2026-09-10 03:12:05.571456452 +0000 UTC m=+1515.946542373"
Sep 10 03:12:06.001056 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:06.001031    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:12:11.566841 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:11.566808    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f23388-c9dc64984-gqgjr"
Sep 10 03:12:14.304573 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:14.304535    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:12:14.546151 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:14.546124    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f23388-6f684696d5-4phh8"
Sep 10 03:12:19.918692 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:19.918656    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:12:25.450902 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:25.450870    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:12:28.372216 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:28.372183    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:12:29.720910 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:29.720859    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:12:38.453504 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:38.453462    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:12:44.051078 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.051035    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:12:44.073734 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.073676    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f0a342-79c5677d67-bjtwm"]
Sep 10 03:12:44.074560 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.074527    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f0a342-79c5677d67-bjtwm" podUID="3f628aac-05cc-4933-8887-1524945a483a" containerName="payload-processing" containerID="cri-o://6422ac4918a26f97a8f34d633d0bb8974281da3df7cedf6c2ff241fa624d5b6f" gracePeriod=30
Sep 10 03:12:44.077942 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.077915    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f0a342-cfbfbb758-2m28m"]
Sep 10 03:12:44.078295 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.078268    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0a342-cfbfbb758-2m28m" podUID="3c73fe4f-e350-418f-bdc9-8f2f83825c9c" containerName="payload-pre-processing" containerID="cri-o://ffaf2ef28c0bdf43baff47c580f6a506c94f2aaeefd304972fb5edb2faa86dd4" gracePeriod=30
Sep 10 03:12:44.152413 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.152388    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:12:44.239730 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.239695    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:12:44.296331 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.296297    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:12:44.371547 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.371524    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0a342-cfbfbb758-2m28m"
Sep 10 03:12:44.375007 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.374986    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f0a342-79c5677d67-bjtwm"
Sep 10 03:12:44.466936 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.466908    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mmdj\" (UniqueName: \"kubernetes.io/projected/3c73fe4f-e350-418f-bdc9-8f2f83825c9c-kube-api-access-2mmdj\") pod \"3c73fe4f-e350-418f-bdc9-8f2f83825c9c\" (UID: \"3c73fe4f-e350-418f-bdc9-8f2f83825c9c\") "
Sep 10 03:12:44.467077 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.466949    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c73fe4f-e350-418f-bdc9-8f2f83825c9c-plugins-config-volume\") pod \"3c73fe4f-e350-418f-bdc9-8f2f83825c9c\" (UID: \"3c73fe4f-e350-418f-bdc9-8f2f83825c9c\") "
Sep 10 03:12:44.467077 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.466993    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w26z8\" (UniqueName: \"kubernetes.io/projected/3f628aac-05cc-4933-8887-1524945a483a-kube-api-access-w26z8\") pod \"3f628aac-05cc-4933-8887-1524945a483a\" (UID: \"3f628aac-05cc-4933-8887-1524945a483a\") "
Sep 10 03:12:44.467077 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.467029    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f628aac-05cc-4933-8887-1524945a483a-plugins-config-volume\") pod \"3f628aac-05cc-4933-8887-1524945a483a\" (UID: \"3f628aac-05cc-4933-8887-1524945a483a\") "
Sep 10 03:12:44.467443 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.467413    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c73fe4f-e350-418f-bdc9-8f2f83825c9c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3c73fe4f-e350-418f-bdc9-8f2f83825c9c" (UID: "3c73fe4f-e350-418f-bdc9-8f2f83825c9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:12:44.467545 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.467434    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f628aac-05cc-4933-8887-1524945a483a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3f628aac-05cc-4933-8887-1524945a483a" (UID: "3f628aac-05cc-4933-8887-1524945a483a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:12:44.468973 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.468953    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f628aac-05cc-4933-8887-1524945a483a-kube-api-access-w26z8" (OuterVolumeSpecName: "kube-api-access-w26z8") pod "3f628aac-05cc-4933-8887-1524945a483a" (UID: "3f628aac-05cc-4933-8887-1524945a483a"). InnerVolumeSpecName "kube-api-access-w26z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:12:44.469084 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.469065    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c73fe4f-e350-418f-bdc9-8f2f83825c9c-kube-api-access-2mmdj" (OuterVolumeSpecName: "kube-api-access-2mmdj") pod "3c73fe4f-e350-418f-bdc9-8f2f83825c9c" (UID: "3c73fe4f-e350-418f-bdc9-8f2f83825c9c"). InnerVolumeSpecName "kube-api-access-2mmdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:12:44.568122 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.568075    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mmdj\" (UniqueName: \"kubernetes.io/projected/3c73fe4f-e350-418f-bdc9-8f2f83825c9c-kube-api-access-2mmdj\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:12:44.568222 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.568128    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c73fe4f-e350-418f-bdc9-8f2f83825c9c-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:12:44.568222 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.568141    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w26z8\" (UniqueName: \"kubernetes.io/projected/3f628aac-05cc-4933-8887-1524945a483a-kube-api-access-w26z8\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:12:44.568222 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.568151    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f628aac-05cc-4933-8887-1524945a483a-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:12:44.607336 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.607265    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:12:44.792734 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.792710    2558 generic.go:356] "Generic (PLEG): container finished" podID="3f628aac-05cc-4933-8887-1524945a483a" containerID="6422ac4918a26f97a8f34d633d0bb8974281da3df7cedf6c2ff241fa624d5b6f" exitCode=0
Sep 10 03:12:44.792860 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.792755    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f0a342-79c5677d67-bjtwm"
Sep 10 03:12:44.792860 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.792805    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f0a342-79c5677d67-bjtwm" event={"ID":"3f628aac-05cc-4933-8887-1524945a483a","Type":"ContainerDied","Data":"6422ac4918a26f97a8f34d633d0bb8974281da3df7cedf6c2ff241fa624d5b6f"}
Sep 10 03:12:44.792985 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.792863    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f0a342-79c5677d67-bjtwm" event={"ID":"3f628aac-05cc-4933-8887-1524945a483a","Type":"ContainerDied","Data":"f26090c23284dcc2a27de10dd7e578ef3befaca405532ccbf4b0d425218d1f3c"}
Sep 10 03:12:44.792985 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.792892    2558 scope.go:117] "RemoveContainer" containerID="6422ac4918a26f97a8f34d633d0bb8974281da3df7cedf6c2ff241fa624d5b6f"
Sep 10 03:12:44.794597 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.794576    2558 generic.go:356] "Generic (PLEG): container finished" podID="3c73fe4f-e350-418f-bdc9-8f2f83825c9c" containerID="ffaf2ef28c0bdf43baff47c580f6a506c94f2aaeefd304972fb5edb2faa86dd4" exitCode=0
Sep 10 03:12:44.794680 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.794613    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0a342-cfbfbb758-2m28m" event={"ID":"3c73fe4f-e350-418f-bdc9-8f2f83825c9c","Type":"ContainerDied","Data":"ffaf2ef28c0bdf43baff47c580f6a506c94f2aaeefd304972fb5edb2faa86dd4"}
Sep 10 03:12:44.794680 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.794630    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0a342-cfbfbb758-2m28m"
Sep 10 03:12:44.794680 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.794642    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0a342-cfbfbb758-2m28m" event={"ID":"3c73fe4f-e350-418f-bdc9-8f2f83825c9c","Type":"ContainerDied","Data":"a315bee18d16db8919880836e9e6f30bc914596e94ee5679dbdfdf873f64126a"}
Sep 10 03:12:44.808958 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.808935    2558 scope.go:117] "RemoveContainer" containerID="6422ac4918a26f97a8f34d633d0bb8974281da3df7cedf6c2ff241fa624d5b6f"
Sep 10 03:12:44.809618 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:12:44.809521    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6422ac4918a26f97a8f34d633d0bb8974281da3df7cedf6c2ff241fa624d5b6f\": container with ID starting with 6422ac4918a26f97a8f34d633d0bb8974281da3df7cedf6c2ff241fa624d5b6f not found: ID does not exist" containerID="6422ac4918a26f97a8f34d633d0bb8974281da3df7cedf6c2ff241fa624d5b6f"
Sep 10 03:12:44.809703 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.809635    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6422ac4918a26f97a8f34d633d0bb8974281da3df7cedf6c2ff241fa624d5b6f"} err="failed to get container status \"6422ac4918a26f97a8f34d633d0bb8974281da3df7cedf6c2ff241fa624d5b6f\": rpc error: code = NotFound desc = could not find container \"6422ac4918a26f97a8f34d633d0bb8974281da3df7cedf6c2ff241fa624d5b6f\": container with ID starting with 6422ac4918a26f97a8f34d633d0bb8974281da3df7cedf6c2ff241fa624d5b6f not found: ID does not exist"
Sep 10 03:12:44.809703 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.809660    2558 scope.go:117] "RemoveContainer" containerID="ffaf2ef28c0bdf43baff47c580f6a506c94f2aaeefd304972fb5edb2faa86dd4"
Sep 10 03:12:44.817895 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.817876    2558 scope.go:117] "RemoveContainer" containerID="ffaf2ef28c0bdf43baff47c580f6a506c94f2aaeefd304972fb5edb2faa86dd4"
Sep 10 03:12:44.818153 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:12:44.818133    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffaf2ef28c0bdf43baff47c580f6a506c94f2aaeefd304972fb5edb2faa86dd4\": container with ID starting with ffaf2ef28c0bdf43baff47c580f6a506c94f2aaeefd304972fb5edb2faa86dd4 not found: ID does not exist" containerID="ffaf2ef28c0bdf43baff47c580f6a506c94f2aaeefd304972fb5edb2faa86dd4"
Sep 10 03:12:44.818223 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.818159    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffaf2ef28c0bdf43baff47c580f6a506c94f2aaeefd304972fb5edb2faa86dd4"} err="failed to get container status \"ffaf2ef28c0bdf43baff47c580f6a506c94f2aaeefd304972fb5edb2faa86dd4\": rpc error: code = NotFound desc = could not find container \"ffaf2ef28c0bdf43baff47c580f6a506c94f2aaeefd304972fb5edb2faa86dd4\": container with ID starting with ffaf2ef28c0bdf43baff47c580f6a506c94f2aaeefd304972fb5edb2faa86dd4 not found: ID does not exist"
Sep 10 03:12:44.823704 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.823683    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f0a342-79c5677d67-bjtwm"]
Sep 10 03:12:44.827572 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.827551    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f0a342-79c5677d67-bjtwm"]
Sep 10 03:12:44.838492 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.838472    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f0a342-cfbfbb758-2m28m"]
Sep 10 03:12:44.844354 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:44.844333    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f0a342-cfbfbb758-2m28m"]
Sep 10 03:12:46.069750 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:46.069718    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c73fe4f-e350-418f-bdc9-8f2f83825c9c" path="/var/lib/kubelet/pods/3c73fe4f-e350-418f-bdc9-8f2f83825c9c/volumes"
Sep 10 03:12:46.070165 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:46.070077    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f628aac-05cc-4933-8887-1524945a483a" path="/var/lib/kubelet/pods/3f628aac-05cc-4933-8887-1524945a483a/volumes"
Sep 10 03:12:50.361910 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:50.361879    2558 scope.go:117] "RemoveContainer" containerID="feba1fbb71bdf347af9808ca9fc2141b5f722a52e64e54c59017132b3ff626c5"
Sep 10 03:12:50.370832 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:50.370811    2558 scope.go:117] "RemoveContainer" containerID="ba11117d2678afc68ece4105d3a802428b45d5939c43d9bc7da4af77255ad101"
Sep 10 03:12:50.574023 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:50.573986    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:12:50.679009 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:50.678951    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:12:50.865195 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:50.865161    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:12:51.299312 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:51.299281    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:12:54.815154 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:54.815092    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw"]
Sep 10 03:12:54.815529 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:54.815520    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c73fe4f-e350-418f-bdc9-8f2f83825c9c" containerName="payload-pre-processing"
Sep 10 03:12:54.815585 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:54.815532    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c73fe4f-e350-418f-bdc9-8f2f83825c9c" containerName="payload-pre-processing"
Sep 10 03:12:54.815585 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:54.815554    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f628aac-05cc-4933-8887-1524945a483a" containerName="payload-processing"
Sep 10 03:12:54.815585 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:54.815560    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f628aac-05cc-4933-8887-1524945a483a" containerName="payload-processing"
Sep 10 03:12:54.815688 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:54.815627    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f628aac-05cc-4933-8887-1524945a483a" containerName="payload-processing"
Sep 10 03:12:54.815688 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:54.815635    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c73fe4f-e350-418f-bdc9-8f2f83825c9c" containerName="payload-pre-processing"
Sep 10 03:12:54.817732 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:54.817713    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw"
Sep 10 03:12:54.819930 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:54.819909    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e6e6b8\"/\"e2e-conflict-b-8ec4ab59-kserve-self-signed-certs\""
Sep 10 03:12:54.828208 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:54.828185    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw"]
Sep 10 03:12:54.839582 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:54.839559    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:12:54.950026 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:54.949998    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0776ba8d-636f-45d7-a537-e587b292b9d6-tls-certs\") pod \"e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw\" (UID: \"0776ba8d-636f-45d7-a537-e587b292b9d6\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw"
Sep 10 03:12:54.950181 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:54.950033    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0776ba8d-636f-45d7-a537-e587b292b9d6-tmp-dir\") pod \"e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw\" (UID: \"0776ba8d-636f-45d7-a537-e587b292b9d6\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw"
Sep 10 03:12:54.950181 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:54.950054    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0776ba8d-636f-45d7-a537-e587b292b9d6-home\") pod \"e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw\" (UID: \"0776ba8d-636f-45d7-a537-e587b292b9d6\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw"
Sep 10 03:12:54.950181 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:54.950133    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q244r\" (UniqueName: \"kubernetes.io/projected/0776ba8d-636f-45d7-a537-e587b292b9d6-kube-api-access-q244r\") pod \"e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw\" (UID: \"0776ba8d-636f-45d7-a537-e587b292b9d6\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw"
Sep 10 03:12:54.950381 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:54.950247    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0776ba8d-636f-45d7-a537-e587b292b9d6-dshm\") pod \"e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw\" (UID: \"0776ba8d-636f-45d7-a537-e587b292b9d6\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw"
Sep 10 03:12:54.950381 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:54.950311    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0776ba8d-636f-45d7-a537-e587b292b9d6-model-cache\") pod \"e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw\" (UID: \"0776ba8d-636f-45d7-a537-e587b292b9d6\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw"
Sep 10 03:12:55.051064 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:55.051035    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0776ba8d-636f-45d7-a537-e587b292b9d6-home\") pod \"e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw\" (UID: \"0776ba8d-636f-45d7-a537-e587b292b9d6\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw"
Sep 10 03:12:55.051177 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:55.051077    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q244r\" (UniqueName: \"kubernetes.io/projected/0776ba8d-636f-45d7-a537-e587b292b9d6-kube-api-access-q244r\") pod \"e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw\" (UID: \"0776ba8d-636f-45d7-a537-e587b292b9d6\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw"
Sep 10 03:12:55.051177 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:55.051134    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0776ba8d-636f-45d7-a537-e587b292b9d6-dshm\") pod \"e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw\" (UID: \"0776ba8d-636f-45d7-a537-e587b292b9d6\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw"
Sep 10 03:12:55.051177 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:55.051166    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0776ba8d-636f-45d7-a537-e587b292b9d6-model-cache\") pod \"e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw\" (UID: \"0776ba8d-636f-45d7-a537-e587b292b9d6\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw"
Sep 10 03:12:55.051352 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:55.051187    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0776ba8d-636f-45d7-a537-e587b292b9d6-tls-certs\") pod \"e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw\" (UID: \"0776ba8d-636f-45d7-a537-e587b292b9d6\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw"
Sep 10 03:12:55.051352 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:55.051207    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0776ba8d-636f-45d7-a537-e587b292b9d6-tmp-dir\") pod \"e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw\" (UID: \"0776ba8d-636f-45d7-a537-e587b292b9d6\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw"
Sep 10 03:12:55.051486 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:55.051461    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0776ba8d-636f-45d7-a537-e587b292b9d6-home\") pod \"e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw\" (UID: \"0776ba8d-636f-45d7-a537-e587b292b9d6\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw"
Sep 10 03:12:55.051556 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:55.051537    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0776ba8d-636f-45d7-a537-e587b292b9d6-tmp-dir\") pod \"e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw\" (UID: \"0776ba8d-636f-45d7-a537-e587b292b9d6\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw"
Sep 10 03:12:55.051595 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:55.051567    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0776ba8d-636f-45d7-a537-e587b292b9d6-model-cache\") pod \"e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw\" (UID: \"0776ba8d-636f-45d7-a537-e587b292b9d6\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw"
Sep 10 03:12:55.053449 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:55.053414    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0776ba8d-636f-45d7-a537-e587b292b9d6-dshm\") pod \"e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw\" (UID: \"0776ba8d-636f-45d7-a537-e587b292b9d6\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw"
Sep 10 03:12:55.053548 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:55.053530    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0776ba8d-636f-45d7-a537-e587b292b9d6-tls-certs\") pod \"e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw\" (UID: \"0776ba8d-636f-45d7-a537-e587b292b9d6\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw"
Sep 10 03:12:55.059516 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:55.059492    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q244r\" (UniqueName: \"kubernetes.io/projected/0776ba8d-636f-45d7-a537-e587b292b9d6-kube-api-access-q244r\") pod \"e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw\" (UID: \"0776ba8d-636f-45d7-a537-e587b292b9d6\") " pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw"
Sep 10 03:12:55.127496 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:55.127435    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw"
Sep 10 03:12:55.400867 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:55.400842    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw"]
Sep 10 03:12:55.401971 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:12:55.401947    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0776ba8d_636f_45d7_a537_e587b292b9d6.slice/crio-0d7375b4d83995f2c83e4306e19cb9c7d88c44260b4d2156c4f1f29b9db90fd1 WatchSource:0}: Error finding container 0d7375b4d83995f2c83e4306e19cb9c7d88c44260b4d2156c4f1f29b9db90fd1: Status 404 returned error can't find the container with id 0d7375b4d83995f2c83e4306e19cb9c7d88c44260b4d2156c4f1f29b9db90fd1
Sep 10 03:12:55.865500 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:55.865463    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw" event={"ID":"0776ba8d-636f-45d7-a537-e587b292b9d6","Type":"ContainerStarted","Data":"ae55e205a91d2ce068aaa93134334b0245d4b01d179b5958db244cae5222834b"}
Sep 10 03:12:55.865870 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:55.865516    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw" event={"ID":"0776ba8d-636f-45d7-a537-e587b292b9d6","Type":"ContainerStarted","Data":"0d7375b4d83995f2c83e4306e19cb9c7d88c44260b4d2156c4f1f29b9db90fd1"}
Sep 10 03:12:55.881032 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:55.880987    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw" podStartSLOduration=1.880967773 podStartE2EDuration="1.880967773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:12:55.8800749 +0000 UTC m=+1566.255160821" watchObservedRunningTime="2026-09-10 03:12:55.880967773 +0000 UTC m=+1566.256053696"
Sep 10 03:12:58.684940 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:12:58.684878    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:13:01.478956 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:01.478921    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:13:05.127715 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:05.127670    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw"
Sep 10 03:13:05.127715 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:05.127722    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw"
Sep 10 03:13:05.139892 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:05.139868    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw"
Sep 10 03:13:05.932701 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:05.932668    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw"
Sep 10 03:13:07.034492 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.034459    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:13:07.080672 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.080644    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f23388-c9dc64984-gqgjr"]
Sep 10 03:13:07.080906 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.080880    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f23388-c9dc64984-gqgjr" podUID="737f6d01-3d1f-49a1-9b5e-e249399b639f" containerName="maas-api" containerID="cri-o://2aad509bfaf783df7b8da1a8c1fe52a7b30dc1be889ebda0b67858f54920ad26" gracePeriod=30
Sep 10 03:13:07.142992 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.142965    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:13:07.149748 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.149723    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f23388-6f684696d5-4phh8"]
Sep 10 03:13:07.149931 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.149910    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f23388-6f684696d5-4phh8" podUID="8aa9fa77-1b97-4ff0-83d5-ad0addcafc33" containerName="payload-processing" containerID="cri-o://a6a933da7ef6bf465b3593df4b86a8d84ef33f828ca9d8c1d5af29d158e40a67" gracePeriod=30
Sep 10 03:13:07.237625 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.237597    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:13:07.356752 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.356680    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:13:07.402986 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.402964    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f23388-c9dc64984-gqgjr"
Sep 10 03:13:07.431316 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.431286    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f23388-6f684696d5-4phh8"
Sep 10 03:13:07.437192 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.437159    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:13:07.448289 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.448263    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hfvf\" (UniqueName: \"kubernetes.io/projected/737f6d01-3d1f-49a1-9b5e-e249399b639f-kube-api-access-7hfvf\") pod \"737f6d01-3d1f-49a1-9b5e-e249399b639f\" (UID: \"737f6d01-3d1f-49a1-9b5e-e249399b639f\") "
Sep 10 03:13:07.448380 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.448306    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/737f6d01-3d1f-49a1-9b5e-e249399b639f-maas-api-tls\") pod \"737f6d01-3d1f-49a1-9b5e-e249399b639f\" (UID: \"737f6d01-3d1f-49a1-9b5e-e249399b639f\") "
Sep 10 03:13:07.448380 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.448332    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/737f6d01-3d1f-49a1-9b5e-e249399b639f-openshift-service-ca\") pod \"737f6d01-3d1f-49a1-9b5e-e249399b639f\" (UID: \"737f6d01-3d1f-49a1-9b5e-e249399b639f\") "
Sep 10 03:13:07.448499 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.448397    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/737f6d01-3d1f-49a1-9b5e-e249399b639f-metrics-certs\") pod \"737f6d01-3d1f-49a1-9b5e-e249399b639f\" (UID: \"737f6d01-3d1f-49a1-9b5e-e249399b639f\") "
Sep 10 03:13:07.448499 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.448422    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4ns8\" (UniqueName: \"kubernetes.io/projected/8aa9fa77-1b97-4ff0-83d5-ad0addcafc33-kube-api-access-d4ns8\") pod \"8aa9fa77-1b97-4ff0-83d5-ad0addcafc33\" (UID: \"8aa9fa77-1b97-4ff0-83d5-ad0addcafc33\") "
Sep 10 03:13:07.448607 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.448500    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aa9fa77-1b97-4ff0-83d5-ad0addcafc33-plugins-config-volume\") pod \"8aa9fa77-1b97-4ff0-83d5-ad0addcafc33\" (UID: \"8aa9fa77-1b97-4ff0-83d5-ad0addcafc33\") "
Sep 10 03:13:07.448772 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.448729    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/737f6d01-3d1f-49a1-9b5e-e249399b639f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "737f6d01-3d1f-49a1-9b5e-e249399b639f" (UID: "737f6d01-3d1f-49a1-9b5e-e249399b639f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:13:07.449131 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.449081    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa9fa77-1b97-4ff0-83d5-ad0addcafc33-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8aa9fa77-1b97-4ff0-83d5-ad0addcafc33" (UID: "8aa9fa77-1b97-4ff0-83d5-ad0addcafc33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:13:07.451057 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.451032    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/737f6d01-3d1f-49a1-9b5e-e249399b639f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "737f6d01-3d1f-49a1-9b5e-e249399b639f" (UID: "737f6d01-3d1f-49a1-9b5e-e249399b639f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:13:07.451332 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.451298    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa9fa77-1b97-4ff0-83d5-ad0addcafc33-kube-api-access-d4ns8" (OuterVolumeSpecName: "kube-api-access-d4ns8") pod "8aa9fa77-1b97-4ff0-83d5-ad0addcafc33" (UID: "8aa9fa77-1b97-4ff0-83d5-ad0addcafc33"). InnerVolumeSpecName "kube-api-access-d4ns8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:13:07.451585 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.451561    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/737f6d01-3d1f-49a1-9b5e-e249399b639f-kube-api-access-7hfvf" (OuterVolumeSpecName: "kube-api-access-7hfvf") pod "737f6d01-3d1f-49a1-9b5e-e249399b639f" (UID: "737f6d01-3d1f-49a1-9b5e-e249399b639f"). InnerVolumeSpecName "kube-api-access-7hfvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:13:07.452253 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.452196    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/737f6d01-3d1f-49a1-9b5e-e249399b639f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "737f6d01-3d1f-49a1-9b5e-e249399b639f" (UID: "737f6d01-3d1f-49a1-9b5e-e249399b639f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:13:07.550131 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.550089    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/737f6d01-3d1f-49a1-9b5e-e249399b639f-metrics-certs\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:13:07.550131 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.550131    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4ns8\" (UniqueName: \"kubernetes.io/projected/8aa9fa77-1b97-4ff0-83d5-ad0addcafc33-kube-api-access-d4ns8\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:13:07.550274 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.550141    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aa9fa77-1b97-4ff0-83d5-ad0addcafc33-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:13:07.550274 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.550150    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hfvf\" (UniqueName: \"kubernetes.io/projected/737f6d01-3d1f-49a1-9b5e-e249399b639f-kube-api-access-7hfvf\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:13:07.550274 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.550160    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/737f6d01-3d1f-49a1-9b5e-e249399b639f-maas-api-tls\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:13:07.550274 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.550167    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/737f6d01-3d1f-49a1-9b5e-e249399b639f-openshift-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:13:07.732280 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.732248    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:13:07.934072 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.934044    2558 generic.go:356] "Generic (PLEG): container finished" podID="8aa9fa77-1b97-4ff0-83d5-ad0addcafc33" containerID="a6a933da7ef6bf465b3593df4b86a8d84ef33f828ca9d8c1d5af29d158e40a67" exitCode=0
Sep 10 03:13:07.934225 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.934087    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f23388-6f684696d5-4phh8"
Sep 10 03:13:07.934225 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.934133    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f23388-6f684696d5-4phh8" event={"ID":"8aa9fa77-1b97-4ff0-83d5-ad0addcafc33","Type":"ContainerDied","Data":"a6a933da7ef6bf465b3593df4b86a8d84ef33f828ca9d8c1d5af29d158e40a67"}
Sep 10 03:13:07.934225 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.934184    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f23388-6f684696d5-4phh8" event={"ID":"8aa9fa77-1b97-4ff0-83d5-ad0addcafc33","Type":"ContainerDied","Data":"e589b85b642bbd0a3deacff8f59eb9df707f03ab10c288f1ef35cb2aea1d31bb"}
Sep 10 03:13:07.934225 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.934200    2558 scope.go:117] "RemoveContainer" containerID="a6a933da7ef6bf465b3593df4b86a8d84ef33f828ca9d8c1d5af29d158e40a67"
Sep 10 03:13:07.935764 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.935738    2558 generic.go:356] "Generic (PLEG): container finished" podID="737f6d01-3d1f-49a1-9b5e-e249399b639f" containerID="2aad509bfaf783df7b8da1a8c1fe52a7b30dc1be889ebda0b67858f54920ad26" exitCode=0
Sep 10 03:13:07.935764 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.935771    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f23388-c9dc64984-gqgjr"
Sep 10 03:13:07.935944 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.935873    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f23388-c9dc64984-gqgjr" event={"ID":"737f6d01-3d1f-49a1-9b5e-e249399b639f","Type":"ContainerDied","Data":"2aad509bfaf783df7b8da1a8c1fe52a7b30dc1be889ebda0b67858f54920ad26"}
Sep 10 03:13:07.935944 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.935930    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f23388-c9dc64984-gqgjr" event={"ID":"737f6d01-3d1f-49a1-9b5e-e249399b639f","Type":"ContainerDied","Data":"29c87bb3ea110ee25fe51ef13f10c373ca1b7a530cce4e163946d826f916ba18"}
Sep 10 03:13:07.944911 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.944892    2558 scope.go:117] "RemoveContainer" containerID="a6a933da7ef6bf465b3593df4b86a8d84ef33f828ca9d8c1d5af29d158e40a67"
Sep 10 03:13:07.945208 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:13:07.945188    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6a933da7ef6bf465b3593df4b86a8d84ef33f828ca9d8c1d5af29d158e40a67\": container with ID starting with a6a933da7ef6bf465b3593df4b86a8d84ef33f828ca9d8c1d5af29d158e40a67 not found: ID does not exist" containerID="a6a933da7ef6bf465b3593df4b86a8d84ef33f828ca9d8c1d5af29d158e40a67"
Sep 10 03:13:07.945283 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.945214    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6a933da7ef6bf465b3593df4b86a8d84ef33f828ca9d8c1d5af29d158e40a67"} err="failed to get container status \"a6a933da7ef6bf465b3593df4b86a8d84ef33f828ca9d8c1d5af29d158e40a67\": rpc error: code = NotFound desc = could not find container \"a6a933da7ef6bf465b3593df4b86a8d84ef33f828ca9d8c1d5af29d158e40a67\": container with ID starting with a6a933da7ef6bf465b3593df4b86a8d84ef33f828ca9d8c1d5af29d158e40a67 not found: ID does not exist"
Sep 10 03:13:07.945283 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.945230    2558 scope.go:117] "RemoveContainer" containerID="2aad509bfaf783df7b8da1a8c1fe52a7b30dc1be889ebda0b67858f54920ad26"
Sep 10 03:13:07.953589 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.953572    2558 scope.go:117] "RemoveContainer" containerID="2aad509bfaf783df7b8da1a8c1fe52a7b30dc1be889ebda0b67858f54920ad26"
Sep 10 03:13:07.953830 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:13:07.953811    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aad509bfaf783df7b8da1a8c1fe52a7b30dc1be889ebda0b67858f54920ad26\": container with ID starting with 2aad509bfaf783df7b8da1a8c1fe52a7b30dc1be889ebda0b67858f54920ad26 not found: ID does not exist" containerID="2aad509bfaf783df7b8da1a8c1fe52a7b30dc1be889ebda0b67858f54920ad26"
Sep 10 03:13:07.953893 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.953835    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aad509bfaf783df7b8da1a8c1fe52a7b30dc1be889ebda0b67858f54920ad26"} err="failed to get container status \"2aad509bfaf783df7b8da1a8c1fe52a7b30dc1be889ebda0b67858f54920ad26\": rpc error: code = NotFound desc = could not find container \"2aad509bfaf783df7b8da1a8c1fe52a7b30dc1be889ebda0b67858f54920ad26\": container with ID starting with 2aad509bfaf783df7b8da1a8c1fe52a7b30dc1be889ebda0b67858f54920ad26 not found: ID does not exist"
Sep 10 03:13:07.961319 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.961297    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f23388-c9dc64984-gqgjr"]
Sep 10 03:13:07.965428 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.965408    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f23388-c9dc64984-gqgjr"]
Sep 10 03:13:07.975846 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.975823    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f23388-6f684696d5-4phh8"]
Sep 10 03:13:07.982546 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:07.982500    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f23388-6f684696d5-4phh8"]
Sep 10 03:13:08.069723 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:08.069699    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="737f6d01-3d1f-49a1-9b5e-e249399b639f" path="/var/lib/kubelet/pods/737f6d01-3d1f-49a1-9b5e-e249399b639f/volumes"
Sep 10 03:13:08.070081 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:08.070065    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8aa9fa77-1b97-4ff0-83d5-ad0addcafc33" path="/var/lib/kubelet/pods/8aa9fa77-1b97-4ff0-83d5-ad0addcafc33/volumes"
Sep 10 03:13:09.498817 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:09.498781    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw"]
Sep 10 03:13:09.532662 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:09.532623    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:13:10.926864 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:10.926838    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:13:13.899196 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:13.899155    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:13:13.999440 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:13.999404    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:13:14.159844 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:14.159758    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:13:14.498037 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:14.498008    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1e0789-hhjph"]
Sep 10 03:13:14.498393 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:14.498380    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa9fa77-1b97-4ff0-83d5-ad0addcafc33" containerName="payload-processing"
Sep 10 03:13:14.498445 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:14.498394    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa9fa77-1b97-4ff0-83d5-ad0addcafc33" containerName="payload-processing"
Sep 10 03:13:14.498445 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:14.498417    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="737f6d01-3d1f-49a1-9b5e-e249399b639f" containerName="maas-api"
Sep 10 03:13:14.498445 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:14.498423    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="737f6d01-3d1f-49a1-9b5e-e249399b639f" containerName="maas-api"
Sep 10 03:13:14.498536 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:14.498480    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aa9fa77-1b97-4ff0-83d5-ad0addcafc33" containerName="payload-processing"
Sep 10 03:13:14.498536 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:14.498487    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="737f6d01-3d1f-49a1-9b5e-e249399b639f" containerName="maas-api"
Sep 10 03:13:14.502330 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:14.502312    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1e0789-hhjph"
Sep 10 03:13:14.504723 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:14.504706    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8jjl\""
Sep 10 03:13:14.509127 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:14.509094    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1e0789-hhjph"]
Sep 10 03:13:14.607609 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:14.607580    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a689b853-5e88-461a-b1b4-53618cc3ac40-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1e0789-hhjph\" (UID: \"a689b853-5e88-461a-b1b4-53618cc3ac40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1e0789-hhjph"
Sep 10 03:13:14.642981 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:14.642959    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:13:14.708693 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:14.708664    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a689b853-5e88-461a-b1b4-53618cc3ac40-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1e0789-hhjph\" (UID: \"a689b853-5e88-461a-b1b4-53618cc3ac40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1e0789-hhjph"
Sep 10 03:13:14.709278 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:14.709262    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a689b853-5e88-461a-b1b4-53618cc3ac40-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1e0789-hhjph\" (UID: \"a689b853-5e88-461a-b1b4-53618cc3ac40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1e0789-hhjph"
Sep 10 03:13:14.814376 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:14.814318    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1e0789-hhjph"
Sep 10 03:13:15.086259 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:15.086233    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1e0789-hhjph"]
Sep 10 03:13:15.087806 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:13:15.087774    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda689b853_5e88_461a_b1b4_53618cc3ac40.slice/crio-3a925901c1b8357edf7b61781079331c20dd81d57999dcd57aaa4f092b4e2809 WatchSource:0}: Error finding container 3a925901c1b8357edf7b61781079331c20dd81d57999dcd57aaa4f092b4e2809: Status 404 returned error can't find the container with id 3a925901c1b8357edf7b61781079331c20dd81d57999dcd57aaa4f092b4e2809
Sep 10 03:13:15.986204 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:15.986172    2558 generic.go:356] "Generic (PLEG): container finished" podID="a689b853-5e88-461a-b1b4-53618cc3ac40" containerID="789b7558363dbfdc1b01c86db9f129bf0cb74c402535e41c8b0a6bdeacd77e56" exitCode=0
Sep 10 03:13:15.986527 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:15.986259    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1e0789-hhjph" event={"ID":"a689b853-5e88-461a-b1b4-53618cc3ac40","Type":"ContainerDied","Data":"789b7558363dbfdc1b01c86db9f129bf0cb74c402535e41c8b0a6bdeacd77e56"}
Sep 10 03:13:15.986527 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:15.986317    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1e0789-hhjph" event={"ID":"a689b853-5e88-461a-b1b4-53618cc3ac40","Type":"ContainerStarted","Data":"3a925901c1b8357edf7b61781079331c20dd81d57999dcd57aaa4f092b4e2809"}
Sep 10 03:13:17.127839 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:17.127811    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1e0789-hhjph"
Sep 10 03:13:17.228573 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:17.228544    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a689b853-5e88-461a-b1b4-53618cc3ac40-maas-api-service-ca\") pod \"a689b853-5e88-461a-b1b4-53618cc3ac40\" (UID: \"a689b853-5e88-461a-b1b4-53618cc3ac40\") "
Sep 10 03:13:17.228867 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:17.228845    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a689b853-5e88-461a-b1b4-53618cc3ac40-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a689b853-5e88-461a-b1b4-53618cc3ac40" (UID: "a689b853-5e88-461a-b1b4-53618cc3ac40"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:13:17.329302 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:17.329231    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a689b853-5e88-461a-b1b4-53618cc3ac40-maas-api-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:13:17.998634 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:17.998609    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1e0789-hhjph"
Sep 10 03:13:17.998793 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:17.998651    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1e0789-hhjph" event={"ID":"a689b853-5e88-461a-b1b4-53618cc3ac40","Type":"ContainerDied","Data":"3a925901c1b8357edf7b61781079331c20dd81d57999dcd57aaa4f092b4e2809"}
Sep 10 03:13:17.998793 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:17.998693    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a925901c1b8357edf7b61781079331c20dd81d57999dcd57aaa4f092b4e2809"
Sep 10 03:13:24.519001 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:24.518960    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw" podUID="0776ba8d-636f-45d7-a537-e587b292b9d6" containerName="main" containerID="cri-o://ae55e205a91d2ce068aaa93134334b0245d4b01d179b5958db244cae5222834b" gracePeriod=45
Sep 10 03:13:24.783420 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:24.783361    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw"
Sep 10 03:13:24.889365 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:24.889336    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0776ba8d-636f-45d7-a537-e587b292b9d6-model-cache\") pod \"0776ba8d-636f-45d7-a537-e587b292b9d6\" (UID: \"0776ba8d-636f-45d7-a537-e587b292b9d6\") "
Sep 10 03:13:24.889509 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:24.889435    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q244r\" (UniqueName: \"kubernetes.io/projected/0776ba8d-636f-45d7-a537-e587b292b9d6-kube-api-access-q244r\") pod \"0776ba8d-636f-45d7-a537-e587b292b9d6\" (UID: \"0776ba8d-636f-45d7-a537-e587b292b9d6\") "
Sep 10 03:13:24.889509 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:24.889463    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0776ba8d-636f-45d7-a537-e587b292b9d6-dshm\") pod \"0776ba8d-636f-45d7-a537-e587b292b9d6\" (UID: \"0776ba8d-636f-45d7-a537-e587b292b9d6\") "
Sep 10 03:13:24.889509 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:24.889502    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0776ba8d-636f-45d7-a537-e587b292b9d6-tls-certs\") pod \"0776ba8d-636f-45d7-a537-e587b292b9d6\" (UID: \"0776ba8d-636f-45d7-a537-e587b292b9d6\") "
Sep 10 03:13:24.889696 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:24.889591    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0776ba8d-636f-45d7-a537-e587b292b9d6-model-cache" (OuterVolumeSpecName: "model-cache") pod "0776ba8d-636f-45d7-a537-e587b292b9d6" (UID: "0776ba8d-636f-45d7-a537-e587b292b9d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:13:24.889696 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:24.889627    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0776ba8d-636f-45d7-a537-e587b292b9d6-tmp-dir\") pod \"0776ba8d-636f-45d7-a537-e587b292b9d6\" (UID: \"0776ba8d-636f-45d7-a537-e587b292b9d6\") "
Sep 10 03:13:24.889696 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:24.889667    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0776ba8d-636f-45d7-a537-e587b292b9d6-home\") pod \"0776ba8d-636f-45d7-a537-e587b292b9d6\" (UID: \"0776ba8d-636f-45d7-a537-e587b292b9d6\") "
Sep 10 03:13:24.889942 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:24.889888    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0776ba8d-636f-45d7-a537-e587b292b9d6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0776ba8d-636f-45d7-a537-e587b292b9d6" (UID: "0776ba8d-636f-45d7-a537-e587b292b9d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:13:24.890013 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:24.889949    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0776ba8d-636f-45d7-a537-e587b292b9d6-home" (OuterVolumeSpecName: "home") pod "0776ba8d-636f-45d7-a537-e587b292b9d6" (UID: "0776ba8d-636f-45d7-a537-e587b292b9d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:13:24.890013 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:24.889963    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0776ba8d-636f-45d7-a537-e587b292b9d6-model-cache\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:13:24.891665 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:24.891635    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0776ba8d-636f-45d7-a537-e587b292b9d6-dshm" (OuterVolumeSpecName: "dshm") pod "0776ba8d-636f-45d7-a537-e587b292b9d6" (UID: "0776ba8d-636f-45d7-a537-e587b292b9d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:13:24.891782 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:24.891692    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0776ba8d-636f-45d7-a537-e587b292b9d6-kube-api-access-q244r" (OuterVolumeSpecName: "kube-api-access-q244r") pod "0776ba8d-636f-45d7-a537-e587b292b9d6" (UID: "0776ba8d-636f-45d7-a537-e587b292b9d6"). InnerVolumeSpecName "kube-api-access-q244r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:13:24.891782 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:24.891696    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0776ba8d-636f-45d7-a537-e587b292b9d6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0776ba8d-636f-45d7-a537-e587b292b9d6" (UID: "0776ba8d-636f-45d7-a537-e587b292b9d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:13:24.990791 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:24.990764    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q244r\" (UniqueName: \"kubernetes.io/projected/0776ba8d-636f-45d7-a537-e587b292b9d6-kube-api-access-q244r\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:13:24.990791 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:24.990787    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0776ba8d-636f-45d7-a537-e587b292b9d6-dshm\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:13:24.990933 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:24.990800    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0776ba8d-636f-45d7-a537-e587b292b9d6-tls-certs\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:13:24.990933 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:24.990813    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0776ba8d-636f-45d7-a537-e587b292b9d6-tmp-dir\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:13:24.990933 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:24.990823    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0776ba8d-636f-45d7-a537-e587b292b9d6-home\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:13:25.041762 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.041706    2558 generic.go:356] "Generic (PLEG): container finished" podID="0776ba8d-636f-45d7-a537-e587b292b9d6" containerID="ae55e205a91d2ce068aaa93134334b0245d4b01d179b5958db244cae5222834b" exitCode=0
Sep 10 03:13:25.041762 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.041744    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw"
Sep 10 03:13:25.041921 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.041802    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw" event={"ID":"0776ba8d-636f-45d7-a537-e587b292b9d6","Type":"ContainerDied","Data":"ae55e205a91d2ce068aaa93134334b0245d4b01d179b5958db244cae5222834b"}
Sep 10 03:13:25.041921 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.041848    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw" event={"ID":"0776ba8d-636f-45d7-a537-e587b292b9d6","Type":"ContainerDied","Data":"0d7375b4d83995f2c83e4306e19cb9c7d88c44260b4d2156c4f1f29b9db90fd1"}
Sep 10 03:13:25.041921 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.041871    2558 scope.go:117] "RemoveContainer" containerID="ae55e205a91d2ce068aaa93134334b0245d4b01d179b5958db244cae5222834b"
Sep 10 03:13:25.052116 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.052085    2558 scope.go:117] "RemoveContainer" containerID="ae55e205a91d2ce068aaa93134334b0245d4b01d179b5958db244cae5222834b"
Sep 10 03:13:25.052413 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:13:25.052394    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae55e205a91d2ce068aaa93134334b0245d4b01d179b5958db244cae5222834b\": container with ID starting with ae55e205a91d2ce068aaa93134334b0245d4b01d179b5958db244cae5222834b not found: ID does not exist" containerID="ae55e205a91d2ce068aaa93134334b0245d4b01d179b5958db244cae5222834b"
Sep 10 03:13:25.052454 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.052422    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae55e205a91d2ce068aaa93134334b0245d4b01d179b5958db244cae5222834b"} err="failed to get container status \"ae55e205a91d2ce068aaa93134334b0245d4b01d179b5958db244cae5222834b\": rpc error: code = NotFound desc = could not find container \"ae55e205a91d2ce068aaa93134334b0245d4b01d179b5958db244cae5222834b\": container with ID starting with ae55e205a91d2ce068aaa93134334b0245d4b01d179b5958db244cae5222834b not found: ID does not exist"
Sep 10 03:13:25.064987 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.064966    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw"]
Sep 10 03:13:25.070724 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.070705    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/e2e-conflict-b-8ec4ab59-kserve-cc745c944-z6dbw"]
Sep 10 03:13:25.227559 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.227529    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:13:25.269805 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.269775    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1e0789-6595f797b5-wr8p8"]
Sep 10 03:13:25.270036 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.270012    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1e0789-6595f797b5-wr8p8" podUID="afdadca6-01cc-481a-bb19-425c3b10d50f" containerName="payload-processing" containerID="cri-o://5c67abab628a2fcde826306ace0c1f365e55a447582f24e07f7efbe1865906fa" gracePeriod=30
Sep 10 03:13:25.276583 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.276555    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1e0789-77bbc99758-4fsxv"]
Sep 10 03:13:25.276802 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.276744    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1e0789-77bbc99758-4fsxv" podUID="f40da815-568b-44f8-90eb-1bcaf4df063c" containerName="payload-pre-processing" containerID="cri-o://7d05feb393af6ffeb5eea75a66eef91009e8b56649c5c881d9f2d465cf9b8f4d" gracePeriod=30
Sep 10 03:13:25.325112 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.325029    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:13:25.415712 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.415678    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:13:25.497002 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.496973    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1e0789-hhjph"]
Sep 10 03:13:25.498402 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.498370    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:13:25.501227 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.501207    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1e0789-hhjph"]
Sep 10 03:13:25.521445 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.521426    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1e0789-6595f797b5-wr8p8"
Sep 10 03:13:25.573394 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.573371    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1e0789-77bbc99758-4fsxv"
Sep 10 03:13:25.573840 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.573813    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:13:25.595513 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.595456    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afdadca6-01cc-481a-bb19-425c3b10d50f-plugins-config-volume\") pod \"afdadca6-01cc-481a-bb19-425c3b10d50f\" (UID: \"afdadca6-01cc-481a-bb19-425c3b10d50f\") "
Sep 10 03:13:25.595634 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.595617    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjv4r\" (UniqueName: \"kubernetes.io/projected/afdadca6-01cc-481a-bb19-425c3b10d50f-kube-api-access-bjv4r\") pod \"afdadca6-01cc-481a-bb19-425c3b10d50f\" (UID: \"afdadca6-01cc-481a-bb19-425c3b10d50f\") "
Sep 10 03:13:25.595848 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.595825    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afdadca6-01cc-481a-bb19-425c3b10d50f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "afdadca6-01cc-481a-bb19-425c3b10d50f" (UID: "afdadca6-01cc-481a-bb19-425c3b10d50f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:13:25.597624 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.597596    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afdadca6-01cc-481a-bb19-425c3b10d50f-kube-api-access-bjv4r" (OuterVolumeSpecName: "kube-api-access-bjv4r") pod "afdadca6-01cc-481a-bb19-425c3b10d50f" (UID: "afdadca6-01cc-481a-bb19-425c3b10d50f"). InnerVolumeSpecName "kube-api-access-bjv4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:13:25.697075 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.697051    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40da815-568b-44f8-90eb-1bcaf4df063c-plugins-config-volume\") pod \"f40da815-568b-44f8-90eb-1bcaf4df063c\" (UID: \"f40da815-568b-44f8-90eb-1bcaf4df063c\") "
Sep 10 03:13:25.697175 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.697096    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfzw4\" (UniqueName: \"kubernetes.io/projected/f40da815-568b-44f8-90eb-1bcaf4df063c-kube-api-access-tfzw4\") pod \"f40da815-568b-44f8-90eb-1bcaf4df063c\" (UID: \"f40da815-568b-44f8-90eb-1bcaf4df063c\") "
Sep 10 03:13:25.697274 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.697261    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjv4r\" (UniqueName: \"kubernetes.io/projected/afdadca6-01cc-481a-bb19-425c3b10d50f-kube-api-access-bjv4r\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:13:25.697346 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.697275    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afdadca6-01cc-481a-bb19-425c3b10d50f-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:13:25.697423 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.697363    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f40da815-568b-44f8-90eb-1bcaf4df063c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f40da815-568b-44f8-90eb-1bcaf4df063c" (UID: "f40da815-568b-44f8-90eb-1bcaf4df063c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:13:25.699189 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.699166    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40da815-568b-44f8-90eb-1bcaf4df063c-kube-api-access-tfzw4" (OuterVolumeSpecName: "kube-api-access-tfzw4") pod "f40da815-568b-44f8-90eb-1bcaf4df063c" (UID: "f40da815-568b-44f8-90eb-1bcaf4df063c"). InnerVolumeSpecName "kube-api-access-tfzw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:13:25.798034 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.798007    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40da815-568b-44f8-90eb-1bcaf4df063c-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:13:25.798034 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.798030    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfzw4\" (UniqueName: \"kubernetes.io/projected/f40da815-568b-44f8-90eb-1bcaf4df063c-kube-api-access-tfzw4\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:13:25.856425 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:25.856353    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:13:26.048626 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:26.048597    2558 generic.go:356] "Generic (PLEG): container finished" podID="f40da815-568b-44f8-90eb-1bcaf4df063c" containerID="7d05feb393af6ffeb5eea75a66eef91009e8b56649c5c881d9f2d465cf9b8f4d" exitCode=0
Sep 10 03:13:26.048722 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:26.048642    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1e0789-77bbc99758-4fsxv"
Sep 10 03:13:26.048722 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:26.048687    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1e0789-77bbc99758-4fsxv" event={"ID":"f40da815-568b-44f8-90eb-1bcaf4df063c","Type":"ContainerDied","Data":"7d05feb393af6ffeb5eea75a66eef91009e8b56649c5c881d9f2d465cf9b8f4d"}
Sep 10 03:13:26.048850 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:26.048738    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1e0789-77bbc99758-4fsxv" event={"ID":"f40da815-568b-44f8-90eb-1bcaf4df063c","Type":"ContainerDied","Data":"c42ed975eaeb2a9e4c5c3e386ff910e4657f017012d4c3ab289d8078a6b39f03"}
Sep 10 03:13:26.048850 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:26.048754    2558 scope.go:117] "RemoveContainer" containerID="7d05feb393af6ffeb5eea75a66eef91009e8b56649c5c881d9f2d465cf9b8f4d"
Sep 10 03:13:26.051450 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:26.051426    2558 generic.go:356] "Generic (PLEG): container finished" podID="afdadca6-01cc-481a-bb19-425c3b10d50f" containerID="5c67abab628a2fcde826306ace0c1f365e55a447582f24e07f7efbe1865906fa" exitCode=0
Sep 10 03:13:26.051570 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:26.051475    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1e0789-6595f797b5-wr8p8"
Sep 10 03:13:26.051570 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:26.051506    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1e0789-6595f797b5-wr8p8" event={"ID":"afdadca6-01cc-481a-bb19-425c3b10d50f","Type":"ContainerDied","Data":"5c67abab628a2fcde826306ace0c1f365e55a447582f24e07f7efbe1865906fa"}
Sep 10 03:13:26.051570 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:26.051536    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1e0789-6595f797b5-wr8p8" event={"ID":"afdadca6-01cc-481a-bb19-425c3b10d50f","Type":"ContainerDied","Data":"dd17690c763aafdd005323f552274d25e0bad4e4cd7877a150025b3f1273b6ba"}
Sep 10 03:13:26.060082 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:26.060047    2558 scope.go:117] "RemoveContainer" containerID="7d05feb393af6ffeb5eea75a66eef91009e8b56649c5c881d9f2d465cf9b8f4d"
Sep 10 03:13:26.060403 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:13:26.060375    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d05feb393af6ffeb5eea75a66eef91009e8b56649c5c881d9f2d465cf9b8f4d\": container with ID starting with 7d05feb393af6ffeb5eea75a66eef91009e8b56649c5c881d9f2d465cf9b8f4d not found: ID does not exist" containerID="7d05feb393af6ffeb5eea75a66eef91009e8b56649c5c881d9f2d465cf9b8f4d"
Sep 10 03:13:26.060501 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:26.060408    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d05feb393af6ffeb5eea75a66eef91009e8b56649c5c881d9f2d465cf9b8f4d"} err="failed to get container status \"7d05feb393af6ffeb5eea75a66eef91009e8b56649c5c881d9f2d465cf9b8f4d\": rpc error: code = NotFound desc = could not find container \"7d05feb393af6ffeb5eea75a66eef91009e8b56649c5c881d9f2d465cf9b8f4d\": container with ID starting with 7d05feb393af6ffeb5eea75a66eef91009e8b56649c5c881d9f2d465cf9b8f4d not found: ID does not exist"
Sep 10 03:13:26.060501 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:26.060428    2558 scope.go:117] "RemoveContainer" containerID="5c67abab628a2fcde826306ace0c1f365e55a447582f24e07f7efbe1865906fa"
Sep 10 03:13:26.068722 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:26.068694    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0776ba8d-636f-45d7-a537-e587b292b9d6" path="/var/lib/kubelet/pods/0776ba8d-636f-45d7-a537-e587b292b9d6/volumes"
Sep 10 03:13:26.069156 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:26.069134    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a689b853-5e88-461a-b1b4-53618cc3ac40" path="/var/lib/kubelet/pods/a689b853-5e88-461a-b1b4-53618cc3ac40/volumes"
Sep 10 03:13:26.071489 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:26.071467    2558 scope.go:117] "RemoveContainer" containerID="5c67abab628a2fcde826306ace0c1f365e55a447582f24e07f7efbe1865906fa"
Sep 10 03:13:26.072016 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:13:26.071992    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c67abab628a2fcde826306ace0c1f365e55a447582f24e07f7efbe1865906fa\": container with ID starting with 5c67abab628a2fcde826306ace0c1f365e55a447582f24e07f7efbe1865906fa not found: ID does not exist" containerID="5c67abab628a2fcde826306ace0c1f365e55a447582f24e07f7efbe1865906fa"
Sep 10 03:13:26.072331 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:26.072023    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c67abab628a2fcde826306ace0c1f365e55a447582f24e07f7efbe1865906fa"} err="failed to get container status \"5c67abab628a2fcde826306ace0c1f365e55a447582f24e07f7efbe1865906fa\": rpc error: code = NotFound desc = could not find container \"5c67abab628a2fcde826306ace0c1f365e55a447582f24e07f7efbe1865906fa\": container with ID starting with 5c67abab628a2fcde826306ace0c1f365e55a447582f24e07f7efbe1865906fa not found: ID does not exist"
Sep 10 03:13:26.086407 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:26.086386    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1e0789-6595f797b5-wr8p8"]
Sep 10 03:13:26.093645 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:26.093626    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1e0789-6595f797b5-wr8p8"]
Sep 10 03:13:26.131692 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:26.131640    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1e0789-77bbc99758-4fsxv"]
Sep 10 03:13:26.145321 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:26.145300    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1e0789-77bbc99758-4fsxv"]
Sep 10 03:13:27.872936 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:27.872904    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:13:28.069202 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:28.069167    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afdadca6-01cc-481a-bb19-425c3b10d50f" path="/var/lib/kubelet/pods/afdadca6-01cc-481a-bb19-425c3b10d50f/volumes"
Sep 10 03:13:28.069588 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:28.069567    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f40da815-568b-44f8-90eb-1bcaf4df063c" path="/var/lib/kubelet/pods/f40da815-568b-44f8-90eb-1bcaf4df063c/volumes"
Sep 10 03:13:32.006854 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:32.006797    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:13:32.074892 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:32.074860    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:13:32.274316 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:32.274234    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:13:32.600402 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:32.600329    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c3058d-wsmf8"]
Sep 10 03:13:32.600703 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:32.600691    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afdadca6-01cc-481a-bb19-425c3b10d50f" containerName="payload-processing"
Sep 10 03:13:32.600748 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:32.600705    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="afdadca6-01cc-481a-bb19-425c3b10d50f" containerName="payload-processing"
Sep 10 03:13:32.600748 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:32.600716    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a689b853-5e88-461a-b1b4-53618cc3ac40" containerName="revoke-keys"
Sep 10 03:13:32.600748 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:32.600722    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a689b853-5e88-461a-b1b4-53618cc3ac40" containerName="revoke-keys"
Sep 10 03:13:32.600748 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:32.600744    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0776ba8d-636f-45d7-a537-e587b292b9d6" containerName="main"
Sep 10 03:13:32.600748 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:32.600749    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="0776ba8d-636f-45d7-a537-e587b292b9d6" containerName="main"
Sep 10 03:13:32.600898 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:32.600756    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f40da815-568b-44f8-90eb-1bcaf4df063c" containerName="payload-pre-processing"
Sep 10 03:13:32.600898 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:32.600761    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f40da815-568b-44f8-90eb-1bcaf4df063c" containerName="payload-pre-processing"
Sep 10 03:13:32.600898 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:32.600817    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f40da815-568b-44f8-90eb-1bcaf4df063c" containerName="payload-pre-processing"
Sep 10 03:13:32.600898 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:32.600827    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="afdadca6-01cc-481a-bb19-425c3b10d50f" containerName="payload-processing"
Sep 10 03:13:32.600898 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:32.600832    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a689b853-5e88-461a-b1b4-53618cc3ac40" containerName="revoke-keys"
Sep 10 03:13:32.600898 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:32.600837    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="0776ba8d-636f-45d7-a537-e587b292b9d6" containerName="main"
Sep 10 03:13:32.605211 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:32.605193    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c3058d-wsmf8"
Sep 10 03:13:32.607530 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:32.607507    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8jjl\""
Sep 10 03:13:32.611781 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:32.611758    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c3058d-wsmf8"]
Sep 10 03:13:32.704158 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:32.704129    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:13:32.752904 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:32.752880    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6dee1c1-7fd9-468b-97de-68760fc09d61-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c3058d-wsmf8\" (UID: \"d6dee1c1-7fd9-468b-97de-68760fc09d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c3058d-wsmf8"
Sep 10 03:13:32.854251 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:32.854192    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6dee1c1-7fd9-468b-97de-68760fc09d61-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c3058d-wsmf8\" (UID: \"d6dee1c1-7fd9-468b-97de-68760fc09d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c3058d-wsmf8"
Sep 10 03:13:32.854764 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:32.854741    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6dee1c1-7fd9-468b-97de-68760fc09d61-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c3058d-wsmf8\" (UID: \"d6dee1c1-7fd9-468b-97de-68760fc09d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c3058d-wsmf8"
Sep 10 03:13:32.917071 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:32.917046    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c3058d-wsmf8"
Sep 10 03:13:33.189178 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:33.189156    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c3058d-wsmf8"]
Sep 10 03:13:33.190458 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:13:33.190429    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6dee1c1_7fd9_468b_97de_68760fc09d61.slice/crio-331211169c2da9c61962c210cae6abf561c2665639fb6d3781ba9dd4bfa7b442 WatchSource:0}: Error finding container 331211169c2da9c61962c210cae6abf561c2665639fb6d3781ba9dd4bfa7b442: Status 404 returned error can't find the container with id 331211169c2da9c61962c210cae6abf561c2665639fb6d3781ba9dd4bfa7b442
Sep 10 03:13:34.111887 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:34.111860    2558 generic.go:356] "Generic (PLEG): container finished" podID="d6dee1c1-7fd9-468b-97de-68760fc09d61" containerID="0db224d05ac744e0d775210a8917395a6307e0144892eb4f1c5b2f68c67618d9" exitCode=0
Sep 10 03:13:34.112005 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:34.111951    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c3058d-wsmf8" event={"ID":"d6dee1c1-7fd9-468b-97de-68760fc09d61","Type":"ContainerDied","Data":"0db224d05ac744e0d775210a8917395a6307e0144892eb4f1c5b2f68c67618d9"}
Sep 10 03:13:34.112060 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:34.112004    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c3058d-wsmf8" event={"ID":"d6dee1c1-7fd9-468b-97de-68760fc09d61","Type":"ContainerStarted","Data":"331211169c2da9c61962c210cae6abf561c2665639fb6d3781ba9dd4bfa7b442"}
Sep 10 03:13:35.254738 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:35.254717    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c3058d-wsmf8"
Sep 10 03:13:35.377044 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:35.376973    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6dee1c1-7fd9-468b-97de-68760fc09d61-maas-api-service-ca\") pod \"d6dee1c1-7fd9-468b-97de-68760fc09d61\" (UID: \"d6dee1c1-7fd9-468b-97de-68760fc09d61\") "
Sep 10 03:13:35.377329 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:35.377307    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6dee1c1-7fd9-468b-97de-68760fc09d61-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d6dee1c1-7fd9-468b-97de-68760fc09d61" (UID: "d6dee1c1-7fd9-468b-97de-68760fc09d61"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:13:35.478223 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:35.478196    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6dee1c1-7fd9-468b-97de-68760fc09d61-maas-api-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:13:36.125645 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:36.125617    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c3058d-wsmf8"
Sep 10 03:13:36.125760 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:36.125647    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c3058d-wsmf8" event={"ID":"d6dee1c1-7fd9-468b-97de-68760fc09d61","Type":"ContainerDied","Data":"331211169c2da9c61962c210cae6abf561c2665639fb6d3781ba9dd4bfa7b442"}
Sep 10 03:13:36.125760 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:36.125687    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="331211169c2da9c61962c210cae6abf561c2665639fb6d3781ba9dd4bfa7b442"
Sep 10 03:13:43.165438 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:43.165296    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:13:43.268195 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:43.268149    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:13:43.359880 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:43.359843    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:13:43.379609 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:43.379583    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c3058d-wsmf8"]
Sep 10 03:13:43.381784 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:43.381763    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c3058d-wsmf8"]
Sep 10 03:13:43.459870 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:43.459843    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:13:43.798059 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:43.797976    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:13:44.069767 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:44.069685    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6dee1c1-7fd9-468b-97de-68760fc09d61" path="/var/lib/kubelet/pods/d6dee1c1-7fd9-468b-97de-68760fc09d61/volumes"
Sep 10 03:13:49.561270 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:49.561220    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:13:49.628889 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:49.628860    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:13:49.830922 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:13:49.830818    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:14:05.929119 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:05.929070    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:14:06.024850 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:06.024789    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:14:23.361830 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:23.361783    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:14:26.892091 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:26.892056    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8015-984cd1"]
Sep 10 03:14:26.892475 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:26.892435    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6dee1c1-7fd9-468b-97de-68760fc09d61" containerName="revoke-keys"
Sep 10 03:14:26.892475 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:26.892448    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6dee1c1-7fd9-468b-97de-68760fc09d61" containerName="revoke-keys"
Sep 10 03:14:26.892554 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:26.892505    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6dee1c1-7fd9-468b-97de-68760fc09d61" containerName="revoke-keys"
Sep 10 03:14:26.895560 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:26.895541    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8015-984cd1"
Sep 10 03:14:26.898015 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:26.897992    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mtn2b\""
Sep 10 03:14:26.900190 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:26.900168    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8015-984cd1"]
Sep 10 03:14:26.960005 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:26.959981    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f97dg\" (UniqueName: \"kubernetes.io/projected/ce7e3a0b-003f-4fa1-b2ef-e8a745851369-kube-api-access-f97dg\") pod \"test-curl-8015-984cd1\" (UID: \"ce7e3a0b-003f-4fa1-b2ef-e8a745851369\") " pod="opendatahub/test-curl-8015-984cd1"
Sep 10 03:14:27.060459 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:27.060437    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f97dg\" (UniqueName: \"kubernetes.io/projected/ce7e3a0b-003f-4fa1-b2ef-e8a745851369-kube-api-access-f97dg\") pod \"test-curl-8015-984cd1\" (UID: \"ce7e3a0b-003f-4fa1-b2ef-e8a745851369\") " pod="opendatahub/test-curl-8015-984cd1"
Sep 10 03:14:27.068005 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:27.067979    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f97dg\" (UniqueName: \"kubernetes.io/projected/ce7e3a0b-003f-4fa1-b2ef-e8a745851369-kube-api-access-f97dg\") pod \"test-curl-8015-984cd1\" (UID: \"ce7e3a0b-003f-4fa1-b2ef-e8a745851369\") " pod="opendatahub/test-curl-8015-984cd1"
Sep 10 03:14:27.206517 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:27.206492    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8015-984cd1"
Sep 10 03:14:27.475421 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:27.475344    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8015-984cd1"]
Sep 10 03:14:27.478207 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:14:27.478177    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce7e3a0b_003f_4fa1_b2ef_e8a745851369.slice/crio-47f55f4d664e40d306e1ca7234460c38e6dc579257669ad17c08e9e616a000eb WatchSource:0}: Error finding container 47f55f4d664e40d306e1ca7234460c38e6dc579257669ad17c08e9e616a000eb: Status 404 returned error can't find the container with id 47f55f4d664e40d306e1ca7234460c38e6dc579257669ad17c08e9e616a000eb
Sep 10 03:14:28.420991 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:28.420957    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8015-984cd1" event={"ID":"ce7e3a0b-003f-4fa1-b2ef-e8a745851369","Type":"ContainerStarted","Data":"591fab626d66c75ff8d03b4875a27594af5f8c62a95269772ac23b77c26227b6"}
Sep 10 03:14:28.421387 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:28.421000    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8015-984cd1" event={"ID":"ce7e3a0b-003f-4fa1-b2ef-e8a745851369","Type":"ContainerStarted","Data":"47f55f4d664e40d306e1ca7234460c38e6dc579257669ad17c08e9e616a000eb"}
Sep 10 03:14:28.438168 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:28.438122    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8015-984cd1" podStartSLOduration=2.438097841 podStartE2EDuration="2.438097841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:14:28.435292716 +0000 UTC m=+1658.810378639" watchObservedRunningTime="2026-09-10 03:14:28.438097841 +0000 UTC m=+1658.813183762"
Sep 10 03:14:29.059633 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:29.059598    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8015-984cd1"]
Sep 10 03:14:29.063244 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:29.063214    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8015-984cd1"]
Sep 10 03:14:29.426836 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:14:29.426805    2558 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8015-984cd1" container="test-curl-8015-984cd1"
Sep 10 03:14:30.431827 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:30.431789    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8015-984cd1" podUID="ce7e3a0b-003f-4fa1-b2ef-e8a745851369" containerName="test-curl-8015-984cd1" containerID="cri-o://591fab626d66c75ff8d03b4875a27594af5f8c62a95269772ac23b77c26227b6" gracePeriod=2
Sep 10 03:14:30.635263 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:30.635233    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:14:31.797261 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:31.797219    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:14:32.597059 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:32.597034    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8015-984cd1"
Sep 10 03:14:32.705545 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:32.705524    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f97dg\" (UniqueName: \"kubernetes.io/projected/ce7e3a0b-003f-4fa1-b2ef-e8a745851369-kube-api-access-f97dg\") pod \"ce7e3a0b-003f-4fa1-b2ef-e8a745851369\" (UID: \"ce7e3a0b-003f-4fa1-b2ef-e8a745851369\") "
Sep 10 03:14:32.707526 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:32.707501    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce7e3a0b-003f-4fa1-b2ef-e8a745851369-kube-api-access-f97dg" (OuterVolumeSpecName: "kube-api-access-f97dg") pod "ce7e3a0b-003f-4fa1-b2ef-e8a745851369" (UID: "ce7e3a0b-003f-4fa1-b2ef-e8a745851369"). InnerVolumeSpecName "kube-api-access-f97dg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:14:32.807015 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:32.806984    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f97dg\" (UniqueName: \"kubernetes.io/projected/ce7e3a0b-003f-4fa1-b2ef-e8a745851369-kube-api-access-f97dg\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:14:33.450052 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:33.450022    2558 generic.go:356] "Generic (PLEG): container finished" podID="ce7e3a0b-003f-4fa1-b2ef-e8a745851369" containerID="591fab626d66c75ff8d03b4875a27594af5f8c62a95269772ac23b77c26227b6" exitCode=137
Sep 10 03:14:33.450238 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:33.450066    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8015-984cd1"
Sep 10 03:14:33.450238 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:33.450114    2558 scope.go:117] "RemoveContainer" containerID="591fab626d66c75ff8d03b4875a27594af5f8c62a95269772ac23b77c26227b6"
Sep 10 03:14:33.459873 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:33.459853    2558 scope.go:117] "RemoveContainer" containerID="591fab626d66c75ff8d03b4875a27594af5f8c62a95269772ac23b77c26227b6"
Sep 10 03:14:33.460199 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:14:33.460176    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"591fab626d66c75ff8d03b4875a27594af5f8c62a95269772ac23b77c26227b6\": container with ID starting with 591fab626d66c75ff8d03b4875a27594af5f8c62a95269772ac23b77c26227b6 not found: ID does not exist" containerID="591fab626d66c75ff8d03b4875a27594af5f8c62a95269772ac23b77c26227b6"
Sep 10 03:14:33.460293 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:33.460205    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"591fab626d66c75ff8d03b4875a27594af5f8c62a95269772ac23b77c26227b6"} err="failed to get container status \"591fab626d66c75ff8d03b4875a27594af5f8c62a95269772ac23b77c26227b6\": rpc error: code = NotFound desc = could not find container \"591fab626d66c75ff8d03b4875a27594af5f8c62a95269772ac23b77c26227b6\": container with ID starting with 591fab626d66c75ff8d03b4875a27594af5f8c62a95269772ac23b77c26227b6 not found: ID does not exist"
Sep 10 03:14:33.462036 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:14:33.462015    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8015-984cd1\" is forbidden: User \"system:node:ip-10-0-128-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-101.ec2.internal' and this object" podUID="ce7e3a0b-003f-4fa1-b2ef-e8a745851369" pod="opendatahub/test-curl-8015-984cd1"
Sep 10 03:14:34.070061 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:34.070032    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce7e3a0b-003f-4fa1-b2ef-e8a745851369" path="/var/lib/kubelet/pods/ce7e3a0b-003f-4fa1-b2ef-e8a745851369/volumes"
Sep 10 03:14:39.647508 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:39.647468    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd"]
Sep 10 03:14:39.648016 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:39.647996    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce7e3a0b-003f-4fa1-b2ef-e8a745851369" containerName="test-curl-8015-984cd1"
Sep 10 03:14:39.648140 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:39.648019    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce7e3a0b-003f-4fa1-b2ef-e8a745851369" containerName="test-curl-8015-984cd1"
Sep 10 03:14:39.648213 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:39.648145    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce7e3a0b-003f-4fa1-b2ef-e8a745851369" containerName="test-curl-8015-984cd1"
Sep 10 03:14:39.652400 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:39.652378    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd"
Sep 10 03:14:39.655463 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:39.655441    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-af7727\"/\"kube-root-ca.crt\""
Sep 10 03:14:39.655594 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:39.655508    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-af7727\"/\"openshift-service-ca.crt\""
Sep 10 03:14:39.656001 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:39.655971    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-af7727\"/\"sub-test-model-af7727-kserve-self-signed-certs\""
Sep 10 03:14:39.656241 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:39.656226    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-af7727\"/\"default-dockercfg-l2qbq\""
Sep 10 03:14:39.691922 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:39.691885    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:14:39.713982 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:39.713960    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd"]
Sep 10 03:14:39.757637 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:39.757606    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34cfdfdb-8ce5-4095-96a1-4d7152d99030-model-cache\") pod \"sub-test-model-af7727-kserve-7b7944f8c5-n4wdd\" (UID: \"34cfdfdb-8ce5-4095-96a1-4d7152d99030\") " pod="ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd"
Sep 10 03:14:39.757755 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:39.757637    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34cfdfdb-8ce5-4095-96a1-4d7152d99030-dshm\") pod \"sub-test-model-af7727-kserve-7b7944f8c5-n4wdd\" (UID: \"34cfdfdb-8ce5-4095-96a1-4d7152d99030\") " pod="ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd"
Sep 10 03:14:39.757755 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:39.757686    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34cfdfdb-8ce5-4095-96a1-4d7152d99030-tmp-dir\") pod \"sub-test-model-af7727-kserve-7b7944f8c5-n4wdd\" (UID: \"34cfdfdb-8ce5-4095-96a1-4d7152d99030\") " pod="ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd"
Sep 10 03:14:39.757755 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:39.757715    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2wnv\" (UniqueName: \"kubernetes.io/projected/34cfdfdb-8ce5-4095-96a1-4d7152d99030-kube-api-access-s2wnv\") pod \"sub-test-model-af7727-kserve-7b7944f8c5-n4wdd\" (UID: \"34cfdfdb-8ce5-4095-96a1-4d7152d99030\") " pod="ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd"
Sep 10 03:14:39.757755 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:39.757744    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34cfdfdb-8ce5-4095-96a1-4d7152d99030-home\") pod \"sub-test-model-af7727-kserve-7b7944f8c5-n4wdd\" (UID: \"34cfdfdb-8ce5-4095-96a1-4d7152d99030\") " pod="ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd"
Sep 10 03:14:39.757979 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:39.757775    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34cfdfdb-8ce5-4095-96a1-4d7152d99030-tls-certs\") pod \"sub-test-model-af7727-kserve-7b7944f8c5-n4wdd\" (UID: \"34cfdfdb-8ce5-4095-96a1-4d7152d99030\") " pod="ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd"
Sep 10 03:14:39.858576 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:39.858552    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34cfdfdb-8ce5-4095-96a1-4d7152d99030-tmp-dir\") pod \"sub-test-model-af7727-kserve-7b7944f8c5-n4wdd\" (UID: \"34cfdfdb-8ce5-4095-96a1-4d7152d99030\") " pod="ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd"
Sep 10 03:14:39.858709 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:39.858583    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2wnv\" (UniqueName: \"kubernetes.io/projected/34cfdfdb-8ce5-4095-96a1-4d7152d99030-kube-api-access-s2wnv\") pod \"sub-test-model-af7727-kserve-7b7944f8c5-n4wdd\" (UID: \"34cfdfdb-8ce5-4095-96a1-4d7152d99030\") " pod="ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd"
Sep 10 03:14:39.858709 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:39.858614    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34cfdfdb-8ce5-4095-96a1-4d7152d99030-home\") pod \"sub-test-model-af7727-kserve-7b7944f8c5-n4wdd\" (UID: \"34cfdfdb-8ce5-4095-96a1-4d7152d99030\") " pod="ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd"
Sep 10 03:14:39.858709 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:39.858644    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34cfdfdb-8ce5-4095-96a1-4d7152d99030-tls-certs\") pod \"sub-test-model-af7727-kserve-7b7944f8c5-n4wdd\" (UID: \"34cfdfdb-8ce5-4095-96a1-4d7152d99030\") " pod="ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd"
Sep 10 03:14:39.858709 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:39.858703    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34cfdfdb-8ce5-4095-96a1-4d7152d99030-model-cache\") pod \"sub-test-model-af7727-kserve-7b7944f8c5-n4wdd\" (UID: \"34cfdfdb-8ce5-4095-96a1-4d7152d99030\") " pod="ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd"
Sep 10 03:14:39.858952 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:39.858734    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34cfdfdb-8ce5-4095-96a1-4d7152d99030-dshm\") pod \"sub-test-model-af7727-kserve-7b7944f8c5-n4wdd\" (UID: \"34cfdfdb-8ce5-4095-96a1-4d7152d99030\") " pod="ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd"
Sep 10 03:14:39.859016 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:39.858997    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34cfdfdb-8ce5-4095-96a1-4d7152d99030-tmp-dir\") pod \"sub-test-model-af7727-kserve-7b7944f8c5-n4wdd\" (UID: \"34cfdfdb-8ce5-4095-96a1-4d7152d99030\") " pod="ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd"
Sep 10 03:14:39.859303 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:39.859279    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34cfdfdb-8ce5-4095-96a1-4d7152d99030-model-cache\") pod \"sub-test-model-af7727-kserve-7b7944f8c5-n4wdd\" (UID: \"34cfdfdb-8ce5-4095-96a1-4d7152d99030\") " pod="ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd"
Sep 10 03:14:39.859454 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:39.859341    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34cfdfdb-8ce5-4095-96a1-4d7152d99030-home\") pod \"sub-test-model-af7727-kserve-7b7944f8c5-n4wdd\" (UID: \"34cfdfdb-8ce5-4095-96a1-4d7152d99030\") " pod="ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd"
Sep 10 03:14:39.861468 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:39.861443    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34cfdfdb-8ce5-4095-96a1-4d7152d99030-dshm\") pod \"sub-test-model-af7727-kserve-7b7944f8c5-n4wdd\" (UID: \"34cfdfdb-8ce5-4095-96a1-4d7152d99030\") " pod="ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd"
Sep 10 03:14:39.861920 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:39.861891    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34cfdfdb-8ce5-4095-96a1-4d7152d99030-tls-certs\") pod \"sub-test-model-af7727-kserve-7b7944f8c5-n4wdd\" (UID: \"34cfdfdb-8ce5-4095-96a1-4d7152d99030\") " pod="ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd"
Sep 10 03:14:39.871093 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:39.871061    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2wnv\" (UniqueName: \"kubernetes.io/projected/34cfdfdb-8ce5-4095-96a1-4d7152d99030-kube-api-access-s2wnv\") pod \"sub-test-model-af7727-kserve-7b7944f8c5-n4wdd\" (UID: \"34cfdfdb-8ce5-4095-96a1-4d7152d99030\") " pod="ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd"
Sep 10 03:14:39.962518 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:39.962495    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd"
Sep 10 03:14:40.246766 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:14:40.246736    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34cfdfdb_8ce5_4095_96a1_4d7152d99030.slice/crio-9decc322b6dd8220c49f078a5098cefffa51d72361242e98ad292402e8c1423d WatchSource:0}: Error finding container 9decc322b6dd8220c49f078a5098cefffa51d72361242e98ad292402e8c1423d: Status 404 returned error can't find the container with id 9decc322b6dd8220c49f078a5098cefffa51d72361242e98ad292402e8c1423d
Sep 10 03:14:40.246915 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:40.246790    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd"]
Sep 10 03:14:40.494558 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:40.494469    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd" event={"ID":"34cfdfdb-8ce5-4095-96a1-4d7152d99030","Type":"ContainerStarted","Data":"bd70b2ecf71809ed3bdf06f04bf7bc4587b4653b2ff42bd2de3de2f5ad7d3cbe"}
Sep 10 03:14:40.494558 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:40.494523    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd" event={"ID":"34cfdfdb-8ce5-4095-96a1-4d7152d99030","Type":"ContainerStarted","Data":"9decc322b6dd8220c49f078a5098cefffa51d72361242e98ad292402e8c1423d"}
Sep 10 03:14:40.530588 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:40.530541    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd" podStartSLOduration=1.530526229 podStartE2EDuration="1.530526229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:14:40.527395705 +0000 UTC m=+1670.902481627" watchObservedRunningTime="2026-09-10 03:14:40.530526229 +0000 UTC m=+1670.905612149"
Sep 10 03:14:49.658245 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:49.658210    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:14:49.963491 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:49.963459    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd"
Sep 10 03:14:49.963644 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:49.963509    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd"
Sep 10 03:14:49.975854 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:49.975828    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd"
Sep 10 03:14:50.566717 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:50.566687    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd"
Sep 10 03:14:58.924083 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:58.924051    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh"]
Sep 10 03:14:58.927177 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:58.927159    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh"
Sep 10 03:14:58.930511 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:58.930486    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7daa14\"/\"default-dockercfg-n5bs2\""
Sep 10 03:14:58.930511 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:58.930500    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7daa14\"/\"kube-root-ca.crt\""
Sep 10 03:14:58.930693 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:58.930595    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7daa14\"/\"sub-test-model-7daa14-kserve-self-signed-certs\""
Sep 10 03:14:58.930693 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:58.930687    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7daa14\"/\"openshift-service-ca.crt\""
Sep 10 03:14:58.939472 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:58.939455    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh"]
Sep 10 03:14:58.959930 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:58.959902    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:14:59.007932 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:59.007901    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a2b3cd7-235a-43f2-a551-051f163717f1-model-cache\") pod \"sub-test-model-7daa14-kserve-5bfd95475f-29hmh\" (UID: \"5a2b3cd7-235a-43f2-a551-051f163717f1\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh"
Sep 10 03:14:59.008045 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:59.007951    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a2b3cd7-235a-43f2-a551-051f163717f1-dshm\") pod \"sub-test-model-7daa14-kserve-5bfd95475f-29hmh\" (UID: \"5a2b3cd7-235a-43f2-a551-051f163717f1\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh"
Sep 10 03:14:59.008045 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:59.007987    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km9pq\" (UniqueName: \"kubernetes.io/projected/5a2b3cd7-235a-43f2-a551-051f163717f1-kube-api-access-km9pq\") pod \"sub-test-model-7daa14-kserve-5bfd95475f-29hmh\" (UID: \"5a2b3cd7-235a-43f2-a551-051f163717f1\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh"
Sep 10 03:14:59.008045 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:59.008033    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a2b3cd7-235a-43f2-a551-051f163717f1-tls-certs\") pod \"sub-test-model-7daa14-kserve-5bfd95475f-29hmh\" (UID: \"5a2b3cd7-235a-43f2-a551-051f163717f1\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh"
Sep 10 03:14:59.008262 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:59.008091    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a2b3cd7-235a-43f2-a551-051f163717f1-tmp-dir\") pod \"sub-test-model-7daa14-kserve-5bfd95475f-29hmh\" (UID: \"5a2b3cd7-235a-43f2-a551-051f163717f1\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh"
Sep 10 03:14:59.008262 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:59.008191    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a2b3cd7-235a-43f2-a551-051f163717f1-home\") pod \"sub-test-model-7daa14-kserve-5bfd95475f-29hmh\" (UID: \"5a2b3cd7-235a-43f2-a551-051f163717f1\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh"
Sep 10 03:14:59.109077 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:59.109049    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a2b3cd7-235a-43f2-a551-051f163717f1-model-cache\") pod \"sub-test-model-7daa14-kserve-5bfd95475f-29hmh\" (UID: \"5a2b3cd7-235a-43f2-a551-051f163717f1\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh"
Sep 10 03:14:59.109190 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:59.109090    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a2b3cd7-235a-43f2-a551-051f163717f1-dshm\") pod \"sub-test-model-7daa14-kserve-5bfd95475f-29hmh\" (UID: \"5a2b3cd7-235a-43f2-a551-051f163717f1\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh"
Sep 10 03:14:59.109190 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:59.109145    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km9pq\" (UniqueName: \"kubernetes.io/projected/5a2b3cd7-235a-43f2-a551-051f163717f1-kube-api-access-km9pq\") pod \"sub-test-model-7daa14-kserve-5bfd95475f-29hmh\" (UID: \"5a2b3cd7-235a-43f2-a551-051f163717f1\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh"
Sep 10 03:14:59.109307 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:59.109187    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a2b3cd7-235a-43f2-a551-051f163717f1-tls-certs\") pod \"sub-test-model-7daa14-kserve-5bfd95475f-29hmh\" (UID: \"5a2b3cd7-235a-43f2-a551-051f163717f1\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh"
Sep 10 03:14:59.109363 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:59.109309    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a2b3cd7-235a-43f2-a551-051f163717f1-tmp-dir\") pod \"sub-test-model-7daa14-kserve-5bfd95475f-29hmh\" (UID: \"5a2b3cd7-235a-43f2-a551-051f163717f1\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh"
Sep 10 03:14:59.109420 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:59.109369    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a2b3cd7-235a-43f2-a551-051f163717f1-home\") pod \"sub-test-model-7daa14-kserve-5bfd95475f-29hmh\" (UID: \"5a2b3cd7-235a-43f2-a551-051f163717f1\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh"
Sep 10 03:14:59.109509 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:59.109489    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a2b3cd7-235a-43f2-a551-051f163717f1-model-cache\") pod \"sub-test-model-7daa14-kserve-5bfd95475f-29hmh\" (UID: \"5a2b3cd7-235a-43f2-a551-051f163717f1\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh"
Sep 10 03:14:59.109689 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:59.109656    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a2b3cd7-235a-43f2-a551-051f163717f1-tmp-dir\") pod \"sub-test-model-7daa14-kserve-5bfd95475f-29hmh\" (UID: \"5a2b3cd7-235a-43f2-a551-051f163717f1\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh"
Sep 10 03:14:59.109689 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:59.109671    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a2b3cd7-235a-43f2-a551-051f163717f1-home\") pod \"sub-test-model-7daa14-kserve-5bfd95475f-29hmh\" (UID: \"5a2b3cd7-235a-43f2-a551-051f163717f1\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh"
Sep 10 03:14:59.111434 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:59.111407    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a2b3cd7-235a-43f2-a551-051f163717f1-dshm\") pod \"sub-test-model-7daa14-kserve-5bfd95475f-29hmh\" (UID: \"5a2b3cd7-235a-43f2-a551-051f163717f1\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh"
Sep 10 03:14:59.111683 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:59.111667    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a2b3cd7-235a-43f2-a551-051f163717f1-tls-certs\") pod \"sub-test-model-7daa14-kserve-5bfd95475f-29hmh\" (UID: \"5a2b3cd7-235a-43f2-a551-051f163717f1\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh"
Sep 10 03:14:59.117762 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:59.117740    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-km9pq\" (UniqueName: \"kubernetes.io/projected/5a2b3cd7-235a-43f2-a551-051f163717f1-kube-api-access-km9pq\") pod \"sub-test-model-7daa14-kserve-5bfd95475f-29hmh\" (UID: \"5a2b3cd7-235a-43f2-a551-051f163717f1\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh"
Sep 10 03:14:59.236049 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:59.236027    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh"
Sep 10 03:14:59.509326 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:59.509299    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh"]
Sep 10 03:14:59.511174 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:14:59.511149    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a2b3cd7_235a_43f2_a551_051f163717f1.slice/crio-eb07472f16f82767905b6e92e9de4242930095ade48fb47573c87d68aa9d4e74 WatchSource:0}: Error finding container eb07472f16f82767905b6e92e9de4242930095ade48fb47573c87d68aa9d4e74: Status 404 returned error can't find the container with id eb07472f16f82767905b6e92e9de4242930095ade48fb47573c87d68aa9d4e74
Sep 10 03:14:59.614224 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:59.614194    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh" event={"ID":"5a2b3cd7-235a-43f2-a551-051f163717f1","Type":"ContainerStarted","Data":"65bd8ddd6b87024ba997a5f45f41a4f67edabe0450cfb3d36f514645023d02af"}
Sep 10 03:14:59.614332 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:59.614236    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh" event={"ID":"5a2b3cd7-235a-43f2-a551-051f163717f1","Type":"ContainerStarted","Data":"eb07472f16f82767905b6e92e9de4242930095ade48fb47573c87d68aa9d4e74"}
Sep 10 03:14:59.638564 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:14:59.638518    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh" podStartSLOduration=1.6385035220000002 podStartE2EDuration="1.638503522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:14:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:14:59.636488222 +0000 UTC m=+1690.011574142" watchObservedRunningTime="2026-09-10 03:14:59.638503522 +0000 UTC m=+1690.013589460"
Sep 10 03:15:00.141656 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:00.141625    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e6e6b8-29816835-xdn2r"]
Sep 10 03:15:00.145055 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:00.145037    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7daa14-29816835-xb9td"]
Sep 10 03:15:00.145186 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:00.145166    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e6e6b8-29816835-xdn2r"
Sep 10 03:15:00.147572 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:00.147554    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8jjl\""
Sep 10 03:15:00.148163 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:00.148144    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7daa14-29816835-xb9td"
Sep 10 03:15:00.151705 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:00.151681    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e6e6b8-29816835-xdn2r"]
Sep 10 03:15:00.156620 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:00.156478    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7daa14-29816835-xb9td"]
Sep 10 03:15:00.161185 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:00.161163    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e6e6b8-29816835-xdn2r"
Sep 10 03:15:00.170566 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:00.170545    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7daa14-29816835-xb9td"
Sep 10 03:15:00.456429 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:00.456394    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e6e6b8-29816835-xdn2r"]
Sep 10 03:15:00.457938 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:15:00.457908    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ead7e2_130a_4595_9b24_abb8ea674d40.slice/crio-a2d317799b720d46ca718452f9327cbb1ce1df3d0ab3534dda3d6bc6a8fde30c WatchSource:0}: Error finding container a2d317799b720d46ca718452f9327cbb1ce1df3d0ab3534dda3d6bc6a8fde30c: Status 404 returned error can't find the container with id a2d317799b720d46ca718452f9327cbb1ce1df3d0ab3534dda3d6bc6a8fde30c
Sep 10 03:15:00.471281 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:00.471259    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7daa14-29816835-xb9td"]
Sep 10 03:15:00.474911 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:15:00.474888    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31cc1407_9a71_49f2_b7f6_3bc16f3b14b7.slice/crio-d18bfe3b92d408e9b8e68e05f682f7e510fc0ed4523c3a84cfc49ccbeed4236d WatchSource:0}: Error finding container d18bfe3b92d408e9b8e68e05f682f7e510fc0ed4523c3a84cfc49ccbeed4236d: Status 404 returned error can't find the container with id d18bfe3b92d408e9b8e68e05f682f7e510fc0ed4523c3a84cfc49ccbeed4236d
Sep 10 03:15:00.622535 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:00.622504    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7daa14-29816835-xb9td" event={"ID":"31cc1407-9a71-49f2-b7f6-3bc16f3b14b7","Type":"ContainerStarted","Data":"d18bfe3b92d408e9b8e68e05f682f7e510fc0ed4523c3a84cfc49ccbeed4236d"}
Sep 10 03:15:00.623861 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:00.623836    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e6e6b8-29816835-xdn2r" event={"ID":"17ead7e2-130a-4595-9b24-abb8ea674d40","Type":"ContainerStarted","Data":"a2d317799b720d46ca718452f9327cbb1ce1df3d0ab3534dda3d6bc6a8fde30c"}
Sep 10 03:15:01.632414 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:01.632339    2558 generic.go:356] "Generic (PLEG): container finished" podID="17ead7e2-130a-4595-9b24-abb8ea674d40" containerID="da69f2c0d03efb46b2336c2c033b6116e8e4de14dc8eae3190cfe76615b4024d" exitCode=0
Sep 10 03:15:01.632766 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:01.632431    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e6e6b8-29816835-xdn2r" event={"ID":"17ead7e2-130a-4595-9b24-abb8ea674d40","Type":"ContainerDied","Data":"da69f2c0d03efb46b2336c2c033b6116e8e4de14dc8eae3190cfe76615b4024d"}
Sep 10 03:15:01.634063 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:01.634039    2558 generic.go:356] "Generic (PLEG): container finished" podID="31cc1407-9a71-49f2-b7f6-3bc16f3b14b7" containerID="5db4beec708fadc8c84cbdfdb10f720f008dc2f7f6024f0371b18e2ef977eeeb" exitCode=0
Sep 10 03:15:01.634199 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:01.634167    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7daa14-29816835-xb9td" event={"ID":"31cc1407-9a71-49f2-b7f6-3bc16f3b14b7","Type":"ContainerDied","Data":"5db4beec708fadc8c84cbdfdb10f720f008dc2f7f6024f0371b18e2ef977eeeb"}
Sep 10 03:15:02.779783 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:02.779751    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7daa14-29816835-xb9td"
Sep 10 03:15:02.810623 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:02.810599    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e6e6b8-29816835-xdn2r"
Sep 10 03:15:03.647734 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:03.647703    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7daa14-29816835-xb9td"
Sep 10 03:15:03.647913 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:03.647742    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7daa14-29816835-xb9td" event={"ID":"31cc1407-9a71-49f2-b7f6-3bc16f3b14b7","Type":"ContainerDied","Data":"d18bfe3b92d408e9b8e68e05f682f7e510fc0ed4523c3a84cfc49ccbeed4236d"}
Sep 10 03:15:03.647913 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:03.647797    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d18bfe3b92d408e9b8e68e05f682f7e510fc0ed4523c3a84cfc49ccbeed4236d"
Sep 10 03:15:03.649136 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:03.649112    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e6e6b8-29816835-xdn2r"
Sep 10 03:15:03.649136 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:03.649125    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e6e6b8-29816835-xdn2r" event={"ID":"17ead7e2-130a-4595-9b24-abb8ea674d40","Type":"ContainerDied","Data":"a2d317799b720d46ca718452f9327cbb1ce1df3d0ab3534dda3d6bc6a8fde30c"}
Sep 10 03:15:03.649335 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:03.649160    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2d317799b720d46ca718452f9327cbb1ce1df3d0ab3534dda3d6bc6a8fde30c"
Sep 10 03:15:09.236863 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:09.236826    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh"
Sep 10 03:15:09.237274 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:09.236901    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh"
Sep 10 03:15:09.249012 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:09.248990    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh"
Sep 10 03:15:09.703904 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:09.703875    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh"
Sep 10 03:15:21.583688 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:21.583656    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:15:21.690942 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:21.690906    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e6e6b8-9h2kw"]
Sep 10 03:15:21.691306 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:21.691290    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17ead7e2-130a-4595-9b24-abb8ea674d40" containerName="cleanup"
Sep 10 03:15:21.691306 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:21.691304    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="17ead7e2-130a-4595-9b24-abb8ea674d40" containerName="cleanup"
Sep 10 03:15:21.691466 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:21.691327    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31cc1407-9a71-49f2-b7f6-3bc16f3b14b7" containerName="cleanup"
Sep 10 03:15:21.691466 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:21.691332    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="31cc1407-9a71-49f2-b7f6-3bc16f3b14b7" containerName="cleanup"
Sep 10 03:15:21.691466 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:21.691394    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="17ead7e2-130a-4595-9b24-abb8ea674d40" containerName="cleanup"
Sep 10 03:15:21.691466 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:21.691403    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="31cc1407-9a71-49f2-b7f6-3bc16f3b14b7" containerName="cleanup"
Sep 10 03:15:21.694311 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:21.694292    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e6e6b8-9h2kw"
Sep 10 03:15:21.696343 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:21.696320    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8jjl\""
Sep 10 03:15:21.700914 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:21.700816    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e6e6b8-9h2kw"]
Sep 10 03:15:21.796722 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:21.796690    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c59542ed-dba2-4e16-9136-ad732d707c3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e6e6b8-9h2kw\" (UID: \"c59542ed-dba2-4e16-9136-ad732d707c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e6e6b8-9h2kw"
Sep 10 03:15:21.897303 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:21.897244    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c59542ed-dba2-4e16-9136-ad732d707c3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e6e6b8-9h2kw\" (UID: \"c59542ed-dba2-4e16-9136-ad732d707c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e6e6b8-9h2kw"
Sep 10 03:15:21.897909 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:21.897891    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c59542ed-dba2-4e16-9136-ad732d707c3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e6e6b8-9h2kw\" (UID: \"c59542ed-dba2-4e16-9136-ad732d707c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e6e6b8-9h2kw"
Sep 10 03:15:22.005937 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:22.005914    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e6e6b8-9h2kw"
Sep 10 03:15:22.277636 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:22.277608    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e6e6b8-9h2kw"]
Sep 10 03:15:22.278999 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:15:22.278974    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc59542ed_dba2_4e16_9136_ad732d707c3e.slice/crio-d133d8105cf7edcf3049ffb89e26c614df710bcafc4a992172d539ba3b2d25fe WatchSource:0}: Error finding container d133d8105cf7edcf3049ffb89e26c614df710bcafc4a992172d539ba3b2d25fe: Status 404 returned error can't find the container with id d133d8105cf7edcf3049ffb89e26c614df710bcafc4a992172d539ba3b2d25fe
Sep 10 03:15:22.770559 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:22.770513    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e6e6b8-9h2kw" event={"ID":"c59542ed-dba2-4e16-9136-ad732d707c3e","Type":"ContainerStarted","Data":"8de4e785f7d00287490060b7c755bd84a3c7a0de1dac60d983135a272e9dfaf8"}
Sep 10 03:15:22.770922 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:22.770586    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e6e6b8-9h2kw" event={"ID":"c59542ed-dba2-4e16-9136-ad732d707c3e","Type":"ContainerStarted","Data":"d133d8105cf7edcf3049ffb89e26c614df710bcafc4a992172d539ba3b2d25fe"}
Sep 10 03:15:22.785266 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:22.785221    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e6e6b8-9h2kw" podStartSLOduration=1.785206648 podStartE2EDuration="1.785206648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:15:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:15:22.784057804 +0000 UTC m=+1713.159143725" watchObservedRunningTime="2026-09-10 03:15:22.785206648 +0000 UTC m=+1713.160292629"
Sep 10 03:15:23.777455 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:23.777425    2558 generic.go:356] "Generic (PLEG): container finished" podID="c59542ed-dba2-4e16-9136-ad732d707c3e" containerID="8de4e785f7d00287490060b7c755bd84a3c7a0de1dac60d983135a272e9dfaf8" exitCode=0
Sep 10 03:15:23.777846 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:23.777493    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e6e6b8-9h2kw" event={"ID":"c59542ed-dba2-4e16-9136-ad732d707c3e","Type":"ContainerDied","Data":"8de4e785f7d00287490060b7c755bd84a3c7a0de1dac60d983135a272e9dfaf8"}
Sep 10 03:15:24.932691 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:24.932667    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e6e6b8-9h2kw"
Sep 10 03:15:25.021718 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:25.021692    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c59542ed-dba2-4e16-9136-ad732d707c3e-maas-api-service-ca\") pod \"c59542ed-dba2-4e16-9136-ad732d707c3e\" (UID: \"c59542ed-dba2-4e16-9136-ad732d707c3e\") "
Sep 10 03:15:25.022010 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:25.021988    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c59542ed-dba2-4e16-9136-ad732d707c3e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c59542ed-dba2-4e16-9136-ad732d707c3e" (UID: "c59542ed-dba2-4e16-9136-ad732d707c3e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:15:25.123045 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:25.122972    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c59542ed-dba2-4e16-9136-ad732d707c3e-maas-api-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:15:25.800536 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:25.800510    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e6e6b8-9h2kw"
Sep 10 03:15:25.800692 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:25.800500    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e6e6b8-9h2kw" event={"ID":"c59542ed-dba2-4e16-9136-ad732d707c3e","Type":"ContainerDied","Data":"d133d8105cf7edcf3049ffb89e26c614df710bcafc4a992172d539ba3b2d25fe"}
Sep 10 03:15:25.800692 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:25.800631    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d133d8105cf7edcf3049ffb89e26c614df710bcafc4a992172d539ba3b2d25fe"
Sep 10 03:15:26.562985 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:26.562940    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf"]
Sep 10 03:15:26.576828 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:26.576801    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2"]
Sep 10 03:15:26.595668 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:26.595644    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw"]
Sep 10 03:15:26.656683 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:26.656635    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:15:37.331256 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.331209    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e6e6b8-29816835-xdn2r"]
Sep 10 03:15:37.335300 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.335255    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e6e6b8-29816820-4xkmz"]
Sep 10 03:15:37.335460 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.335293    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:15:37.337998 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.337974    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e6e6b8-29816820-4xkmz"]
Sep 10 03:15:37.340531 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.340465    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e6e6b8-29816835-xdn2r"]
Sep 10 03:15:37.363124 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.358002    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e6e6b8-b8cf98d95-dtvhn"]
Sep 10 03:15:37.363124 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.358253    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-e6e6b8-b8cf98d95-dtvhn" podUID="9eb1f35e-640b-49f1-bd86-38a8d56b56a6" containerName="payload-processing" containerID="cri-o://6c5aadab4c633ce15da418359ebc680d79fa7d0003fd44eb2fca38917d9e1853" gracePeriod=30
Sep 10 03:15:37.363688 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.363620    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e6e6b8-7978b8f866-wxwzs"]
Sep 10 03:15:37.363874 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.363849    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e6e6b8-7978b8f866-wxwzs" podUID="bd55b6aa-8203-4025-97ed-0fa65ca98c73" containerName="payload-pre-processing" containerID="cri-o://e3d4684c53cef23f359f4ac32240164acfb83a68a6855d194a8bc5ebd25124f5" gracePeriod=30
Sep 10 03:15:37.399962 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.399941    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:15:37.500207 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.500179    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:15:37.523558 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.523529    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e6e6b8-9h2kw"]
Sep 10 03:15:37.529370 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.529346    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e6e6b8-9h2kw"]
Sep 10 03:15:37.602423 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.602358    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:15:37.642554 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.642534    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e6e6b8-b8cf98d95-dtvhn"
Sep 10 03:15:37.645457 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.645436    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e6e6b8-7978b8f866-wxwzs"
Sep 10 03:15:37.729710 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.729683    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znk2q\" (UniqueName: \"kubernetes.io/projected/bd55b6aa-8203-4025-97ed-0fa65ca98c73-kube-api-access-znk2q\") pod \"bd55b6aa-8203-4025-97ed-0fa65ca98c73\" (UID: \"bd55b6aa-8203-4025-97ed-0fa65ca98c73\") "
Sep 10 03:15:37.729859 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.729730    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd55b6aa-8203-4025-97ed-0fa65ca98c73-plugins-config-volume\") pod \"bd55b6aa-8203-4025-97ed-0fa65ca98c73\" (UID: \"bd55b6aa-8203-4025-97ed-0fa65ca98c73\") "
Sep 10 03:15:37.729859 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.729779    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6pmg\" (UniqueName: \"kubernetes.io/projected/9eb1f35e-640b-49f1-bd86-38a8d56b56a6-kube-api-access-k6pmg\") pod \"9eb1f35e-640b-49f1-bd86-38a8d56b56a6\" (UID: \"9eb1f35e-640b-49f1-bd86-38a8d56b56a6\") "
Sep 10 03:15:37.729859 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.729796    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb1f35e-640b-49f1-bd86-38a8d56b56a6-plugins-config-volume\") pod \"9eb1f35e-640b-49f1-bd86-38a8d56b56a6\" (UID: \"9eb1f35e-640b-49f1-bd86-38a8d56b56a6\") "
Sep 10 03:15:37.730165 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.730136    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd55b6aa-8203-4025-97ed-0fa65ca98c73-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bd55b6aa-8203-4025-97ed-0fa65ca98c73" (UID: "bd55b6aa-8203-4025-97ed-0fa65ca98c73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:15:37.730258 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.730188    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eb1f35e-640b-49f1-bd86-38a8d56b56a6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9eb1f35e-640b-49f1-bd86-38a8d56b56a6" (UID: "9eb1f35e-640b-49f1-bd86-38a8d56b56a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:15:37.731805 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.731784    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb1f35e-640b-49f1-bd86-38a8d56b56a6-kube-api-access-k6pmg" (OuterVolumeSpecName: "kube-api-access-k6pmg") pod "9eb1f35e-640b-49f1-bd86-38a8d56b56a6" (UID: "9eb1f35e-640b-49f1-bd86-38a8d56b56a6"). InnerVolumeSpecName "kube-api-access-k6pmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:15:37.731870 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.731826    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd55b6aa-8203-4025-97ed-0fa65ca98c73-kube-api-access-znk2q" (OuterVolumeSpecName: "kube-api-access-znk2q") pod "bd55b6aa-8203-4025-97ed-0fa65ca98c73" (UID: "bd55b6aa-8203-4025-97ed-0fa65ca98c73"). InnerVolumeSpecName "kube-api-access-znk2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:15:37.830816 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.830788    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znk2q\" (UniqueName: \"kubernetes.io/projected/bd55b6aa-8203-4025-97ed-0fa65ca98c73-kube-api-access-znk2q\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:15:37.830816 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.830814    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd55b6aa-8203-4025-97ed-0fa65ca98c73-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:15:37.830968 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.830829    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6pmg\" (UniqueName: \"kubernetes.io/projected/9eb1f35e-640b-49f1-bd86-38a8d56b56a6-kube-api-access-k6pmg\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:15:37.830968 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.830841    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb1f35e-640b-49f1-bd86-38a8d56b56a6-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:15:37.877453 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.877386    2558 generic.go:356] "Generic (PLEG): container finished" podID="bd55b6aa-8203-4025-97ed-0fa65ca98c73" containerID="e3d4684c53cef23f359f4ac32240164acfb83a68a6855d194a8bc5ebd25124f5" exitCode=0
Sep 10 03:15:37.877453 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.877429    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e6e6b8-7978b8f866-wxwzs"
Sep 10 03:15:37.877596 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.877468    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e6e6b8-7978b8f866-wxwzs" event={"ID":"bd55b6aa-8203-4025-97ed-0fa65ca98c73","Type":"ContainerDied","Data":"e3d4684c53cef23f359f4ac32240164acfb83a68a6855d194a8bc5ebd25124f5"}
Sep 10 03:15:37.877596 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.877524    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e6e6b8-7978b8f866-wxwzs" event={"ID":"bd55b6aa-8203-4025-97ed-0fa65ca98c73","Type":"ContainerDied","Data":"ac9b767bde29536c5f3b44813f1bbd85955906bd786115b29e19818f68131b69"}
Sep 10 03:15:37.877596 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.877552    2558 scope.go:117] "RemoveContainer" containerID="e3d4684c53cef23f359f4ac32240164acfb83a68a6855d194a8bc5ebd25124f5"
Sep 10 03:15:37.879029 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.879005    2558 generic.go:356] "Generic (PLEG): container finished" podID="9eb1f35e-640b-49f1-bd86-38a8d56b56a6" containerID="6c5aadab4c633ce15da418359ebc680d79fa7d0003fd44eb2fca38917d9e1853" exitCode=0
Sep 10 03:15:37.879176 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.879040    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e6e6b8-b8cf98d95-dtvhn"
Sep 10 03:15:37.879176 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.879081    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e6e6b8-b8cf98d95-dtvhn" event={"ID":"9eb1f35e-640b-49f1-bd86-38a8d56b56a6","Type":"ContainerDied","Data":"6c5aadab4c633ce15da418359ebc680d79fa7d0003fd44eb2fca38917d9e1853"}
Sep 10 03:15:37.879303 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.879208    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e6e6b8-b8cf98d95-dtvhn" event={"ID":"9eb1f35e-640b-49f1-bd86-38a8d56b56a6","Type":"ContainerDied","Data":"e8e93d47a8a37412cbeff44819322a89e76f9c581762878af506b5a5722e5253"}
Sep 10 03:15:37.887983 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.887963    2558 scope.go:117] "RemoveContainer" containerID="e3d4684c53cef23f359f4ac32240164acfb83a68a6855d194a8bc5ebd25124f5"
Sep 10 03:15:37.888270 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:15:37.888246    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3d4684c53cef23f359f4ac32240164acfb83a68a6855d194a8bc5ebd25124f5\": container with ID starting with e3d4684c53cef23f359f4ac32240164acfb83a68a6855d194a8bc5ebd25124f5 not found: ID does not exist" containerID="e3d4684c53cef23f359f4ac32240164acfb83a68a6855d194a8bc5ebd25124f5"
Sep 10 03:15:37.888367 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.888281    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3d4684c53cef23f359f4ac32240164acfb83a68a6855d194a8bc5ebd25124f5"} err="failed to get container status \"e3d4684c53cef23f359f4ac32240164acfb83a68a6855d194a8bc5ebd25124f5\": rpc error: code = NotFound desc = could not find container \"e3d4684c53cef23f359f4ac32240164acfb83a68a6855d194a8bc5ebd25124f5\": container with ID starting with e3d4684c53cef23f359f4ac32240164acfb83a68a6855d194a8bc5ebd25124f5 not found: ID does not exist"
Sep 10 03:15:37.888367 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.888306    2558 scope.go:117] "RemoveContainer" containerID="6c5aadab4c633ce15da418359ebc680d79fa7d0003fd44eb2fca38917d9e1853"
Sep 10 03:15:37.889439 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.889412    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:15:37.896796 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.896779    2558 scope.go:117] "RemoveContainer" containerID="6c5aadab4c633ce15da418359ebc680d79fa7d0003fd44eb2fca38917d9e1853"
Sep 10 03:15:37.897047 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:15:37.897028    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c5aadab4c633ce15da418359ebc680d79fa7d0003fd44eb2fca38917d9e1853\": container with ID starting with 6c5aadab4c633ce15da418359ebc680d79fa7d0003fd44eb2fca38917d9e1853 not found: ID does not exist" containerID="6c5aadab4c633ce15da418359ebc680d79fa7d0003fd44eb2fca38917d9e1853"
Sep 10 03:15:37.897092 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.897053    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c5aadab4c633ce15da418359ebc680d79fa7d0003fd44eb2fca38917d9e1853"} err="failed to get container status \"6c5aadab4c633ce15da418359ebc680d79fa7d0003fd44eb2fca38917d9e1853\": rpc error: code = NotFound desc = could not find container \"6c5aadab4c633ce15da418359ebc680d79fa7d0003fd44eb2fca38917d9e1853\": container with ID starting with 6c5aadab4c633ce15da418359ebc680d79fa7d0003fd44eb2fca38917d9e1853 not found: ID does not exist"
Sep 10 03:15:37.901957 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.901937    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e6e6b8-7978b8f866-wxwzs"]
Sep 10 03:15:37.905362 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.905343    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e6e6b8-7978b8f866-wxwzs"]
Sep 10 03:15:37.914501 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.914479    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e6e6b8-b8cf98d95-dtvhn"]
Sep 10 03:15:37.918141 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:37.918123    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e6e6b8-b8cf98d95-dtvhn"]
Sep 10 03:15:38.071120 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:38.071067    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17ead7e2-130a-4595-9b24-abb8ea674d40" path="/var/lib/kubelet/pods/17ead7e2-130a-4595-9b24-abb8ea674d40/volumes"
Sep 10 03:15:38.073177 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:38.071602    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9eb1f35e-640b-49f1-bd86-38a8d56b56a6" path="/var/lib/kubelet/pods/9eb1f35e-640b-49f1-bd86-38a8d56b56a6/volumes"
Sep 10 03:15:38.073177 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:38.072133    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd55b6aa-8203-4025-97ed-0fa65ca98c73" path="/var/lib/kubelet/pods/bd55b6aa-8203-4025-97ed-0fa65ca98c73/volumes"
Sep 10 03:15:38.073177 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:38.072597    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c59542ed-dba2-4e16-9136-ad732d707c3e" path="/var/lib/kubelet/pods/c59542ed-dba2-4e16-9136-ad732d707c3e/volumes"
Sep 10 03:15:38.075240 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:38.073622    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c73eee88-4bcf-4c52-9676-1d6b8c3e8557" path="/var/lib/kubelet/pods/c73eee88-4bcf-4c52-9676-1d6b8c3e8557/volumes"
Sep 10 03:15:41.583386 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.583344    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf" podUID="90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4" containerName="main" containerID="cri-o://fc28b82c0b9adfdd4963bbc542f6fd800a2537e06d35142962332789937f1214" gracePeriod=45
Sep 10 03:15:41.596197 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.596166    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2" podUID="5b74f9d2-9e8c-478f-9f38-1fef910c943c" containerName="main" containerID="cri-o://79ba55ebcfba69e042beefd8f8d74ec5beacfcd1facbf5734f47bb19f0d50d17" gracePeriod=45
Sep 10 03:15:41.614153 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.614123    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw" podUID="19549650-bd12-4ea4-83b6-30df64ab2587" containerName="main" containerID="cri-o://4e3f382c3f1323da3130efdab465987dc8916b15df1b36b7406c2956a83e6f94" gracePeriod=45
Sep 10 03:15:41.903283 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.903257    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf"
Sep 10 03:15:41.906345 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.906320    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2"
Sep 10 03:15:41.906980 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.906954    2558 generic.go:356] "Generic (PLEG): container finished" podID="5b74f9d2-9e8c-478f-9f38-1fef910c943c" containerID="79ba55ebcfba69e042beefd8f8d74ec5beacfcd1facbf5734f47bb19f0d50d17" exitCode=0
Sep 10 03:15:41.907082 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.907039    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2" event={"ID":"5b74f9d2-9e8c-478f-9f38-1fef910c943c","Type":"ContainerDied","Data":"79ba55ebcfba69e042beefd8f8d74ec5beacfcd1facbf5734f47bb19f0d50d17"}
Sep 10 03:15:41.907182 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.907094    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2" event={"ID":"5b74f9d2-9e8c-478f-9f38-1fef910c943c","Type":"ContainerDied","Data":"827fddedfda15bf924b1ae6f99acdcdb3d6e8c07ac51bba6df793aad4f8486be"}
Sep 10 03:15:41.907182 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.907138    2558 scope.go:117] "RemoveContainer" containerID="79ba55ebcfba69e042beefd8f8d74ec5beacfcd1facbf5734f47bb19f0d50d17"
Sep 10 03:15:41.908717 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.908550    2558 generic.go:356] "Generic (PLEG): container finished" podID="90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4" containerID="fc28b82c0b9adfdd4963bbc542f6fd800a2537e06d35142962332789937f1214" exitCode=0
Sep 10 03:15:41.908717 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.908578    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf" event={"ID":"90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4","Type":"ContainerDied","Data":"fc28b82c0b9adfdd4963bbc542f6fd800a2537e06d35142962332789937f1214"}
Sep 10 03:15:41.908717 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.908596    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf"
Sep 10 03:15:41.908717 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.908625    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf" event={"ID":"90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4","Type":"ContainerDied","Data":"5ccd6bb1024cdcc0d19922661ac701f44fda5fc27183e08ec72c73d182c6ee95"}
Sep 10 03:15:41.909917 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.909898    2558 generic.go:356] "Generic (PLEG): container finished" podID="19549650-bd12-4ea4-83b6-30df64ab2587" containerID="4e3f382c3f1323da3130efdab465987dc8916b15df1b36b7406c2956a83e6f94" exitCode=0
Sep 10 03:15:41.910014 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.909961    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw" event={"ID":"19549650-bd12-4ea4-83b6-30df64ab2587","Type":"ContainerDied","Data":"4e3f382c3f1323da3130efdab465987dc8916b15df1b36b7406c2956a83e6f94"}
Sep 10 03:15:41.910014 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.909992    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw" event={"ID":"19549650-bd12-4ea4-83b6-30df64ab2587","Type":"ContainerDied","Data":"1064179d3b86a6edf9ef9147eff782db1eec36c8ad5a198aeff017d4f188828c"}
Sep 10 03:15:41.910014 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.910008    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1064179d3b86a6edf9ef9147eff782db1eec36c8ad5a198aeff017d4f188828c"
Sep 10 03:15:41.910430 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.910413    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw"
Sep 10 03:15:41.916271 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.916255    2558 scope.go:117] "RemoveContainer" containerID="79ba55ebcfba69e042beefd8f8d74ec5beacfcd1facbf5734f47bb19f0d50d17"
Sep 10 03:15:41.916510 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:15:41.916493    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79ba55ebcfba69e042beefd8f8d74ec5beacfcd1facbf5734f47bb19f0d50d17\": container with ID starting with 79ba55ebcfba69e042beefd8f8d74ec5beacfcd1facbf5734f47bb19f0d50d17 not found: ID does not exist" containerID="79ba55ebcfba69e042beefd8f8d74ec5beacfcd1facbf5734f47bb19f0d50d17"
Sep 10 03:15:41.916572 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.916526    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79ba55ebcfba69e042beefd8f8d74ec5beacfcd1facbf5734f47bb19f0d50d17"} err="failed to get container status \"79ba55ebcfba69e042beefd8f8d74ec5beacfcd1facbf5734f47bb19f0d50d17\": rpc error: code = NotFound desc = could not find container \"79ba55ebcfba69e042beefd8f8d74ec5beacfcd1facbf5734f47bb19f0d50d17\": container with ID starting with 79ba55ebcfba69e042beefd8f8d74ec5beacfcd1facbf5734f47bb19f0d50d17 not found: ID does not exist"
Sep 10 03:15:41.916572 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.916543    2558 scope.go:117] "RemoveContainer" containerID="fc28b82c0b9adfdd4963bbc542f6fd800a2537e06d35142962332789937f1214"
Sep 10 03:15:41.926062 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.926047    2558 scope.go:117] "RemoveContainer" containerID="fc28b82c0b9adfdd4963bbc542f6fd800a2537e06d35142962332789937f1214"
Sep 10 03:15:41.926404 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:15:41.926365    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc28b82c0b9adfdd4963bbc542f6fd800a2537e06d35142962332789937f1214\": container with ID starting with fc28b82c0b9adfdd4963bbc542f6fd800a2537e06d35142962332789937f1214 not found: ID does not exist" containerID="fc28b82c0b9adfdd4963bbc542f6fd800a2537e06d35142962332789937f1214"
Sep 10 03:15:41.926477 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.926413    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc28b82c0b9adfdd4963bbc542f6fd800a2537e06d35142962332789937f1214"} err="failed to get container status \"fc28b82c0b9adfdd4963bbc542f6fd800a2537e06d35142962332789937f1214\": rpc error: code = NotFound desc = could not find container \"fc28b82c0b9adfdd4963bbc542f6fd800a2537e06d35142962332789937f1214\": container with ID starting with fc28b82c0b9adfdd4963bbc542f6fd800a2537e06d35142962332789937f1214 not found: ID does not exist"
Sep 10 03:15:41.962785 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.962744    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19549650-bd12-4ea4-83b6-30df64ab2587-tmp-dir\") pod \"19549650-bd12-4ea4-83b6-30df64ab2587\" (UID: \"19549650-bd12-4ea4-83b6-30df64ab2587\") "
Sep 10 03:15:41.962785 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.962783    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19549650-bd12-4ea4-83b6-30df64ab2587-dshm\") pod \"19549650-bd12-4ea4-83b6-30df64ab2587\" (UID: \"19549650-bd12-4ea4-83b6-30df64ab2587\") "
Sep 10 03:15:41.963016 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.962851    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-tls-certs\") pod \"90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4\" (UID: \"90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4\") "
Sep 10 03:15:41.963016 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.962876    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stmjp\" (UniqueName: \"kubernetes.io/projected/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-kube-api-access-stmjp\") pod \"90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4\" (UID: \"90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4\") "
Sep 10 03:15:41.963016 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.962904    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b74f9d2-9e8c-478f-9f38-1fef910c943c-tmp-dir\") pod \"5b74f9d2-9e8c-478f-9f38-1fef910c943c\" (UID: \"5b74f9d2-9e8c-478f-9f38-1fef910c943c\") "
Sep 10 03:15:41.963016 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.962929    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b74f9d2-9e8c-478f-9f38-1fef910c943c-dshm\") pod \"5b74f9d2-9e8c-478f-9f38-1fef910c943c\" (UID: \"5b74f9d2-9e8c-478f-9f38-1fef910c943c\") "
Sep 10 03:15:41.963016 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.962968    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b74f9d2-9e8c-478f-9f38-1fef910c943c-model-cache\") pod \"5b74f9d2-9e8c-478f-9f38-1fef910c943c\" (UID: \"5b74f9d2-9e8c-478f-9f38-1fef910c943c\") "
Sep 10 03:15:41.963016 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.962994    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-model-cache\") pod \"90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4\" (UID: \"90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4\") "
Sep 10 03:15:41.963453 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.963033    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19549650-bd12-4ea4-83b6-30df64ab2587-tls-certs\") pod \"19549650-bd12-4ea4-83b6-30df64ab2587\" (UID: \"19549650-bd12-4ea4-83b6-30df64ab2587\") "
Sep 10 03:15:41.963453 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.963055    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b74f9d2-9e8c-478f-9f38-1fef910c943c-home\") pod \"5b74f9d2-9e8c-478f-9f38-1fef910c943c\" (UID: \"5b74f9d2-9e8c-478f-9f38-1fef910c943c\") "
Sep 10 03:15:41.963453 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.963093    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-tmp-dir\") pod \"90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4\" (UID: \"90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4\") "
Sep 10 03:15:41.963453 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.963136    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-dshm\") pod \"90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4\" (UID: \"90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4\") "
Sep 10 03:15:41.963453 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.963154    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-home\") pod \"90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4\" (UID: \"90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4\") "
Sep 10 03:15:41.963453 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.963174    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19549650-bd12-4ea4-83b6-30df64ab2587-home\") pod \"19549650-bd12-4ea4-83b6-30df64ab2587\" (UID: \"19549650-bd12-4ea4-83b6-30df64ab2587\") "
Sep 10 03:15:41.963453 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.963205    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19549650-bd12-4ea4-83b6-30df64ab2587-model-cache\") pod \"19549650-bd12-4ea4-83b6-30df64ab2587\" (UID: \"19549650-bd12-4ea4-83b6-30df64ab2587\") "
Sep 10 03:15:41.963453 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.963236    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwjb8\" (UniqueName: \"kubernetes.io/projected/5b74f9d2-9e8c-478f-9f38-1fef910c943c-kube-api-access-jwjb8\") pod \"5b74f9d2-9e8c-478f-9f38-1fef910c943c\" (UID: \"5b74f9d2-9e8c-478f-9f38-1fef910c943c\") "
Sep 10 03:15:41.963453 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.963269    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gp7l\" (UniqueName: \"kubernetes.io/projected/19549650-bd12-4ea4-83b6-30df64ab2587-kube-api-access-8gp7l\") pod \"19549650-bd12-4ea4-83b6-30df64ab2587\" (UID: \"19549650-bd12-4ea4-83b6-30df64ab2587\") "
Sep 10 03:15:41.963453 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.963283    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b74f9d2-9e8c-478f-9f38-1fef910c943c-tls-certs\") pod \"5b74f9d2-9e8c-478f-9f38-1fef910c943c\" (UID: \"5b74f9d2-9e8c-478f-9f38-1fef910c943c\") "
Sep 10 03:15:41.964282 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.964235    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b74f9d2-9e8c-478f-9f38-1fef910c943c-home" (OuterVolumeSpecName: "home") pod "5b74f9d2-9e8c-478f-9f38-1fef910c943c" (UID: "5b74f9d2-9e8c-478f-9f38-1fef910c943c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:15:41.965357 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.964460    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4" (UID: "90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:15:41.965357 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.964641    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19549650-bd12-4ea4-83b6-30df64ab2587-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "19549650-bd12-4ea4-83b6-30df64ab2587" (UID: "19549650-bd12-4ea4-83b6-30df64ab2587"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:15:41.965357 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.965068    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b74f9d2-9e8c-478f-9f38-1fef910c943c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5b74f9d2-9e8c-478f-9f38-1fef910c943c" (UID: "5b74f9d2-9e8c-478f-9f38-1fef910c943c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:15:41.966893 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.966232    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-home" (OuterVolumeSpecName: "home") pod "90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4" (UID: "90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:15:41.966893 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.966297    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19549650-bd12-4ea4-83b6-30df64ab2587-model-cache" (OuterVolumeSpecName: "model-cache") pod "19549650-bd12-4ea4-83b6-30df64ab2587" (UID: "19549650-bd12-4ea4-83b6-30df64ab2587"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:15:41.966893 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.966340    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19549650-bd12-4ea4-83b6-30df64ab2587-home" (OuterVolumeSpecName: "home") pod "19549650-bd12-4ea4-83b6-30df64ab2587" (UID: "19549650-bd12-4ea4-83b6-30df64ab2587"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:15:41.966893 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.966601    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-model-cache" (OuterVolumeSpecName: "model-cache") pod "90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4" (UID: "90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:15:41.967731 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.967472    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b74f9d2-9e8c-478f-9f38-1fef910c943c-model-cache" (OuterVolumeSpecName: "model-cache") pod "5b74f9d2-9e8c-478f-9f38-1fef910c943c" (UID: "5b74f9d2-9e8c-478f-9f38-1fef910c943c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:15:41.967937 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.967891    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b74f9d2-9e8c-478f-9f38-1fef910c943c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b74f9d2-9e8c-478f-9f38-1fef910c943c" (UID: "5b74f9d2-9e8c-478f-9f38-1fef910c943c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:15:41.970817 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.970692    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b74f9d2-9e8c-478f-9f38-1fef910c943c-kube-api-access-jwjb8" (OuterVolumeSpecName: "kube-api-access-jwjb8") pod "5b74f9d2-9e8c-478f-9f38-1fef910c943c" (UID: "5b74f9d2-9e8c-478f-9f38-1fef910c943c"). InnerVolumeSpecName "kube-api-access-jwjb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:15:41.970817 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.970728    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-kube-api-access-stmjp" (OuterVolumeSpecName: "kube-api-access-stmjp") pod "90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4" (UID: "90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4"). InnerVolumeSpecName "kube-api-access-stmjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:15:41.970817 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.970784    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-dshm" (OuterVolumeSpecName: "dshm") pod "90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4" (UID: "90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:15:41.970817 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.970810    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19549650-bd12-4ea4-83b6-30df64ab2587-dshm" (OuterVolumeSpecName: "dshm") pod "19549650-bd12-4ea4-83b6-30df64ab2587" (UID: "19549650-bd12-4ea4-83b6-30df64ab2587"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:15:41.972319 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.972292    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b74f9d2-9e8c-478f-9f38-1fef910c943c-dshm" (OuterVolumeSpecName: "dshm") pod "5b74f9d2-9e8c-478f-9f38-1fef910c943c" (UID: "5b74f9d2-9e8c-478f-9f38-1fef910c943c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:15:41.972884 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.972787    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4" (UID: "90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:15:41.973513 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.973486    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19549650-bd12-4ea4-83b6-30df64ab2587-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "19549650-bd12-4ea4-83b6-30df64ab2587" (UID: "19549650-bd12-4ea4-83b6-30df64ab2587"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:15:41.973703 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:41.973682    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19549650-bd12-4ea4-83b6-30df64ab2587-kube-api-access-8gp7l" (OuterVolumeSpecName: "kube-api-access-8gp7l") pod "19549650-bd12-4ea4-83b6-30df64ab2587" (UID: "19549650-bd12-4ea4-83b6-30df64ab2587"). InnerVolumeSpecName "kube-api-access-8gp7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:15:42.064411 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:42.064387    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gp7l\" (UniqueName: \"kubernetes.io/projected/19549650-bd12-4ea4-83b6-30df64ab2587-kube-api-access-8gp7l\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.064411 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:42.064410    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b74f9d2-9e8c-478f-9f38-1fef910c943c-tls-certs\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.064550 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:42.064420    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19549650-bd12-4ea4-83b6-30df64ab2587-tmp-dir\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.064550 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:42.064428    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19549650-bd12-4ea4-83b6-30df64ab2587-dshm\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.064550 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:42.064435    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-tls-certs\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.064550 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:42.064443    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stmjp\" (UniqueName: \"kubernetes.io/projected/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-kube-api-access-stmjp\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.064550 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:42.064453    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b74f9d2-9e8c-478f-9f38-1fef910c943c-tmp-dir\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.064550 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:42.064462    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b74f9d2-9e8c-478f-9f38-1fef910c943c-dshm\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.064550 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:42.064469    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b74f9d2-9e8c-478f-9f38-1fef910c943c-model-cache\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.064550 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:42.064476    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-model-cache\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.064550 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:42.064484    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19549650-bd12-4ea4-83b6-30df64ab2587-tls-certs\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.064550 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:42.064491    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b74f9d2-9e8c-478f-9f38-1fef910c943c-home\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.064550 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:42.064498    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-tmp-dir\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.064550 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:42.064505    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-dshm\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.064550 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:42.064511    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4-home\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.064550 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:42.064518    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19549650-bd12-4ea4-83b6-30df64ab2587-home\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.064550 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:42.064524    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19549650-bd12-4ea4-83b6-30df64ab2587-model-cache\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.064550 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:42.064532    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwjb8\" (UniqueName: \"kubernetes.io/projected/5b74f9d2-9e8c-478f-9f38-1fef910c943c-kube-api-access-jwjb8\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:15:42.224023 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:42.223996    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf"]
Sep 10 03:15:42.229904 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:42.229883    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/distinct-2-w4-e6e6b8-kserve-757b4df47d-6nllf"]
Sep 10 03:15:42.918085 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:42.918010    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw"
Sep 10 03:15:42.918508 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:42.918012    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2"
Sep 10 03:15:42.932654 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:42.932630    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw"]
Sep 10 03:15:42.936452 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:42.936428    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/simulator-w4-e6e6b8-kserve-bdfbb885f-5gmpw"]
Sep 10 03:15:42.944272 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:42.944253    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2"]
Sep 10 03:15:42.947826 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:42.947807    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e6e6b8/embedding-w4-e6e6b8-kserve-7cf8b9f694-ksln2"]
Sep 10 03:15:43.777782 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:43.777743    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:15:43.842006 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:43.841971    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:15:43.916562 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:43.916534    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:15:44.069506 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:44.069439    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19549650-bd12-4ea4-83b6-30df64ab2587" path="/var/lib/kubelet/pods/19549650-bd12-4ea4-83b6-30df64ab2587/volumes"
Sep 10 03:15:44.069838 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:44.069805    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b74f9d2-9e8c-478f-9f38-1fef910c943c" path="/var/lib/kubelet/pods/5b74f9d2-9e8c-478f-9f38-1fef910c943c/volumes"
Sep 10 03:15:44.070157 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:44.070143    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4" path="/var/lib/kubelet/pods/90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4/volumes"
Sep 10 03:15:50.449537 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:50.449507    2558 scope.go:117] "RemoveContainer" containerID="4e3f382c3f1323da3130efdab465987dc8916b15df1b36b7406c2956a83e6f94"
Sep 10 03:15:50.459418 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:50.459401    2558 scope.go:117] "RemoveContainer" containerID="eecb95d629d987d38267fba13f113cd709a880466b20f54452e3fc9375f2e663"
Sep 10 03:15:50.467590 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:15:50.467555    2558 scope.go:117] "RemoveContainer" containerID="d83240527519750e6bdcfdc15aa8e40cdb04827de1ff7b5dc6f41715805af3a8"
Sep 10 03:16:03.288786 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:03.288744    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:16:03.388046 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:03.388011    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:16:03.895015 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:03.894983    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:16:08.943311 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:08.943275    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd"]
Sep 10 03:16:09.002699 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:09.002665    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:16:09.681143 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:09.681094    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:16:10.718839 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:10.718804    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh"]
Sep 10 03:16:10.737624 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:10.737601    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:16:15.245740 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:15.245704    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:16:17.268176 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:17.268146    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:16:23.966990 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:23.966928    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd" podUID="34cfdfdb-8ce5-4095-96a1-4d7152d99030" containerName="main" containerID="cri-o://bd70b2ecf71809ed3bdf06f04bf7bc4587b4653b2ff42bd2de3de2f5ad7d3cbe" gracePeriod=45
Sep 10 03:16:24.056764 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:24.056727    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:16:24.164942 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:24.164911    2558 generic.go:356] "Generic (PLEG): container finished" podID="34cfdfdb-8ce5-4095-96a1-4d7152d99030" containerID="bd70b2ecf71809ed3bdf06f04bf7bc4587b4653b2ff42bd2de3de2f5ad7d3cbe" exitCode=0
Sep 10 03:16:24.165144 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:24.164990    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd" event={"ID":"34cfdfdb-8ce5-4095-96a1-4d7152d99030","Type":"ContainerDied","Data":"bd70b2ecf71809ed3bdf06f04bf7bc4587b4653b2ff42bd2de3de2f5ad7d3cbe"}
Sep 10 03:16:24.214022 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:24.213998    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd"
Sep 10 03:16:24.404269 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:24.404186    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34cfdfdb-8ce5-4095-96a1-4d7152d99030-model-cache\") pod \"34cfdfdb-8ce5-4095-96a1-4d7152d99030\" (UID: \"34cfdfdb-8ce5-4095-96a1-4d7152d99030\") "
Sep 10 03:16:24.404269 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:24.404227    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34cfdfdb-8ce5-4095-96a1-4d7152d99030-tls-certs\") pod \"34cfdfdb-8ce5-4095-96a1-4d7152d99030\" (UID: \"34cfdfdb-8ce5-4095-96a1-4d7152d99030\") "
Sep 10 03:16:24.404269 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:24.404255    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34cfdfdb-8ce5-4095-96a1-4d7152d99030-home\") pod \"34cfdfdb-8ce5-4095-96a1-4d7152d99030\" (UID: \"34cfdfdb-8ce5-4095-96a1-4d7152d99030\") "
Sep 10 03:16:24.404564 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:24.404315    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34cfdfdb-8ce5-4095-96a1-4d7152d99030-dshm\") pod \"34cfdfdb-8ce5-4095-96a1-4d7152d99030\" (UID: \"34cfdfdb-8ce5-4095-96a1-4d7152d99030\") "
Sep 10 03:16:24.404564 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:24.404343    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2wnv\" (UniqueName: \"kubernetes.io/projected/34cfdfdb-8ce5-4095-96a1-4d7152d99030-kube-api-access-s2wnv\") pod \"34cfdfdb-8ce5-4095-96a1-4d7152d99030\" (UID: \"34cfdfdb-8ce5-4095-96a1-4d7152d99030\") "
Sep 10 03:16:24.404564 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:24.404395    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34cfdfdb-8ce5-4095-96a1-4d7152d99030-tmp-dir\") pod \"34cfdfdb-8ce5-4095-96a1-4d7152d99030\" (UID: \"34cfdfdb-8ce5-4095-96a1-4d7152d99030\") "
Sep 10 03:16:24.404564 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:24.404502    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34cfdfdb-8ce5-4095-96a1-4d7152d99030-model-cache" (OuterVolumeSpecName: "model-cache") pod "34cfdfdb-8ce5-4095-96a1-4d7152d99030" (UID: "34cfdfdb-8ce5-4095-96a1-4d7152d99030"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:16:24.404763 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:24.404558    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34cfdfdb-8ce5-4095-96a1-4d7152d99030-home" (OuterVolumeSpecName: "home") pod "34cfdfdb-8ce5-4095-96a1-4d7152d99030" (UID: "34cfdfdb-8ce5-4095-96a1-4d7152d99030"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:16:24.404763 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:24.404677    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34cfdfdb-8ce5-4095-96a1-4d7152d99030-model-cache\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:16:24.404763 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:24.404670    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34cfdfdb-8ce5-4095-96a1-4d7152d99030-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "34cfdfdb-8ce5-4095-96a1-4d7152d99030" (UID: "34cfdfdb-8ce5-4095-96a1-4d7152d99030"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:16:24.404763 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:24.404694    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34cfdfdb-8ce5-4095-96a1-4d7152d99030-home\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:16:24.406677 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:24.406653    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34cfdfdb-8ce5-4095-96a1-4d7152d99030-kube-api-access-s2wnv" (OuterVolumeSpecName: "kube-api-access-s2wnv") pod "34cfdfdb-8ce5-4095-96a1-4d7152d99030" (UID: "34cfdfdb-8ce5-4095-96a1-4d7152d99030"). InnerVolumeSpecName "kube-api-access-s2wnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:16:24.406804 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:24.406786    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34cfdfdb-8ce5-4095-96a1-4d7152d99030-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "34cfdfdb-8ce5-4095-96a1-4d7152d99030" (UID: "34cfdfdb-8ce5-4095-96a1-4d7152d99030"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:16:24.406804 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:24.406794    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34cfdfdb-8ce5-4095-96a1-4d7152d99030-dshm" (OuterVolumeSpecName: "dshm") pod "34cfdfdb-8ce5-4095-96a1-4d7152d99030" (UID: "34cfdfdb-8ce5-4095-96a1-4d7152d99030"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:16:24.505458 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:24.505432    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34cfdfdb-8ce5-4095-96a1-4d7152d99030-dshm\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:16:24.505614 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:24.505461    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2wnv\" (UniqueName: \"kubernetes.io/projected/34cfdfdb-8ce5-4095-96a1-4d7152d99030-kube-api-access-s2wnv\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:16:24.505614 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:24.505476    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34cfdfdb-8ce5-4095-96a1-4d7152d99030-tmp-dir\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:16:24.505614 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:24.505488    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34cfdfdb-8ce5-4095-96a1-4d7152d99030-tls-certs\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:16:25.172089 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:25.172059    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd"
Sep 10 03:16:25.172645 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:25.172088    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd" event={"ID":"34cfdfdb-8ce5-4095-96a1-4d7152d99030","Type":"ContainerDied","Data":"9decc322b6dd8220c49f078a5098cefffa51d72361242e98ad292402e8c1423d"}
Sep 10 03:16:25.172645 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:25.172151    2558 scope.go:117] "RemoveContainer" containerID="bd70b2ecf71809ed3bdf06f04bf7bc4587b4653b2ff42bd2de3de2f5ad7d3cbe"
Sep 10 03:16:25.193073 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:25.193045    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd"]
Sep 10 03:16:25.198824 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:25.198803    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7727/sub-test-model-af7727-kserve-7b7944f8c5-n4wdd"]
Sep 10 03:16:25.738162 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:25.738089    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh" podUID="5a2b3cd7-235a-43f2-a551-051f163717f1" containerName="main" containerID="cri-o://65bd8ddd6b87024ba997a5f45f41a4f67edabe0450cfb3d36f514645023d02af" gracePeriod=45
Sep 10 03:16:25.996856 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:25.996800    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh"
Sep 10 03:16:26.069034 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:26.068997    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34cfdfdb-8ce5-4095-96a1-4d7152d99030" path="/var/lib/kubelet/pods/34cfdfdb-8ce5-4095-96a1-4d7152d99030/volumes"
Sep 10 03:16:26.117235 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:26.117214    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a2b3cd7-235a-43f2-a551-051f163717f1-tls-certs\") pod \"5a2b3cd7-235a-43f2-a551-051f163717f1\" (UID: \"5a2b3cd7-235a-43f2-a551-051f163717f1\") "
Sep 10 03:16:26.117353 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:26.117252    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a2b3cd7-235a-43f2-a551-051f163717f1-model-cache\") pod \"5a2b3cd7-235a-43f2-a551-051f163717f1\" (UID: \"5a2b3cd7-235a-43f2-a551-051f163717f1\") "
Sep 10 03:16:26.117353 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:26.117286    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a2b3cd7-235a-43f2-a551-051f163717f1-home\") pod \"5a2b3cd7-235a-43f2-a551-051f163717f1\" (UID: \"5a2b3cd7-235a-43f2-a551-051f163717f1\") "
Sep 10 03:16:26.117353 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:26.117307    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a2b3cd7-235a-43f2-a551-051f163717f1-tmp-dir\") pod \"5a2b3cd7-235a-43f2-a551-051f163717f1\" (UID: \"5a2b3cd7-235a-43f2-a551-051f163717f1\") "
Sep 10 03:16:26.117353 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:26.117331    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-km9pq\" (UniqueName: \"kubernetes.io/projected/5a2b3cd7-235a-43f2-a551-051f163717f1-kube-api-access-km9pq\") pod \"5a2b3cd7-235a-43f2-a551-051f163717f1\" (UID: \"5a2b3cd7-235a-43f2-a551-051f163717f1\") "
Sep 10 03:16:26.117566 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:26.117355    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a2b3cd7-235a-43f2-a551-051f163717f1-dshm\") pod \"5a2b3cd7-235a-43f2-a551-051f163717f1\" (UID: \"5a2b3cd7-235a-43f2-a551-051f163717f1\") "
Sep 10 03:16:26.117623 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:26.117554    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a2b3cd7-235a-43f2-a551-051f163717f1-home" (OuterVolumeSpecName: "home") pod "5a2b3cd7-235a-43f2-a551-051f163717f1" (UID: "5a2b3cd7-235a-43f2-a551-051f163717f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:16:26.117623 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:26.117595    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a2b3cd7-235a-43f2-a551-051f163717f1-model-cache" (OuterVolumeSpecName: "model-cache") pod "5a2b3cd7-235a-43f2-a551-051f163717f1" (UID: "5a2b3cd7-235a-43f2-a551-051f163717f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:16:26.117700 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:26.117604    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a2b3cd7-235a-43f2-a551-051f163717f1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5a2b3cd7-235a-43f2-a551-051f163717f1" (UID: "5a2b3cd7-235a-43f2-a551-051f163717f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:16:26.119544 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:26.119516    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a2b3cd7-235a-43f2-a551-051f163717f1-kube-api-access-km9pq" (OuterVolumeSpecName: "kube-api-access-km9pq") pod "5a2b3cd7-235a-43f2-a551-051f163717f1" (UID: "5a2b3cd7-235a-43f2-a551-051f163717f1"). InnerVolumeSpecName "kube-api-access-km9pq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:16:26.119544 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:26.119517    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a2b3cd7-235a-43f2-a551-051f163717f1-dshm" (OuterVolumeSpecName: "dshm") pod "5a2b3cd7-235a-43f2-a551-051f163717f1" (UID: "5a2b3cd7-235a-43f2-a551-051f163717f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:16:26.119544 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:26.119528    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a2b3cd7-235a-43f2-a551-051f163717f1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5a2b3cd7-235a-43f2-a551-051f163717f1" (UID: "5a2b3cd7-235a-43f2-a551-051f163717f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:16:26.178956 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:26.178930    2558 generic.go:356] "Generic (PLEG): container finished" podID="5a2b3cd7-235a-43f2-a551-051f163717f1" containerID="65bd8ddd6b87024ba997a5f45f41a4f67edabe0450cfb3d36f514645023d02af" exitCode=0
Sep 10 03:16:26.179353 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:26.178973    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh"
Sep 10 03:16:26.179353 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:26.179010    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh" event={"ID":"5a2b3cd7-235a-43f2-a551-051f163717f1","Type":"ContainerDied","Data":"65bd8ddd6b87024ba997a5f45f41a4f67edabe0450cfb3d36f514645023d02af"}
Sep 10 03:16:26.179353 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:26.179050    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh" event={"ID":"5a2b3cd7-235a-43f2-a551-051f163717f1","Type":"ContainerDied","Data":"eb07472f16f82767905b6e92e9de4242930095ade48fb47573c87d68aa9d4e74"}
Sep 10 03:16:26.179353 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:26.179077    2558 scope.go:117] "RemoveContainer" containerID="65bd8ddd6b87024ba997a5f45f41a4f67edabe0450cfb3d36f514645023d02af"
Sep 10 03:16:26.187912 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:26.187897    2558 scope.go:117] "RemoveContainer" containerID="65bd8ddd6b87024ba997a5f45f41a4f67edabe0450cfb3d36f514645023d02af"
Sep 10 03:16:26.188166 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:16:26.188145    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65bd8ddd6b87024ba997a5f45f41a4f67edabe0450cfb3d36f514645023d02af\": container with ID starting with 65bd8ddd6b87024ba997a5f45f41a4f67edabe0450cfb3d36f514645023d02af not found: ID does not exist" containerID="65bd8ddd6b87024ba997a5f45f41a4f67edabe0450cfb3d36f514645023d02af"
Sep 10 03:16:26.188223 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:26.188174    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65bd8ddd6b87024ba997a5f45f41a4f67edabe0450cfb3d36f514645023d02af"} err="failed to get container status \"65bd8ddd6b87024ba997a5f45f41a4f67edabe0450cfb3d36f514645023d02af\": rpc error: code = NotFound desc = could not find container \"65bd8ddd6b87024ba997a5f45f41a4f67edabe0450cfb3d36f514645023d02af\": container with ID starting with 65bd8ddd6b87024ba997a5f45f41a4f67edabe0450cfb3d36f514645023d02af not found: ID does not exist"
Sep 10 03:16:26.203037 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:26.200180    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh"]
Sep 10 03:16:26.204269 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:26.204246    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7daa14/sub-test-model-7daa14-kserve-5bfd95475f-29hmh"]
Sep 10 03:16:26.217922 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:26.217898    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a2b3cd7-235a-43f2-a551-051f163717f1-dshm\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:16:26.217994 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:26.217925    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a2b3cd7-235a-43f2-a551-051f163717f1-tls-certs\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:16:26.217994 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:26.217938    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a2b3cd7-235a-43f2-a551-051f163717f1-model-cache\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:16:26.217994 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:26.217950    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a2b3cd7-235a-43f2-a551-051f163717f1-home\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:16:26.217994 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:26.217961    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a2b3cd7-235a-43f2-a551-051f163717f1-tmp-dir\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:16:26.217994 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:26.217972    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km9pq\" (UniqueName: \"kubernetes.io/projected/5a2b3cd7-235a-43f2-a551-051f163717f1-kube-api-access-km9pq\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:16:28.068921 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:28.068887    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a2b3cd7-235a-43f2-a551-051f163717f1" path="/var/lib/kubelet/pods/5a2b3cd7-235a-43f2-a551-051f163717f1/volumes"
Sep 10 03:16:55.415009 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.414971    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp"]
Sep 10 03:16:55.415594 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.415474    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd55b6aa-8203-4025-97ed-0fa65ca98c73" containerName="payload-pre-processing"
Sep 10 03:16:55.415594 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.415490    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd55b6aa-8203-4025-97ed-0fa65ca98c73" containerName="payload-pre-processing"
Sep 10 03:16:55.415594 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.415499    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19549650-bd12-4ea4-83b6-30df64ab2587" containerName="main"
Sep 10 03:16:55.415594 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.415505    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="19549650-bd12-4ea4-83b6-30df64ab2587" containerName="main"
Sep 10 03:16:55.415594 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.415515    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eb1f35e-640b-49f1-bd86-38a8d56b56a6" containerName="payload-processing"
Sep 10 03:16:55.415594 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.415520    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eb1f35e-640b-49f1-bd86-38a8d56b56a6" containerName="payload-processing"
Sep 10 03:16:55.415594 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.415526    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34cfdfdb-8ce5-4095-96a1-4d7152d99030" containerName="main"
Sep 10 03:16:55.415594 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.415531    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="34cfdfdb-8ce5-4095-96a1-4d7152d99030" containerName="main"
Sep 10 03:16:55.415594 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.415540    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b74f9d2-9e8c-478f-9f38-1fef910c943c" containerName="main"
Sep 10 03:16:55.415594 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.415545    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b74f9d2-9e8c-478f-9f38-1fef910c943c" containerName="main"
Sep 10 03:16:55.415594 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.415552    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c59542ed-dba2-4e16-9136-ad732d707c3e" containerName="revoke-keys"
Sep 10 03:16:55.415594 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.415557    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c59542ed-dba2-4e16-9136-ad732d707c3e" containerName="revoke-keys"
Sep 10 03:16:55.415594 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.415571    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a2b3cd7-235a-43f2-a551-051f163717f1" containerName="main"
Sep 10 03:16:55.415594 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.415575    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a2b3cd7-235a-43f2-a551-051f163717f1" containerName="main"
Sep 10 03:16:55.415594 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.415584    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4" containerName="main"
Sep 10 03:16:55.415594 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.415588    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4" containerName="main"
Sep 10 03:16:55.416082 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.415638    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a2b3cd7-235a-43f2-a551-051f163717f1" containerName="main"
Sep 10 03:16:55.416082 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.415647    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="34cfdfdb-8ce5-4095-96a1-4d7152d99030" containerName="main"
Sep 10 03:16:55.416082 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.415654    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c59542ed-dba2-4e16-9136-ad732d707c3e" containerName="revoke-keys"
Sep 10 03:16:55.416082 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.415660    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="90cdbd86-1f59-4cf7-81c7-d04bf29cc0c4" containerName="main"
Sep 10 03:16:55.416082 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.415665    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="19549650-bd12-4ea4-83b6-30df64ab2587" containerName="main"
Sep 10 03:16:55.416082 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.415670    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9eb1f35e-640b-49f1-bd86-38a8d56b56a6" containerName="payload-processing"
Sep 10 03:16:55.416082 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.415676    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd55b6aa-8203-4025-97ed-0fa65ca98c73" containerName="payload-pre-processing"
Sep 10 03:16:55.416082 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.415682    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b74f9d2-9e8c-478f-9f38-1fef910c943c" containerName="main"
Sep 10 03:16:55.418688 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.418670    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp"
Sep 10 03:16:55.420742 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.420715    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7daa14\"/\"kube-root-ca.crt\""
Sep 10 03:16:55.420860 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.420782    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7daa14\"/\"openshift-service-ca.crt\""
Sep 10 03:16:55.421364 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.421343    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7daa14\"/\"default-dockercfg-n5bs2\""
Sep 10 03:16:55.421470 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.421364    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7daa14\"/\"rate-test-model-7daa14-kserve-self-signed-certs\""
Sep 10 03:16:55.426707 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.426679    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp"]
Sep 10 03:16:55.434920 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.434898    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:16:55.521274 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.521243    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88c0259c-b5f7-4375-b995-7b041628eb9b-tmp-dir\") pod \"rate-test-model-7daa14-kserve-678dc44c47-zlwqp\" (UID: \"88c0259c-b5f7-4375-b995-7b041628eb9b\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp"
Sep 10 03:16:55.521388 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.521295    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88c0259c-b5f7-4375-b995-7b041628eb9b-tls-certs\") pod \"rate-test-model-7daa14-kserve-678dc44c47-zlwqp\" (UID: \"88c0259c-b5f7-4375-b995-7b041628eb9b\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp"
Sep 10 03:16:55.521388 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.521322    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88c0259c-b5f7-4375-b995-7b041628eb9b-model-cache\") pod \"rate-test-model-7daa14-kserve-678dc44c47-zlwqp\" (UID: \"88c0259c-b5f7-4375-b995-7b041628eb9b\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp"
Sep 10 03:16:55.521388 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.521381    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88c0259c-b5f7-4375-b995-7b041628eb9b-dshm\") pod \"rate-test-model-7daa14-kserve-678dc44c47-zlwqp\" (UID: \"88c0259c-b5f7-4375-b995-7b041628eb9b\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp"
Sep 10 03:16:55.521497 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.521417    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7hsl\" (UniqueName: \"kubernetes.io/projected/88c0259c-b5f7-4375-b995-7b041628eb9b-kube-api-access-s7hsl\") pod \"rate-test-model-7daa14-kserve-678dc44c47-zlwqp\" (UID: \"88c0259c-b5f7-4375-b995-7b041628eb9b\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp"
Sep 10 03:16:55.521497 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.521435    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88c0259c-b5f7-4375-b995-7b041628eb9b-home\") pod \"rate-test-model-7daa14-kserve-678dc44c47-zlwqp\" (UID: \"88c0259c-b5f7-4375-b995-7b041628eb9b\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp"
Sep 10 03:16:55.622539 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.622516    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88c0259c-b5f7-4375-b995-7b041628eb9b-dshm\") pod \"rate-test-model-7daa14-kserve-678dc44c47-zlwqp\" (UID: \"88c0259c-b5f7-4375-b995-7b041628eb9b\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp"
Sep 10 03:16:55.622643 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.622554    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7hsl\" (UniqueName: \"kubernetes.io/projected/88c0259c-b5f7-4375-b995-7b041628eb9b-kube-api-access-s7hsl\") pod \"rate-test-model-7daa14-kserve-678dc44c47-zlwqp\" (UID: \"88c0259c-b5f7-4375-b995-7b041628eb9b\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp"
Sep 10 03:16:55.622643 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.622577    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88c0259c-b5f7-4375-b995-7b041628eb9b-home\") pod \"rate-test-model-7daa14-kserve-678dc44c47-zlwqp\" (UID: \"88c0259c-b5f7-4375-b995-7b041628eb9b\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp"
Sep 10 03:16:55.622643 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.622608    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88c0259c-b5f7-4375-b995-7b041628eb9b-tmp-dir\") pod \"rate-test-model-7daa14-kserve-678dc44c47-zlwqp\" (UID: \"88c0259c-b5f7-4375-b995-7b041628eb9b\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp"
Sep 10 03:16:55.622643 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.622623    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88c0259c-b5f7-4375-b995-7b041628eb9b-tls-certs\") pod \"rate-test-model-7daa14-kserve-678dc44c47-zlwqp\" (UID: \"88c0259c-b5f7-4375-b995-7b041628eb9b\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp"
Sep 10 03:16:55.622804 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.622645    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88c0259c-b5f7-4375-b995-7b041628eb9b-model-cache\") pod \"rate-test-model-7daa14-kserve-678dc44c47-zlwqp\" (UID: \"88c0259c-b5f7-4375-b995-7b041628eb9b\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp"
Sep 10 03:16:55.622993 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.622971    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88c0259c-b5f7-4375-b995-7b041628eb9b-tmp-dir\") pod \"rate-test-model-7daa14-kserve-678dc44c47-zlwqp\" (UID: \"88c0259c-b5f7-4375-b995-7b041628eb9b\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp"
Sep 10 03:16:55.623130 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.623018    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88c0259c-b5f7-4375-b995-7b041628eb9b-home\") pod \"rate-test-model-7daa14-kserve-678dc44c47-zlwqp\" (UID: \"88c0259c-b5f7-4375-b995-7b041628eb9b\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp"
Sep 10 03:16:55.623130 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.623030    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88c0259c-b5f7-4375-b995-7b041628eb9b-model-cache\") pod \"rate-test-model-7daa14-kserve-678dc44c47-zlwqp\" (UID: \"88c0259c-b5f7-4375-b995-7b041628eb9b\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp"
Sep 10 03:16:55.624563 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.624542    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88c0259c-b5f7-4375-b995-7b041628eb9b-dshm\") pod \"rate-test-model-7daa14-kserve-678dc44c47-zlwqp\" (UID: \"88c0259c-b5f7-4375-b995-7b041628eb9b\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp"
Sep 10 03:16:55.624896 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.624879    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88c0259c-b5f7-4375-b995-7b041628eb9b-tls-certs\") pod \"rate-test-model-7daa14-kserve-678dc44c47-zlwqp\" (UID: \"88c0259c-b5f7-4375-b995-7b041628eb9b\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp"
Sep 10 03:16:55.629479 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.629455    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7hsl\" (UniqueName: \"kubernetes.io/projected/88c0259c-b5f7-4375-b995-7b041628eb9b-kube-api-access-s7hsl\") pod \"rate-test-model-7daa14-kserve-678dc44c47-zlwqp\" (UID: \"88c0259c-b5f7-4375-b995-7b041628eb9b\") " pod="ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp"
Sep 10 03:16:55.730090 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:55.730065    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp"
Sep 10 03:16:56.000347 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:56.000323    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp"]
Sep 10 03:16:56.002679 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:16:56.002650    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88c0259c_b5f7_4375_b995_7b041628eb9b.slice/crio-2d68f92842d4066d440109c1a1d0f1595271eb7e4422f7a8c86815c7e847bb69 WatchSource:0}: Error finding container 2d68f92842d4066d440109c1a1d0f1595271eb7e4422f7a8c86815c7e847bb69: Status 404 returned error can't find the container with id 2d68f92842d4066d440109c1a1d0f1595271eb7e4422f7a8c86815c7e847bb69
Sep 10 03:16:56.353431 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:56.353346    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp" event={"ID":"88c0259c-b5f7-4375-b995-7b041628eb9b","Type":"ContainerStarted","Data":"c5d196e41dfb07bbaa4cac3954b1f284b3e92ff8b99e8f1d3ed88ed3b00e2892"}
Sep 10 03:16:56.353431 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:56.353394    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp" event={"ID":"88c0259c-b5f7-4375-b995-7b041628eb9b","Type":"ContainerStarted","Data":"2d68f92842d4066d440109c1a1d0f1595271eb7e4422f7a8c86815c7e847bb69"}
Sep 10 03:16:56.369201 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:16:56.369157    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp" podStartSLOduration=1.369142537 podStartE2EDuration="1.369142537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:16:56.367740811 +0000 UTC m=+1806.742826733" watchObservedRunningTime="2026-09-10 03:16:56.369142537 +0000 UTC m=+1806.744228458"
Sep 10 03:17:05.730213 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:05.730177    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp"
Sep 10 03:17:05.732492 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:05.730271    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp"
Sep 10 03:17:05.743019 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:05.742994    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp"
Sep 10 03:17:06.418397 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:06.418361    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp"
Sep 10 03:17:29.810976 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:29.810939    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:17:30.693982 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:30.693946    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp"]
Sep 10 03:17:31.508592 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:31.508556    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:17:31.631953 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:31.631918    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:17:34.310666 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:34.310630    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:17:34.404069 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:34.404025    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:17:35.462647 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:35.462614    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d"]
Sep 10 03:17:35.466350 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:35.466321    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d"
Sep 10 03:17:35.468885 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:35.468866    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 03:17:35.469008 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:35.468872    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-c9cef5-serving-cert\""
Sep 10 03:17:35.469848 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:35.469825    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fqv2s\""
Sep 10 03:17:35.477756 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:35.477208    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d"]
Sep 10 03:17:35.502286 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:35.502265    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:17:35.591085 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:35.591060    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:17:35.619266 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:35.619236    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6dc7868-800f-46fa-af8e-103b4ea82d36-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d\" (UID: \"c6dc7868-800f-46fa-af8e-103b4ea82d36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d"
Sep 10 03:17:35.619359 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:35.619281    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6dc7868-800f-46fa-af8e-103b4ea82d36-metrics-certs\") pod \"maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d\" (UID: \"c6dc7868-800f-46fa-af8e-103b4ea82d36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d"
Sep 10 03:17:35.619359 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:35.619308    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6dc7868-800f-46fa-af8e-103b4ea82d36-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d\" (UID: \"c6dc7868-800f-46fa-af8e-103b4ea82d36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d"
Sep 10 03:17:35.619437 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:35.619381    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7whwq\" (UniqueName: \"kubernetes.io/projected/c6dc7868-800f-46fa-af8e-103b4ea82d36-kube-api-access-7whwq\") pod \"maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d\" (UID: \"c6dc7868-800f-46fa-af8e-103b4ea82d36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d"
Sep 10 03:17:35.720256 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:35.720198    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6dc7868-800f-46fa-af8e-103b4ea82d36-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d\" (UID: \"c6dc7868-800f-46fa-af8e-103b4ea82d36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d"
Sep 10 03:17:35.720354 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:35.720290    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6dc7868-800f-46fa-af8e-103b4ea82d36-metrics-certs\") pod \"maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d\" (UID: \"c6dc7868-800f-46fa-af8e-103b4ea82d36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d"
Sep 10 03:17:35.720354 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:35.720329    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6dc7868-800f-46fa-af8e-103b4ea82d36-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d\" (UID: \"c6dc7868-800f-46fa-af8e-103b4ea82d36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d"
Sep 10 03:17:35.720456 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:35.720363    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7whwq\" (UniqueName: \"kubernetes.io/projected/c6dc7868-800f-46fa-af8e-103b4ea82d36-kube-api-access-7whwq\") pod \"maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d\" (UID: \"c6dc7868-800f-46fa-af8e-103b4ea82d36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d"
Sep 10 03:17:35.720510 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:17:35.720491    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9cef5-serving-cert: secret "maas-api-e2e-ipp-a-c9cef5-serving-cert" not found
Sep 10 03:17:35.720580 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:17:35.720565    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6dc7868-800f-46fa-af8e-103b4ea82d36-maas-api-tls podName:c6dc7868-800f-46fa-af8e-103b4ea82d36 nodeName:}" failed. No retries permitted until 2026-09-10 03:17:36.220544489 +0000 UTC m=+1846.595630390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c6dc7868-800f-46fa-af8e-103b4ea82d36-maas-api-tls") pod "maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d" (UID: "c6dc7868-800f-46fa-af8e-103b4ea82d36") : secret "maas-api-e2e-ipp-a-c9cef5-serving-cert" not found
Sep 10 03:17:35.720918 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:35.720900    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6dc7868-800f-46fa-af8e-103b4ea82d36-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d\" (UID: \"c6dc7868-800f-46fa-af8e-103b4ea82d36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d"
Sep 10 03:17:35.722628 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:35.722607    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6dc7868-800f-46fa-af8e-103b4ea82d36-metrics-certs\") pod \"maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d\" (UID: \"c6dc7868-800f-46fa-af8e-103b4ea82d36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d"
Sep 10 03:17:35.739798 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:35.739772    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7whwq\" (UniqueName: \"kubernetes.io/projected/c6dc7868-800f-46fa-af8e-103b4ea82d36-kube-api-access-7whwq\") pod \"maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d\" (UID: \"c6dc7868-800f-46fa-af8e-103b4ea82d36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d"
Sep 10 03:17:35.864288 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:35.864258    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:17:36.058447 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:36.058352    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:17:36.224373 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:36.224342    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6dc7868-800f-46fa-af8e-103b4ea82d36-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d\" (UID: \"c6dc7868-800f-46fa-af8e-103b4ea82d36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d"
Sep 10 03:17:36.225805 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:36.225775    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:17:36.226712 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:36.226690    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6dc7868-800f-46fa-af8e-103b4ea82d36-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d\" (UID: \"c6dc7868-800f-46fa-af8e-103b4ea82d36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d"
Sep 10 03:17:36.265098 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:36.265073    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c9cef5-5bb46b7fb6-gc2m8"]
Sep 10 03:17:36.268494 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:36.268478    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9cef5-5bb46b7fb6-gc2m8"
Sep 10 03:17:36.271214 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:36.271196    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c9cef5\""
Sep 10 03:17:36.280822 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:36.280801    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c9cef5-5bb46b7fb6-gc2m8"]
Sep 10 03:17:36.345360 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:36.345295    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c9cef5-d578999ff-7h2vd"]
Sep 10 03:17:36.348588 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:36.348569    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c9cef5-d578999ff-7h2vd"
Sep 10 03:17:36.350757 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:36.350735    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c9cef5-dockercfg-559j9\""
Sep 10 03:17:36.361477 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:36.361456    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c9cef5-d578999ff-7h2vd"]
Sep 10 03:17:36.378960 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:36.378940    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d"
Sep 10 03:17:36.425743 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:36.425711    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a678a31-a511-4f52-9890-ec6026611cbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c9cef5-5bb46b7fb6-gc2m8\" (UID: \"3a678a31-a511-4f52-9890-ec6026611cbe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9cef5-5bb46b7fb6-gc2m8"
Sep 10 03:17:36.425917 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:36.425769    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwnbp\" (UniqueName: \"kubernetes.io/projected/3a678a31-a511-4f52-9890-ec6026611cbe-kube-api-access-rwnbp\") pod \"payload-pre-processing-e2e-ipp-a-c9cef5-5bb46b7fb6-gc2m8\" (UID: \"3a678a31-a511-4f52-9890-ec6026611cbe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9cef5-5bb46b7fb6-gc2m8"
Sep 10 03:17:36.526668 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:36.526626    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a678a31-a511-4f52-9890-ec6026611cbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c9cef5-5bb46b7fb6-gc2m8\" (UID: \"3a678a31-a511-4f52-9890-ec6026611cbe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9cef5-5bb46b7fb6-gc2m8"
Sep 10 03:17:36.527081 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:36.526687    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de8d4b8-4fa7-46db-9390-6fa3b4249971-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c9cef5-d578999ff-7h2vd\" (UID: \"2de8d4b8-4fa7-46db-9390-6fa3b4249971\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c9cef5-d578999ff-7h2vd"
Sep 10 03:17:36.527081 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:36.526732    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwnbp\" (UniqueName: \"kubernetes.io/projected/3a678a31-a511-4f52-9890-ec6026611cbe-kube-api-access-rwnbp\") pod \"payload-pre-processing-e2e-ipp-a-c9cef5-5bb46b7fb6-gc2m8\" (UID: \"3a678a31-a511-4f52-9890-ec6026611cbe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9cef5-5bb46b7fb6-gc2m8"
Sep 10 03:17:36.527081 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:36.526790    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcnhr\" (UniqueName: \"kubernetes.io/projected/2de8d4b8-4fa7-46db-9390-6fa3b4249971-kube-api-access-kcnhr\") pod \"payload-processing-e2e-ipp-a-c9cef5-d578999ff-7h2vd\" (UID: \"2de8d4b8-4fa7-46db-9390-6fa3b4249971\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c9cef5-d578999ff-7h2vd"
Sep 10 03:17:36.527374 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:36.527353    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a678a31-a511-4f52-9890-ec6026611cbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c9cef5-5bb46b7fb6-gc2m8\" (UID: \"3a678a31-a511-4f52-9890-ec6026611cbe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9cef5-5bb46b7fb6-gc2m8"
Sep 10 03:17:36.535703 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:36.535670    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwnbp\" (UniqueName: \"kubernetes.io/projected/3a678a31-a511-4f52-9890-ec6026611cbe-kube-api-access-rwnbp\") pod \"payload-pre-processing-e2e-ipp-a-c9cef5-5bb46b7fb6-gc2m8\" (UID: \"3a678a31-a511-4f52-9890-ec6026611cbe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9cef5-5bb46b7fb6-gc2m8"
Sep 10 03:17:36.579470 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:36.579438    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9cef5-5bb46b7fb6-gc2m8"
Sep 10 03:17:36.628060 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:36.628022    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de8d4b8-4fa7-46db-9390-6fa3b4249971-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c9cef5-d578999ff-7h2vd\" (UID: \"2de8d4b8-4fa7-46db-9390-6fa3b4249971\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c9cef5-d578999ff-7h2vd"
Sep 10 03:17:36.628186 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:36.628147    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcnhr\" (UniqueName: \"kubernetes.io/projected/2de8d4b8-4fa7-46db-9390-6fa3b4249971-kube-api-access-kcnhr\") pod \"payload-processing-e2e-ipp-a-c9cef5-d578999ff-7h2vd\" (UID: \"2de8d4b8-4fa7-46db-9390-6fa3b4249971\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c9cef5-d578999ff-7h2vd"
Sep 10 03:17:36.628989 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:36.628962    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de8d4b8-4fa7-46db-9390-6fa3b4249971-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c9cef5-d578999ff-7h2vd\" (UID: \"2de8d4b8-4fa7-46db-9390-6fa3b4249971\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c9cef5-d578999ff-7h2vd"
Sep 10 03:17:36.637678 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:36.637654    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcnhr\" (UniqueName: \"kubernetes.io/projected/2de8d4b8-4fa7-46db-9390-6fa3b4249971-kube-api-access-kcnhr\") pod \"payload-processing-e2e-ipp-a-c9cef5-d578999ff-7h2vd\" (UID: \"2de8d4b8-4fa7-46db-9390-6fa3b4249971\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c9cef5-d578999ff-7h2vd"
Sep 10 03:17:36.659286 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:36.659261    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c9cef5-d578999ff-7h2vd"
Sep 10 03:17:36.662369 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:36.662348    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d"]
Sep 10 03:17:36.664699 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:17:36.664676    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6dc7868_800f_46fa_af8e_103b4ea82d36.slice/crio-b5026c85f0111a0ec5f8cd477b4d98e0d44e8d194d91b09f45e9209ebf956fc9 WatchSource:0}: Error finding container b5026c85f0111a0ec5f8cd477b4d98e0d44e8d194d91b09f45e9209ebf956fc9: Status 404 returned error can't find the container with id b5026c85f0111a0ec5f8cd477b4d98e0d44e8d194d91b09f45e9209ebf956fc9
Sep 10 03:17:36.666079 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:36.666059    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 03:17:36.868740 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:36.868715    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c9cef5-5bb46b7fb6-gc2m8"]
Sep 10 03:17:36.870544 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:17:36.870516    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a678a31_a511_4f52_9890_ec6026611cbe.slice/crio-9cc32c1da3e19b3f5a66dbe955e292ffad132aff1f200d0fcf63fd4647d055d7 WatchSource:0}: Error finding container 9cc32c1da3e19b3f5a66dbe955e292ffad132aff1f200d0fcf63fd4647d055d7: Status 404 returned error can't find the container with id 9cc32c1da3e19b3f5a66dbe955e292ffad132aff1f200d0fcf63fd4647d055d7
Sep 10 03:17:36.942740 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:17:36.942713    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2de8d4b8_4fa7_46db_9390_6fa3b4249971.slice/crio-66979e9d114660976c72e7a016727c13e9e2e0c63f5901985b70686909f9e7fb WatchSource:0}: Error finding container 66979e9d114660976c72e7a016727c13e9e2e0c63f5901985b70686909f9e7fb: Status 404 returned error can't find the container with id 66979e9d114660976c72e7a016727c13e9e2e0c63f5901985b70686909f9e7fb
Sep 10 03:17:36.943505 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:36.943486    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c9cef5-d578999ff-7h2vd"]
Sep 10 03:17:37.591677 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:37.591645    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d" event={"ID":"c6dc7868-800f-46fa-af8e-103b4ea82d36","Type":"ContainerStarted","Data":"5b0a6991436efc88fe51aef80644242e697376c16c8e2e9fd281e61cba801e22"}
Sep 10 03:17:37.592136 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:37.591705    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d" event={"ID":"c6dc7868-800f-46fa-af8e-103b4ea82d36","Type":"ContainerStarted","Data":"b5026c85f0111a0ec5f8cd477b4d98e0d44e8d194d91b09f45e9209ebf956fc9"}
Sep 10 03:17:37.592136 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:37.591806    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d"
Sep 10 03:17:37.593327 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:37.593299    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c9cef5-d578999ff-7h2vd" event={"ID":"2de8d4b8-4fa7-46db-9390-6fa3b4249971","Type":"ContainerStarted","Data":"ce7eeb800a9053781fcb6461fe9feac82703cc6fca195dfe9361b5654787e1ac"}
Sep 10 03:17:37.593454 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:37.593355    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c9cef5-d578999ff-7h2vd"
Sep 10 03:17:37.593454 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:37.593373    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c9cef5-d578999ff-7h2vd" event={"ID":"2de8d4b8-4fa7-46db-9390-6fa3b4249971","Type":"ContainerStarted","Data":"66979e9d114660976c72e7a016727c13e9e2e0c63f5901985b70686909f9e7fb"}
Sep 10 03:17:37.594987 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:37.594961    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9cef5-5bb46b7fb6-gc2m8" event={"ID":"3a678a31-a511-4f52-9890-ec6026611cbe","Type":"ContainerStarted","Data":"f2bf3f74ee089400488ab268dd21076acc4c1b0a8abd007eb69acf7699f6b4a3"}
Sep 10 03:17:37.595129 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:37.595005    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9cef5-5bb46b7fb6-gc2m8" event={"ID":"3a678a31-a511-4f52-9890-ec6026611cbe","Type":"ContainerStarted","Data":"9cc32c1da3e19b3f5a66dbe955e292ffad132aff1f200d0fcf63fd4647d055d7"}
Sep 10 03:17:37.595129 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:37.595058    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9cef5-5bb46b7fb6-gc2m8"
Sep 10 03:17:37.610762 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:37.610725    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d" podStartSLOduration=2.188506507 podStartE2EDuration="2.61071104s" totalImagesPullingTime="422.204533ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:17:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:17:37.610144576 +0000 UTC m=+1847.985230505" watchObservedRunningTime="2026-09-10 03:17:37.61071104 +0000 UTC m=+1847.985796960"
Sep 10 03:17:37.627718 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:37.627678    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c9cef5-d578999ff-7h2vd" podStartSLOduration=1.627666115 podStartE2EDuration="1.627666115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:17:37.625945113 +0000 UTC m=+1848.001031034" watchObservedRunningTime="2026-09-10 03:17:37.627666115 +0000 UTC m=+1848.002752040"
Sep 10 03:17:37.646898 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:37.646859    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9cef5-5bb46b7fb6-gc2m8" podStartSLOduration=1.6468509569999998 podStartE2EDuration="1.646850957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:17:37.645336274 +0000 UTC m=+1848.020422195" watchObservedRunningTime="2026-09-10 03:17:37.646850957 +0000 UTC m=+1848.021936923"
Sep 10 03:17:42.796575 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:42.796541    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:17:43.606949 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:43.606918    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d"
Sep 10 03:17:45.713331 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:45.713290    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp" podUID="88c0259c-b5f7-4375-b995-7b041628eb9b" containerName="main" containerID="cri-o://c5d196e41dfb07bbaa4cac3954b1f284b3e92ff8b99e8f1d3ed88ed3b00e2892" gracePeriod=45
Sep 10 03:17:45.955911 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:45.955885    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp"
Sep 10 03:17:46.102866 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:46.102791    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88c0259c-b5f7-4375-b995-7b041628eb9b-tmp-dir\") pod \"88c0259c-b5f7-4375-b995-7b041628eb9b\" (UID: \"88c0259c-b5f7-4375-b995-7b041628eb9b\") "
Sep 10 03:17:46.102866 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:46.102852    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88c0259c-b5f7-4375-b995-7b041628eb9b-home\") pod \"88c0259c-b5f7-4375-b995-7b041628eb9b\" (UID: \"88c0259c-b5f7-4375-b995-7b041628eb9b\") "
Sep 10 03:17:46.103196 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:46.102895    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88c0259c-b5f7-4375-b995-7b041628eb9b-tls-certs\") pod \"88c0259c-b5f7-4375-b995-7b041628eb9b\" (UID: \"88c0259c-b5f7-4375-b995-7b041628eb9b\") "
Sep 10 03:17:46.103196 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:46.102940    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7hsl\" (UniqueName: \"kubernetes.io/projected/88c0259c-b5f7-4375-b995-7b041628eb9b-kube-api-access-s7hsl\") pod \"88c0259c-b5f7-4375-b995-7b041628eb9b\" (UID: \"88c0259c-b5f7-4375-b995-7b041628eb9b\") "
Sep 10 03:17:46.103196 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:46.102989    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88c0259c-b5f7-4375-b995-7b041628eb9b-model-cache\") pod \"88c0259c-b5f7-4375-b995-7b041628eb9b\" (UID: \"88c0259c-b5f7-4375-b995-7b041628eb9b\") "
Sep 10 03:17:46.103196 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:46.103010    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88c0259c-b5f7-4375-b995-7b041628eb9b-dshm\") pod \"88c0259c-b5f7-4375-b995-7b041628eb9b\" (UID: \"88c0259c-b5f7-4375-b995-7b041628eb9b\") "
Sep 10 03:17:46.103196 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:46.103069    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c0259c-b5f7-4375-b995-7b041628eb9b-home" (OuterVolumeSpecName: "home") pod "88c0259c-b5f7-4375-b995-7b041628eb9b" (UID: "88c0259c-b5f7-4375-b995-7b041628eb9b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:17:46.103196 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:46.103086    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c0259c-b5f7-4375-b995-7b041628eb9b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "88c0259c-b5f7-4375-b995-7b041628eb9b" (UID: "88c0259c-b5f7-4375-b995-7b041628eb9b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:17:46.103487 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:46.103316    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c0259c-b5f7-4375-b995-7b041628eb9b-model-cache" (OuterVolumeSpecName: "model-cache") pod "88c0259c-b5f7-4375-b995-7b041628eb9b" (UID: "88c0259c-b5f7-4375-b995-7b041628eb9b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:17:46.103487 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:46.103323    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88c0259c-b5f7-4375-b995-7b041628eb9b-tmp-dir\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:17:46.103487 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:46.103363    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88c0259c-b5f7-4375-b995-7b041628eb9b-home\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:17:46.105047 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:46.105028    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c0259c-b5f7-4375-b995-7b041628eb9b-dshm" (OuterVolumeSpecName: "dshm") pod "88c0259c-b5f7-4375-b995-7b041628eb9b" (UID: "88c0259c-b5f7-4375-b995-7b041628eb9b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:17:46.105152 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:46.105074    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88c0259c-b5f7-4375-b995-7b041628eb9b-kube-api-access-s7hsl" (OuterVolumeSpecName: "kube-api-access-s7hsl") pod "88c0259c-b5f7-4375-b995-7b041628eb9b" (UID: "88c0259c-b5f7-4375-b995-7b041628eb9b"). InnerVolumeSpecName "kube-api-access-s7hsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:17:46.105211 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:46.105162    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88c0259c-b5f7-4375-b995-7b041628eb9b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88c0259c-b5f7-4375-b995-7b041628eb9b" (UID: "88c0259c-b5f7-4375-b995-7b041628eb9b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:17:46.203681 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:46.203653    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88c0259c-b5f7-4375-b995-7b041628eb9b-model-cache\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:17:46.203681 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:46.203679    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88c0259c-b5f7-4375-b995-7b041628eb9b-dshm\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:17:46.203855 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:46.203693    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88c0259c-b5f7-4375-b995-7b041628eb9b-tls-certs\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:17:46.203855 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:46.203706    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7hsl\" (UniqueName: \"kubernetes.io/projected/88c0259c-b5f7-4375-b995-7b041628eb9b-kube-api-access-s7hsl\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:17:46.648482 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:46.648445    2558 generic.go:356] "Generic (PLEG): container finished" podID="88c0259c-b5f7-4375-b995-7b041628eb9b" containerID="c5d196e41dfb07bbaa4cac3954b1f284b3e92ff8b99e8f1d3ed88ed3b00e2892" exitCode=0
Sep 10 03:17:46.648662 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:46.648505    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp"
Sep 10 03:17:46.648662 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:46.648522    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp" event={"ID":"88c0259c-b5f7-4375-b995-7b041628eb9b","Type":"ContainerDied","Data":"c5d196e41dfb07bbaa4cac3954b1f284b3e92ff8b99e8f1d3ed88ed3b00e2892"}
Sep 10 03:17:46.648662 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:46.648557    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp" event={"ID":"88c0259c-b5f7-4375-b995-7b041628eb9b","Type":"ContainerDied","Data":"2d68f92842d4066d440109c1a1d0f1595271eb7e4422f7a8c86815c7e847bb69"}
Sep 10 03:17:46.648662 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:46.648576    2558 scope.go:117] "RemoveContainer" containerID="c5d196e41dfb07bbaa4cac3954b1f284b3e92ff8b99e8f1d3ed88ed3b00e2892"
Sep 10 03:17:46.658598 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:46.658579    2558 scope.go:117] "RemoveContainer" containerID="c5d196e41dfb07bbaa4cac3954b1f284b3e92ff8b99e8f1d3ed88ed3b00e2892"
Sep 10 03:17:46.658883 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:17:46.658864    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5d196e41dfb07bbaa4cac3954b1f284b3e92ff8b99e8f1d3ed88ed3b00e2892\": container with ID starting with c5d196e41dfb07bbaa4cac3954b1f284b3e92ff8b99e8f1d3ed88ed3b00e2892 not found: ID does not exist" containerID="c5d196e41dfb07bbaa4cac3954b1f284b3e92ff8b99e8f1d3ed88ed3b00e2892"
Sep 10 03:17:46.658946 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:46.658890    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5d196e41dfb07bbaa4cac3954b1f284b3e92ff8b99e8f1d3ed88ed3b00e2892"} err="failed to get container status \"c5d196e41dfb07bbaa4cac3954b1f284b3e92ff8b99e8f1d3ed88ed3b00e2892\": rpc error: code = NotFound desc = could not find container \"c5d196e41dfb07bbaa4cac3954b1f284b3e92ff8b99e8f1d3ed88ed3b00e2892\": container with ID starting with c5d196e41dfb07bbaa4cac3954b1f284b3e92ff8b99e8f1d3ed88ed3b00e2892 not found: ID does not exist"
Sep 10 03:17:46.671456 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:46.671433    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp"]
Sep 10 03:17:46.673641 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:46.673619    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7daa14/rate-test-model-7daa14-kserve-678dc44c47-zlwqp"]
Sep 10 03:17:48.068596 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:48.068566    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88c0259c-b5f7-4375-b995-7b041628eb9b" path="/var/lib/kubelet/pods/88c0259c-b5f7-4375-b995-7b041628eb9b/volumes"
Sep 10 03:17:48.604140 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:48.604088    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c9cef5-d578999ff-7h2vd"
Sep 10 03:17:48.604334 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:48.604193    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9cef5-5bb46b7fb6-gc2m8"
Sep 10 03:17:50.529248 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:50.529218    2558 scope.go:117] "RemoveContainer" containerID="327495bcc2ac3e7c0ea27c2e87a7c5c77f85e5b309bc1866f66fe54b7d136323"
Sep 10 03:17:59.337723 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:59.337686    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:17:59.439057 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:17:59.439013    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:18:00.063236 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:00.063197    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:18:00.346921 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:00.346819    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:18:00.608930 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:00.608840    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:18:03.581337 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:03.581295    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:18:03.812659 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:03.812619    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:18:04.753327 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:04.753288    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv"]
Sep 10 03:18:04.753985 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:04.753961    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88c0259c-b5f7-4375-b995-7b041628eb9b" containerName="main"
Sep 10 03:18:04.754098 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:04.753988    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="88c0259c-b5f7-4375-b995-7b041628eb9b" containerName="main"
Sep 10 03:18:04.754237 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:04.754143    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="88c0259c-b5f7-4375-b995-7b041628eb9b" containerName="main"
Sep 10 03:18:04.757607 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:04.757576    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv"
Sep 10 03:18:04.759780 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:04.759751    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-c970f3-serving-cert\""
Sep 10 03:18:04.768188 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:04.768149    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv"]
Sep 10 03:18:04.838248 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:04.838213    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:18:04.854476 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:04.854436    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxwpk\" (UniqueName: \"kubernetes.io/projected/7ad7c90c-9e2e-4bd3-b01f-00236dde69da-kube-api-access-nxwpk\") pod \"maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv\" (UID: \"7ad7c90c-9e2e-4bd3-b01f-00236dde69da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv"
Sep 10 03:18:04.854668 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:04.854514    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad7c90c-9e2e-4bd3-b01f-00236dde69da-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv\" (UID: \"7ad7c90c-9e2e-4bd3-b01f-00236dde69da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv"
Sep 10 03:18:04.854668 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:04.854536    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ad7c90c-9e2e-4bd3-b01f-00236dde69da-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv\" (UID: \"7ad7c90c-9e2e-4bd3-b01f-00236dde69da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv"
Sep 10 03:18:04.854668 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:04.854579    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ad7c90c-9e2e-4bd3-b01f-00236dde69da-metrics-certs\") pod \"maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv\" (UID: \"7ad7c90c-9e2e-4bd3-b01f-00236dde69da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv"
Sep 10 03:18:04.938838 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:04.938802    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:18:04.955625 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:04.955588    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxwpk\" (UniqueName: \"kubernetes.io/projected/7ad7c90c-9e2e-4bd3-b01f-00236dde69da-kube-api-access-nxwpk\") pod \"maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv\" (UID: \"7ad7c90c-9e2e-4bd3-b01f-00236dde69da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv"
Sep 10 03:18:04.955625 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:04.955645    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad7c90c-9e2e-4bd3-b01f-00236dde69da-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv\" (UID: \"7ad7c90c-9e2e-4bd3-b01f-00236dde69da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv"
Sep 10 03:18:04.955900 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:04.955668    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ad7c90c-9e2e-4bd3-b01f-00236dde69da-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv\" (UID: \"7ad7c90c-9e2e-4bd3-b01f-00236dde69da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv"
Sep 10 03:18:04.955900 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:04.955722    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ad7c90c-9e2e-4bd3-b01f-00236dde69da-metrics-certs\") pod \"maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv\" (UID: \"7ad7c90c-9e2e-4bd3-b01f-00236dde69da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv"
Sep 10 03:18:04.955900 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:18:04.955851    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-c970f3-serving-cert: secret "maas-api-e2e-ipp-b-c970f3-serving-cert" not found
Sep 10 03:18:04.956068 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:18:04.955937    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ad7c90c-9e2e-4bd3-b01f-00236dde69da-maas-api-tls podName:7ad7c90c-9e2e-4bd3-b01f-00236dde69da nodeName:}" failed. No retries permitted until 2026-09-10 03:18:05.455915417 +0000 UTC m=+1875.831001336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7ad7c90c-9e2e-4bd3-b01f-00236dde69da-maas-api-tls") pod "maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv" (UID: "7ad7c90c-9e2e-4bd3-b01f-00236dde69da") : secret "maas-api-e2e-ipp-b-c970f3-serving-cert" not found
Sep 10 03:18:04.956450 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:04.956425    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad7c90c-9e2e-4bd3-b01f-00236dde69da-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv\" (UID: \"7ad7c90c-9e2e-4bd3-b01f-00236dde69da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv"
Sep 10 03:18:04.958323 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:04.958287    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ad7c90c-9e2e-4bd3-b01f-00236dde69da-metrics-certs\") pod \"maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv\" (UID: \"7ad7c90c-9e2e-4bd3-b01f-00236dde69da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv"
Sep 10 03:18:04.969481 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:04.969450    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxwpk\" (UniqueName: \"kubernetes.io/projected/7ad7c90c-9e2e-4bd3-b01f-00236dde69da-kube-api-access-nxwpk\") pod \"maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv\" (UID: \"7ad7c90c-9e2e-4bd3-b01f-00236dde69da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv"
Sep 10 03:18:05.041897 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:05.041801    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:18:05.386065 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:05.385979    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:18:05.461431 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:05.461388    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ad7c90c-9e2e-4bd3-b01f-00236dde69da-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv\" (UID: \"7ad7c90c-9e2e-4bd3-b01f-00236dde69da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv"
Sep 10 03:18:05.463985 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:05.463952    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ad7c90c-9e2e-4bd3-b01f-00236dde69da-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv\" (UID: \"7ad7c90c-9e2e-4bd3-b01f-00236dde69da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv"
Sep 10 03:18:05.497341 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:05.497294    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c970f3-6bf9c9c56b-74cb9"]
Sep 10 03:18:05.501186 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:05.501162    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c970f3-6bf9c9c56b-74cb9"
Sep 10 03:18:05.503831 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:05.503581    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c970f3-dockercfg-8lxk6\""
Sep 10 03:18:05.503831 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:05.503605    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c970f3\""
Sep 10 03:18:05.508661 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:05.508592    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:18:05.510125 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:05.510075    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c970f3-6bf9c9c56b-74cb9"]
Sep 10 03:18:05.562346 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:05.562302    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kftbb\" (UniqueName: \"kubernetes.io/projected/6dd97aa1-a0fe-48d2-8b19-72f5769880a8-kube-api-access-kftbb\") pod \"payload-pre-processing-e2e-ipp-b-c970f3-6bf9c9c56b-74cb9\" (UID: \"6dd97aa1-a0fe-48d2-8b19-72f5769880a8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c970f3-6bf9c9c56b-74cb9"
Sep 10 03:18:05.562616 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:05.562422    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd97aa1-a0fe-48d2-8b19-72f5769880a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c970f3-6bf9c9c56b-74cb9\" (UID: \"6dd97aa1-a0fe-48d2-8b19-72f5769880a8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c970f3-6bf9c9c56b-74cb9"
Sep 10 03:18:05.663389 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:05.663282    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd97aa1-a0fe-48d2-8b19-72f5769880a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c970f3-6bf9c9c56b-74cb9\" (UID: \"6dd97aa1-a0fe-48d2-8b19-72f5769880a8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c970f3-6bf9c9c56b-74cb9"
Sep 10 03:18:05.663389 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:05.663370    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kftbb\" (UniqueName: \"kubernetes.io/projected/6dd97aa1-a0fe-48d2-8b19-72f5769880a8-kube-api-access-kftbb\") pod \"payload-pre-processing-e2e-ipp-b-c970f3-6bf9c9c56b-74cb9\" (UID: \"6dd97aa1-a0fe-48d2-8b19-72f5769880a8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c970f3-6bf9c9c56b-74cb9"
Sep 10 03:18:05.664018 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:05.663995    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd97aa1-a0fe-48d2-8b19-72f5769880a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c970f3-6bf9c9c56b-74cb9\" (UID: \"6dd97aa1-a0fe-48d2-8b19-72f5769880a8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c970f3-6bf9c9c56b-74cb9"
Sep 10 03:18:05.670507 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:05.670462    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv"
Sep 10 03:18:05.670840 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:05.670815    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kftbb\" (UniqueName: \"kubernetes.io/projected/6dd97aa1-a0fe-48d2-8b19-72f5769880a8-kube-api-access-kftbb\") pod \"payload-pre-processing-e2e-ipp-b-c970f3-6bf9c9c56b-74cb9\" (UID: \"6dd97aa1-a0fe-48d2-8b19-72f5769880a8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c970f3-6bf9c9c56b-74cb9"
Sep 10 03:18:05.814860 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:05.814795    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c970f3-6bf9c9c56b-74cb9"
Sep 10 03:18:05.947900 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:05.947870    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv"]
Sep 10 03:18:05.950222 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:18:05.950184    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad7c90c_9e2e_4bd3_b01f_00236dde69da.slice/crio-03bfbd40d49768f2b8e7804c87ed09eef9dc85eb851647d79adb70405d713006 WatchSource:0}: Error finding container 03bfbd40d49768f2b8e7804c87ed09eef9dc85eb851647d79adb70405d713006: Status 404 returned error can't find the container with id 03bfbd40d49768f2b8e7804c87ed09eef9dc85eb851647d79adb70405d713006
Sep 10 03:18:06.095699 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:06.095670    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c970f3-6bf9c9c56b-74cb9"]
Sep 10 03:18:06.098490 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:18:06.098442    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd97aa1_a0fe_48d2_8b19_72f5769880a8.slice/crio-8be09f88869141d6eb549917679ac215317b1979da2efd13f18238d56200d385 WatchSource:0}: Error finding container 8be09f88869141d6eb549917679ac215317b1979da2efd13f18238d56200d385: Status 404 returned error can't find the container with id 8be09f88869141d6eb549917679ac215317b1979da2efd13f18238d56200d385
Sep 10 03:18:06.769138 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:06.769053    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv" event={"ID":"7ad7c90c-9e2e-4bd3-b01f-00236dde69da","Type":"ContainerStarted","Data":"5fdf815835f5e47145893a958c387e80f6a08843f1e69b561902a591861afa2a"}
Sep 10 03:18:06.769361 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:06.769201    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv"
Sep 10 03:18:06.769361 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:06.769232    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv" event={"ID":"7ad7c90c-9e2e-4bd3-b01f-00236dde69da","Type":"ContainerStarted","Data":"03bfbd40d49768f2b8e7804c87ed09eef9dc85eb851647d79adb70405d713006"}
Sep 10 03:18:06.770989 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:06.770957    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c970f3-6bf9c9c56b-74cb9" event={"ID":"6dd97aa1-a0fe-48d2-8b19-72f5769880a8","Type":"ContainerStarted","Data":"f7617243173c6a3f23a1eb94cc127821a7287343b43f444cdc6d436e3e7e4ae1"}
Sep 10 03:18:06.771175 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:06.771012    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c970f3-6bf9c9c56b-74cb9"
Sep 10 03:18:06.771175 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:06.771028    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c970f3-6bf9c9c56b-74cb9" event={"ID":"6dd97aa1-a0fe-48d2-8b19-72f5769880a8","Type":"ContainerStarted","Data":"8be09f88869141d6eb549917679ac215317b1979da2efd13f18238d56200d385"}
Sep 10 03:18:06.788079 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:06.788004    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv" podStartSLOduration=2.165205763 podStartE2EDuration="2.787982893s" totalImagesPullingTime="622.77713ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:18:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:18:06.787342727 +0000 UTC m=+1877.162428649" watchObservedRunningTime="2026-09-10 03:18:06.787982893 +0000 UTC m=+1877.163068815"
Sep 10 03:18:06.807643 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:06.807571    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c970f3-6bf9c9c56b-74cb9" podStartSLOduration=1.807549776 podStartE2EDuration="1.807549776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:18:06.804807708 +0000 UTC m=+1877.179893669" watchObservedRunningTime="2026-09-10 03:18:06.807549776 +0000 UTC m=+1877.182635698"
Sep 10 03:18:12.635608 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:12.635568    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:18:12.784954 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:12.784873    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv"
Sep 10 03:18:17.780311 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:17.780271    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c970f3-6bf9c9c56b-74cb9"
Sep 10 03:18:28.479164 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:28.479097    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:18:33.511023 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:33.510988    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-twvbt_7d5508ae-a8b6-4aa6-a63f-112e565afc69/payload-processing/0.log"
Sep 10 03:18:35.275022 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:35.274987    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:18:51.511299 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:18:51.511263    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:19:09.518807 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:09.518754    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-c9cef5-d578999ff-7h2vd_2de8d4b8-4fa7-46db-9390-6fa3b4249971/payload-processing/0.log"
Sep 10 03:19:09.643627 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:09.643590    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-twvbt_7d5508ae-a8b6-4aa6-a63f-112e565afc69/payload-processing/0.log"
Sep 10 03:19:10.042847 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:10.042812    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:19:10.329462 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:10.329375    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:19:10.634197 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:10.634087    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:19:13.855548 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:13.855510    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:19:13.950315 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:13.950281    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:19:15.053427 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:15.053385    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54"]
Sep 10 03:19:15.057312 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:15.057281    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54"
Sep 10 03:19:15.059724 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:15.059692    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-be281e-serving-cert\""
Sep 10 03:19:15.067225 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:15.067186    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54"]
Sep 10 03:19:15.084921 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:15.084878    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:19:15.225627 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:15.225580    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56fb4\" (UniqueName: \"kubernetes.io/projected/5cb24320-6749-48e4-a6d4-70564b121f00-kube-api-access-56fb4\") pod \"maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54\" (UID: \"5cb24320-6749-48e4-a6d4-70564b121f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54"
Sep 10 03:19:15.225860 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:15.225652    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cb24320-6749-48e4-a6d4-70564b121f00-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54\" (UID: \"5cb24320-6749-48e4-a6d4-70564b121f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54"
Sep 10 03:19:15.225860 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:15.225681    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cb24320-6749-48e4-a6d4-70564b121f00-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54\" (UID: \"5cb24320-6749-48e4-a6d4-70564b121f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54"
Sep 10 03:19:15.225860 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:15.225810    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cb24320-6749-48e4-a6d4-70564b121f00-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54\" (UID: \"5cb24320-6749-48e4-a6d4-70564b121f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54"
Sep 10 03:19:15.252790 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:15.252756    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:19:15.327150 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:15.326994    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56fb4\" (UniqueName: \"kubernetes.io/projected/5cb24320-6749-48e4-a6d4-70564b121f00-kube-api-access-56fb4\") pod \"maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54\" (UID: \"5cb24320-6749-48e4-a6d4-70564b121f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54"
Sep 10 03:19:15.327150 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:15.327070    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cb24320-6749-48e4-a6d4-70564b121f00-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54\" (UID: \"5cb24320-6749-48e4-a6d4-70564b121f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54"
Sep 10 03:19:15.327150 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:15.327098    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cb24320-6749-48e4-a6d4-70564b121f00-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54\" (UID: \"5cb24320-6749-48e4-a6d4-70564b121f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54"
Sep 10 03:19:15.327509 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:19:15.327212    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-be281e-serving-cert: secret "maas-api-e2e-ipp-cleanup-be281e-serving-cert" not found
Sep 10 03:19:15.327509 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:15.327215    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cb24320-6749-48e4-a6d4-70564b121f00-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54\" (UID: \"5cb24320-6749-48e4-a6d4-70564b121f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54"
Sep 10 03:19:15.327509 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:19:15.327301    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cb24320-6749-48e4-a6d4-70564b121f00-maas-api-tls podName:5cb24320-6749-48e4-a6d4-70564b121f00 nodeName:}" failed. No retries permitted until 2026-09-10 03:19:15.827275567 +0000 UTC m=+1946.202361466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5cb24320-6749-48e4-a6d4-70564b121f00-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54" (UID: "5cb24320-6749-48e4-a6d4-70564b121f00") : secret "maas-api-e2e-ipp-cleanup-be281e-serving-cert" not found
Sep 10 03:19:15.327919 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:15.327895    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cb24320-6749-48e4-a6d4-70564b121f00-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54\" (UID: \"5cb24320-6749-48e4-a6d4-70564b121f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54"
Sep 10 03:19:15.329881 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:15.329853    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cb24320-6749-48e4-a6d4-70564b121f00-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54\" (UID: \"5cb24320-6749-48e4-a6d4-70564b121f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54"
Sep 10 03:19:15.338014 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:15.337990    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56fb4\" (UniqueName: \"kubernetes.io/projected/5cb24320-6749-48e4-a6d4-70564b121f00-kube-api-access-56fb4\") pod \"maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54\" (UID: \"5cb24320-6749-48e4-a6d4-70564b121f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54"
Sep 10 03:19:15.382824 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:15.382784    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:19:15.577430 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:15.577333    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:19:15.673173 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:15.673135    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:19:15.717996 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:15.717949    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-be281e-78bcb8d77-8wlz2"]
Sep 10 03:19:15.721694 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:15.721671    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-be281e-78bcb8d77-8wlz2"
Sep 10 03:19:15.723794 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:15.723772    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-be281e-dockercfg-6mpwz\""
Sep 10 03:19:15.723922 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:15.723838    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-be281e\""
Sep 10 03:19:15.733480 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:15.733445    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-be281e-78bcb8d77-8wlz2"]
Sep 10 03:19:15.831714 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:15.831588    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4kl9\" (UniqueName: \"kubernetes.io/projected/a66208ee-3ea7-477b-9142-9c1986dfa47a-kube-api-access-j4kl9\") pod \"payload-processing-e2e-ipp-cleanup-be281e-78bcb8d77-8wlz2\" (UID: \"a66208ee-3ea7-477b-9142-9c1986dfa47a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-be281e-78bcb8d77-8wlz2"
Sep 10 03:19:15.831920 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:15.831771    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cb24320-6749-48e4-a6d4-70564b121f00-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54\" (UID: \"5cb24320-6749-48e4-a6d4-70564b121f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54"
Sep 10 03:19:15.831920 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:15.831828    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a66208ee-3ea7-477b-9142-9c1986dfa47a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-be281e-78bcb8d77-8wlz2\" (UID: \"a66208ee-3ea7-477b-9142-9c1986dfa47a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-be281e-78bcb8d77-8wlz2"
Sep 10 03:19:15.834408 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:15.834383    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cb24320-6749-48e4-a6d4-70564b121f00-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54\" (UID: \"5cb24320-6749-48e4-a6d4-70564b121f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54"
Sep 10 03:19:15.933360 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:15.933310    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a66208ee-3ea7-477b-9142-9c1986dfa47a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-be281e-78bcb8d77-8wlz2\" (UID: \"a66208ee-3ea7-477b-9142-9c1986dfa47a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-be281e-78bcb8d77-8wlz2"
Sep 10 03:19:15.933593 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:15.933389    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4kl9\" (UniqueName: \"kubernetes.io/projected/a66208ee-3ea7-477b-9142-9c1986dfa47a-kube-api-access-j4kl9\") pod \"payload-processing-e2e-ipp-cleanup-be281e-78bcb8d77-8wlz2\" (UID: \"a66208ee-3ea7-477b-9142-9c1986dfa47a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-be281e-78bcb8d77-8wlz2"
Sep 10 03:19:15.934139 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:15.934079    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a66208ee-3ea7-477b-9142-9c1986dfa47a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-be281e-78bcb8d77-8wlz2\" (UID: \"a66208ee-3ea7-477b-9142-9c1986dfa47a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-be281e-78bcb8d77-8wlz2"
Sep 10 03:19:15.941294 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:15.941271    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4kl9\" (UniqueName: \"kubernetes.io/projected/a66208ee-3ea7-477b-9142-9c1986dfa47a-kube-api-access-j4kl9\") pod \"payload-processing-e2e-ipp-cleanup-be281e-78bcb8d77-8wlz2\" (UID: \"a66208ee-3ea7-477b-9142-9c1986dfa47a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-be281e-78bcb8d77-8wlz2"
Sep 10 03:19:15.973898 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:15.973852    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54"
Sep 10 03:19:16.035823 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:16.035784    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-be281e-78bcb8d77-8wlz2"
Sep 10 03:19:16.257558 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:16.257525    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54"]
Sep 10 03:19:16.260808 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:19:16.260763    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cb24320_6749_48e4_a6d4_70564b121f00.slice/crio-209442fe2171955344e7ccc0f89f3eb460a23b0790d55e64c2f598904955b389 WatchSource:0}: Error finding container 209442fe2171955344e7ccc0f89f3eb460a23b0790d55e64c2f598904955b389: Status 404 returned error can't find the container with id 209442fe2171955344e7ccc0f89f3eb460a23b0790d55e64c2f598904955b389
Sep 10 03:19:16.320926 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:16.320900    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-be281e-78bcb8d77-8wlz2"]
Sep 10 03:19:16.322597 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:19:16.322567    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda66208ee_3ea7_477b_9142_9c1986dfa47a.slice/crio-2c3e0d7760c1d2d929b02f3c1eb26adb16a70b756a2befd406e336c246dace40 WatchSource:0}: Error finding container 2c3e0d7760c1d2d929b02f3c1eb26adb16a70b756a2befd406e336c246dace40: Status 404 returned error can't find the container with id 2c3e0d7760c1d2d929b02f3c1eb26adb16a70b756a2befd406e336c246dace40
Sep 10 03:19:17.205544 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:17.205505    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-be281e-78bcb8d77-8wlz2" event={"ID":"a66208ee-3ea7-477b-9142-9c1986dfa47a","Type":"ContainerStarted","Data":"f6886014b8c46dd50391bbd415a1f11fc1a111ce1c4f930f8d93d90ae48a14fb"}
Sep 10 03:19:17.205784 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:17.205574    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-be281e-78bcb8d77-8wlz2" event={"ID":"a66208ee-3ea7-477b-9142-9c1986dfa47a","Type":"ContainerStarted","Data":"2c3e0d7760c1d2d929b02f3c1eb26adb16a70b756a2befd406e336c246dace40"}
Sep 10 03:19:17.205784 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:17.205721    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-be281e-78bcb8d77-8wlz2"
Sep 10 03:19:17.207331 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:17.207304    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54" event={"ID":"5cb24320-6749-48e4-a6d4-70564b121f00","Type":"ContainerStarted","Data":"36b6d25d7691a11892f968beeda2ec8ee827deb878b104757154f5e2a17f73a3"}
Sep 10 03:19:17.207479 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:17.207353    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54" event={"ID":"5cb24320-6749-48e4-a6d4-70564b121f00","Type":"ContainerStarted","Data":"209442fe2171955344e7ccc0f89f3eb460a23b0790d55e64c2f598904955b389"}
Sep 10 03:19:17.207479 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:17.207388    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54"
Sep 10 03:19:17.243747 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:17.243683    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54" podStartSLOduration=1.912548549 podStartE2EDuration="2.243662767s" totalImagesPullingTime="331.114218ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:19:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:19:17.242243802 +0000 UTC m=+1947.617329723" watchObservedRunningTime="2026-09-10 03:19:17.243662767 +0000 UTC m=+1947.618748688"
Sep 10 03:19:17.245294 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:17.245255    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-be281e-78bcb8d77-8wlz2" podStartSLOduration=2.245244714 podStartE2EDuration="2.245244714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:19:17.222765369 +0000 UTC m=+1947.597851290" watchObservedRunningTime="2026-09-10 03:19:17.245244714 +0000 UTC m=+1947.620330634"
Sep 10 03:19:23.219948 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:23.219910    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54"
Sep 10 03:19:28.217542 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:28.217506    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-be281e-78bcb8d77-8wlz2"
Sep 10 03:19:38.839707 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:38.839664    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:19:45.458027 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:45.457822    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54"]
Sep 10 03:19:45.458986 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:45.458945    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54" podUID="5cb24320-6749-48e4-a6d4-70564b121f00" containerName="maas-api" containerID="cri-o://36b6d25d7691a11892f968beeda2ec8ee827deb878b104757154f5e2a17f73a3" gracePeriod=30
Sep 10 03:19:45.478792 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:45.478670    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:19:45.501798 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:45.501758    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-be281e-78bcb8d77-8wlz2"]
Sep 10 03:19:45.502067 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:45.502039    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-be281e-78bcb8d77-8wlz2" podUID="a66208ee-3ea7-477b-9142-9c1986dfa47a" containerName="payload-processing" containerID="cri-o://f6886014b8c46dd50391bbd415a1f11fc1a111ce1c4f930f8d93d90ae48a14fb" gracePeriod=30
Sep 10 03:19:45.579240 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:45.579205    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:19:45.650320 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:45.650272    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:19:45.708804 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:45.708654    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:19:45.784522 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:45.784492    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54"
Sep 10 03:19:45.803163 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:45.803133    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-be281e-78bcb8d77-8wlz2"
Sep 10 03:19:45.893558 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:45.893522    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a66208ee-3ea7-477b-9142-9c1986dfa47a-plugins-config-volume\") pod \"a66208ee-3ea7-477b-9142-9c1986dfa47a\" (UID: \"a66208ee-3ea7-477b-9142-9c1986dfa47a\") "
Sep 10 03:19:45.893775 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:45.893572    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cb24320-6749-48e4-a6d4-70564b121f00-maas-api-tls\") pod \"5cb24320-6749-48e4-a6d4-70564b121f00\" (UID: \"5cb24320-6749-48e4-a6d4-70564b121f00\") "
Sep 10 03:19:45.893775 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:45.893644    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4kl9\" (UniqueName: \"kubernetes.io/projected/a66208ee-3ea7-477b-9142-9c1986dfa47a-kube-api-access-j4kl9\") pod \"a66208ee-3ea7-477b-9142-9c1986dfa47a\" (UID: \"a66208ee-3ea7-477b-9142-9c1986dfa47a\") "
Sep 10 03:19:45.893775 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:45.893720    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cb24320-6749-48e4-a6d4-70564b121f00-openshift-service-ca\") pod \"5cb24320-6749-48e4-a6d4-70564b121f00\" (UID: \"5cb24320-6749-48e4-a6d4-70564b121f00\") "
Sep 10 03:19:45.893775 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:45.893755    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56fb4\" (UniqueName: \"kubernetes.io/projected/5cb24320-6749-48e4-a6d4-70564b121f00-kube-api-access-56fb4\") pod \"5cb24320-6749-48e4-a6d4-70564b121f00\" (UID: \"5cb24320-6749-48e4-a6d4-70564b121f00\") "
Sep 10 03:19:45.893775 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:45.893771    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cb24320-6749-48e4-a6d4-70564b121f00-metrics-certs\") pod \"5cb24320-6749-48e4-a6d4-70564b121f00\" (UID: \"5cb24320-6749-48e4-a6d4-70564b121f00\") "
Sep 10 03:19:45.894161 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:45.894092    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a66208ee-3ea7-477b-9142-9c1986dfa47a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a66208ee-3ea7-477b-9142-9c1986dfa47a" (UID: "a66208ee-3ea7-477b-9142-9c1986dfa47a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:19:45.894244 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:45.894182    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cb24320-6749-48e4-a6d4-70564b121f00-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5cb24320-6749-48e4-a6d4-70564b121f00" (UID: "5cb24320-6749-48e4-a6d4-70564b121f00"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:19:45.896032 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:45.895998    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a66208ee-3ea7-477b-9142-9c1986dfa47a-kube-api-access-j4kl9" (OuterVolumeSpecName: "kube-api-access-j4kl9") pod "a66208ee-3ea7-477b-9142-9c1986dfa47a" (UID: "a66208ee-3ea7-477b-9142-9c1986dfa47a"). InnerVolumeSpecName "kube-api-access-j4kl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:19:45.896166 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:45.896035    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cb24320-6749-48e4-a6d4-70564b121f00-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5cb24320-6749-48e4-a6d4-70564b121f00" (UID: "5cb24320-6749-48e4-a6d4-70564b121f00"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:19:45.896477 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:45.896437    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cb24320-6749-48e4-a6d4-70564b121f00-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5cb24320-6749-48e4-a6d4-70564b121f00" (UID: "5cb24320-6749-48e4-a6d4-70564b121f00"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:19:45.896583 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:45.896449    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cb24320-6749-48e4-a6d4-70564b121f00-kube-api-access-56fb4" (OuterVolumeSpecName: "kube-api-access-56fb4") pod "5cb24320-6749-48e4-a6d4-70564b121f00" (UID: "5cb24320-6749-48e4-a6d4-70564b121f00"). InnerVolumeSpecName "kube-api-access-56fb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:19:45.994800 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:45.994690    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4kl9\" (UniqueName: \"kubernetes.io/projected/a66208ee-3ea7-477b-9142-9c1986dfa47a-kube-api-access-j4kl9\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:19:45.994800 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:45.994725    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cb24320-6749-48e4-a6d4-70564b121f00-openshift-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:19:45.994800 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:45.994738    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56fb4\" (UniqueName: \"kubernetes.io/projected/5cb24320-6749-48e4-a6d4-70564b121f00-kube-api-access-56fb4\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:19:45.994800 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:45.994749    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cb24320-6749-48e4-a6d4-70564b121f00-metrics-certs\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:19:45.994800 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:45.994758    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a66208ee-3ea7-477b-9142-9c1986dfa47a-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:19:45.994800 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:45.994768    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cb24320-6749-48e4-a6d4-70564b121f00-maas-api-tls\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:19:46.006891 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:46.006854    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:19:46.387497 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:46.387404    2558 generic.go:356] "Generic (PLEG): container finished" podID="a66208ee-3ea7-477b-9142-9c1986dfa47a" containerID="f6886014b8c46dd50391bbd415a1f11fc1a111ce1c4f930f8d93d90ae48a14fb" exitCode=0
Sep 10 03:19:46.387497 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:46.387454    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-be281e-78bcb8d77-8wlz2"
Sep 10 03:19:46.387497 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:46.387484    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-be281e-78bcb8d77-8wlz2" event={"ID":"a66208ee-3ea7-477b-9142-9c1986dfa47a","Type":"ContainerDied","Data":"f6886014b8c46dd50391bbd415a1f11fc1a111ce1c4f930f8d93d90ae48a14fb"}
Sep 10 03:19:46.387806 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:46.387556    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-be281e-78bcb8d77-8wlz2" event={"ID":"a66208ee-3ea7-477b-9142-9c1986dfa47a","Type":"ContainerDied","Data":"2c3e0d7760c1d2d929b02f3c1eb26adb16a70b756a2befd406e336c246dace40"}
Sep 10 03:19:46.387806 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:46.387588    2558 scope.go:117] "RemoveContainer" containerID="f6886014b8c46dd50391bbd415a1f11fc1a111ce1c4f930f8d93d90ae48a14fb"
Sep 10 03:19:46.389133 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:46.389089    2558 generic.go:356] "Generic (PLEG): container finished" podID="5cb24320-6749-48e4-a6d4-70564b121f00" containerID="36b6d25d7691a11892f968beeda2ec8ee827deb878b104757154f5e2a17f73a3" exitCode=0
Sep 10 03:19:46.389263 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:46.389131    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54" event={"ID":"5cb24320-6749-48e4-a6d4-70564b121f00","Type":"ContainerDied","Data":"36b6d25d7691a11892f968beeda2ec8ee827deb878b104757154f5e2a17f73a3"}
Sep 10 03:19:46.389263 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:46.389163    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54"
Sep 10 03:19:46.389263 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:46.389179    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54" event={"ID":"5cb24320-6749-48e4-a6d4-70564b121f00","Type":"ContainerDied","Data":"209442fe2171955344e7ccc0f89f3eb460a23b0790d55e64c2f598904955b389"}
Sep 10 03:19:46.398166 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:46.398138    2558 scope.go:117] "RemoveContainer" containerID="f6886014b8c46dd50391bbd415a1f11fc1a111ce1c4f930f8d93d90ae48a14fb"
Sep 10 03:19:46.398521 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:19:46.398502    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6886014b8c46dd50391bbd415a1f11fc1a111ce1c4f930f8d93d90ae48a14fb\": container with ID starting with f6886014b8c46dd50391bbd415a1f11fc1a111ce1c4f930f8d93d90ae48a14fb not found: ID does not exist" containerID="f6886014b8c46dd50391bbd415a1f11fc1a111ce1c4f930f8d93d90ae48a14fb"
Sep 10 03:19:46.398602 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:46.398533    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6886014b8c46dd50391bbd415a1f11fc1a111ce1c4f930f8d93d90ae48a14fb"} err="failed to get container status \"f6886014b8c46dd50391bbd415a1f11fc1a111ce1c4f930f8d93d90ae48a14fb\": rpc error: code = NotFound desc = could not find container \"f6886014b8c46dd50391bbd415a1f11fc1a111ce1c4f930f8d93d90ae48a14fb\": container with ID starting with f6886014b8c46dd50391bbd415a1f11fc1a111ce1c4f930f8d93d90ae48a14fb not found: ID does not exist"
Sep 10 03:19:46.398602 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:46.398554    2558 scope.go:117] "RemoveContainer" containerID="36b6d25d7691a11892f968beeda2ec8ee827deb878b104757154f5e2a17f73a3"
Sep 10 03:19:46.406292 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:46.406258    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-be281e-78bcb8d77-8wlz2"]
Sep 10 03:19:46.407917 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:46.407893    2558 scope.go:117] "RemoveContainer" containerID="36b6d25d7691a11892f968beeda2ec8ee827deb878b104757154f5e2a17f73a3"
Sep 10 03:19:46.408477 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:19:46.408452    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36b6d25d7691a11892f968beeda2ec8ee827deb878b104757154f5e2a17f73a3\": container with ID starting with 36b6d25d7691a11892f968beeda2ec8ee827deb878b104757154f5e2a17f73a3 not found: ID does not exist" containerID="36b6d25d7691a11892f968beeda2ec8ee827deb878b104757154f5e2a17f73a3"
Sep 10 03:19:46.408559 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:46.408483    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36b6d25d7691a11892f968beeda2ec8ee827deb878b104757154f5e2a17f73a3"} err="failed to get container status \"36b6d25d7691a11892f968beeda2ec8ee827deb878b104757154f5e2a17f73a3\": rpc error: code = NotFound desc = could not find container \"36b6d25d7691a11892f968beeda2ec8ee827deb878b104757154f5e2a17f73a3\": container with ID starting with 36b6d25d7691a11892f968beeda2ec8ee827deb878b104757154f5e2a17f73a3 not found: ID does not exist"
Sep 10 03:19:46.411083 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:46.411052    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-be281e-78bcb8d77-8wlz2"]
Sep 10 03:19:46.422380 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:46.422343    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54"]
Sep 10 03:19:46.426017 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:46.425981    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-be281e-fd9cdbd76-w5v54"]
Sep 10 03:19:48.069035 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:48.068998    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cb24320-6749-48e4-a6d4-70564b121f00" path="/var/lib/kubelet/pods/5cb24320-6749-48e4-a6d4-70564b121f00/volumes"
Sep 10 03:19:48.069438 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:48.069427    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a66208ee-3ea7-477b-9142-9c1986dfa47a" path="/var/lib/kubelet/pods/a66208ee-3ea7-477b-9142-9c1986dfa47a/volumes"
Sep 10 03:19:50.566179 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:50.566140    2558 scope.go:117] "RemoveContainer" containerID="0db224d05ac744e0d775210a8917395a6307e0144892eb4f1c5b2f68c67618d9"
Sep 10 03:19:50.575908 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:50.575885    2558 scope.go:117] "RemoveContainer" containerID="789b7558363dbfdc1b01c86db9f129bf0cb74c402535e41c8b0a6bdeacd77e56"
Sep 10 03:19:50.967057 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:50.967020    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:19:51.022459 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:51.022421    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:19:51.893612 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:51.893556    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:19:51.966801 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:51.966757    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:19:52.246861 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:52.246824    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:19:56.168868 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:19:56.168826    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:20:11.440712 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:11.440665    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d"]
Sep 10 03:20:11.441345 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:11.441309    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d" podUID="c6dc7868-800f-46fa-af8e-103b4ea82d36" containerName="maas-api" containerID="cri-o://5b0a6991436efc88fe51aef80644242e697376c16c8e2e9fd281e61cba801e22" gracePeriod=30
Sep 10 03:20:11.475315 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:11.475281    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:20:11.493421 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:11.493368    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c9cef5-d578999ff-7h2vd"]
Sep 10 03:20:11.493697 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:11.493645    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c9cef5-d578999ff-7h2vd" podUID="2de8d4b8-4fa7-46db-9390-6fa3b4249971" containerName="payload-processing" containerID="cri-o://ce7eeb800a9053781fcb6461fe9feac82703cc6fca195dfe9361b5654787e1ac" gracePeriod=30
Sep 10 03:20:11.498331 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:11.498292    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c9cef5-5bb46b7fb6-gc2m8"]
Sep 10 03:20:11.498608 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:11.498550    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9cef5-5bb46b7fb6-gc2m8" podUID="3a678a31-a511-4f52-9890-ec6026611cbe" containerName="payload-pre-processing" containerID="cri-o://f2bf3f74ee089400488ab268dd21076acc4c1b0a8abd007eb69acf7699f6b4a3" gracePeriod=30
Sep 10 03:20:11.578802 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:11.578760    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:20:11.650894 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:11.650851    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:20:11.715192 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:11.715134    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:20:11.780608 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:11.780574    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d"
Sep 10 03:20:11.784234 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:11.784203    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c9cef5-d578999ff-7h2vd"
Sep 10 03:20:11.807431 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:11.807396    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9cef5-5bb46b7fb6-gc2m8"
Sep 10 03:20:11.932427 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:11.932384    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a678a31-a511-4f52-9890-ec6026611cbe-plugins-config-volume\") pod \"3a678a31-a511-4f52-9890-ec6026611cbe\" (UID: \"3a678a31-a511-4f52-9890-ec6026611cbe\") "
Sep 10 03:20:11.932427 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:11.932427    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de8d4b8-4fa7-46db-9390-6fa3b4249971-plugins-config-volume\") pod \"2de8d4b8-4fa7-46db-9390-6fa3b4249971\" (UID: \"2de8d4b8-4fa7-46db-9390-6fa3b4249971\") "
Sep 10 03:20:11.932700 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:11.932456    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6dc7868-800f-46fa-af8e-103b4ea82d36-metrics-certs\") pod \"c6dc7868-800f-46fa-af8e-103b4ea82d36\" (UID: \"c6dc7868-800f-46fa-af8e-103b4ea82d36\") "
Sep 10 03:20:11.932700 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:11.932489    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6dc7868-800f-46fa-af8e-103b4ea82d36-openshift-service-ca\") pod \"c6dc7868-800f-46fa-af8e-103b4ea82d36\" (UID: \"c6dc7868-800f-46fa-af8e-103b4ea82d36\") "
Sep 10 03:20:11.932700 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:11.932617    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcnhr\" (UniqueName: \"kubernetes.io/projected/2de8d4b8-4fa7-46db-9390-6fa3b4249971-kube-api-access-kcnhr\") pod \"2de8d4b8-4fa7-46db-9390-6fa3b4249971\" (UID: \"2de8d4b8-4fa7-46db-9390-6fa3b4249971\") "
Sep 10 03:20:11.932867 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:11.932712    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7whwq\" (UniqueName: \"kubernetes.io/projected/c6dc7868-800f-46fa-af8e-103b4ea82d36-kube-api-access-7whwq\") pod \"c6dc7868-800f-46fa-af8e-103b4ea82d36\" (UID: \"c6dc7868-800f-46fa-af8e-103b4ea82d36\") "
Sep 10 03:20:11.932867 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:11.932800    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6dc7868-800f-46fa-af8e-103b4ea82d36-maas-api-tls\") pod \"c6dc7868-800f-46fa-af8e-103b4ea82d36\" (UID: \"c6dc7868-800f-46fa-af8e-103b4ea82d36\") "
Sep 10 03:20:11.932867 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:11.932837    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6dc7868-800f-46fa-af8e-103b4ea82d36-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c6dc7868-800f-46fa-af8e-103b4ea82d36" (UID: "c6dc7868-800f-46fa-af8e-103b4ea82d36"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:20:11.933027 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:11.932858    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a678a31-a511-4f52-9890-ec6026611cbe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3a678a31-a511-4f52-9890-ec6026611cbe" (UID: "3a678a31-a511-4f52-9890-ec6026611cbe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:20:11.933027 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:11.932846    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwnbp\" (UniqueName: \"kubernetes.io/projected/3a678a31-a511-4f52-9890-ec6026611cbe-kube-api-access-rwnbp\") pod \"3a678a31-a511-4f52-9890-ec6026611cbe\" (UID: \"3a678a31-a511-4f52-9890-ec6026611cbe\") "
Sep 10 03:20:11.933027 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:11.932875    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2de8d4b8-4fa7-46db-9390-6fa3b4249971-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2de8d4b8-4fa7-46db-9390-6fa3b4249971" (UID: "2de8d4b8-4fa7-46db-9390-6fa3b4249971"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:20:11.933283 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:11.933257    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a678a31-a511-4f52-9890-ec6026611cbe-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:20:11.933410 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:11.933286    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de8d4b8-4fa7-46db-9390-6fa3b4249971-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:20:11.933410 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:11.933304    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6dc7868-800f-46fa-af8e-103b4ea82d36-openshift-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:20:11.934983 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:11.934957    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6dc7868-800f-46fa-af8e-103b4ea82d36-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c6dc7868-800f-46fa-af8e-103b4ea82d36" (UID: "c6dc7868-800f-46fa-af8e-103b4ea82d36"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:20:11.934983 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:11.934974    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a678a31-a511-4f52-9890-ec6026611cbe-kube-api-access-rwnbp" (OuterVolumeSpecName: "kube-api-access-rwnbp") pod "3a678a31-a511-4f52-9890-ec6026611cbe" (UID: "3a678a31-a511-4f52-9890-ec6026611cbe"). InnerVolumeSpecName "kube-api-access-rwnbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:20:11.934983 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:11.934957    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6dc7868-800f-46fa-af8e-103b4ea82d36-kube-api-access-7whwq" (OuterVolumeSpecName: "kube-api-access-7whwq") pod "c6dc7868-800f-46fa-af8e-103b4ea82d36" (UID: "c6dc7868-800f-46fa-af8e-103b4ea82d36"). InnerVolumeSpecName "kube-api-access-7whwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:20:11.935261 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:11.935244    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2de8d4b8-4fa7-46db-9390-6fa3b4249971-kube-api-access-kcnhr" (OuterVolumeSpecName: "kube-api-access-kcnhr") pod "2de8d4b8-4fa7-46db-9390-6fa3b4249971" (UID: "2de8d4b8-4fa7-46db-9390-6fa3b4249971"). InnerVolumeSpecName "kube-api-access-kcnhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:20:11.935320 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:11.935304    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6dc7868-800f-46fa-af8e-103b4ea82d36-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c6dc7868-800f-46fa-af8e-103b4ea82d36" (UID: "c6dc7868-800f-46fa-af8e-103b4ea82d36"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:20:12.020439 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:12.020387    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:20:12.034710 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:12.034663    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6dc7868-800f-46fa-af8e-103b4ea82d36-metrics-certs\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:20:12.034710 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:12.034701    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcnhr\" (UniqueName: \"kubernetes.io/projected/2de8d4b8-4fa7-46db-9390-6fa3b4249971-kube-api-access-kcnhr\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:20:12.034710 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:12.034711    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7whwq\" (UniqueName: \"kubernetes.io/projected/c6dc7868-800f-46fa-af8e-103b4ea82d36-kube-api-access-7whwq\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:20:12.034949 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:12.034721    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6dc7868-800f-46fa-af8e-103b4ea82d36-maas-api-tls\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:20:12.034949 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:12.034732    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwnbp\" (UniqueName: \"kubernetes.io/projected/3a678a31-a511-4f52-9890-ec6026611cbe-kube-api-access-rwnbp\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:20:12.554650 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:12.554610    2558 generic.go:356] "Generic (PLEG): container finished" podID="c6dc7868-800f-46fa-af8e-103b4ea82d36" containerID="5b0a6991436efc88fe51aef80644242e697376c16c8e2e9fd281e61cba801e22" exitCode=0
Sep 10 03:20:12.555179 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:12.554666    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d"
Sep 10 03:20:12.555179 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:12.554700    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d" event={"ID":"c6dc7868-800f-46fa-af8e-103b4ea82d36","Type":"ContainerDied","Data":"5b0a6991436efc88fe51aef80644242e697376c16c8e2e9fd281e61cba801e22"}
Sep 10 03:20:12.555179 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:12.554791    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d" event={"ID":"c6dc7868-800f-46fa-af8e-103b4ea82d36","Type":"ContainerDied","Data":"b5026c85f0111a0ec5f8cd477b4d98e0d44e8d194d91b09f45e9209ebf956fc9"}
Sep 10 03:20:12.555179 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:12.554821    2558 scope.go:117] "RemoveContainer" containerID="5b0a6991436efc88fe51aef80644242e697376c16c8e2e9fd281e61cba801e22"
Sep 10 03:20:12.556634 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:12.556606    2558 generic.go:356] "Generic (PLEG): container finished" podID="2de8d4b8-4fa7-46db-9390-6fa3b4249971" containerID="ce7eeb800a9053781fcb6461fe9feac82703cc6fca195dfe9361b5654787e1ac" exitCode=0
Sep 10 03:20:12.556757 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:12.556698    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c9cef5-d578999ff-7h2vd" event={"ID":"2de8d4b8-4fa7-46db-9390-6fa3b4249971","Type":"ContainerDied","Data":"ce7eeb800a9053781fcb6461fe9feac82703cc6fca195dfe9361b5654787e1ac"}
Sep 10 03:20:12.556757 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:12.556748    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c9cef5-d578999ff-7h2vd" event={"ID":"2de8d4b8-4fa7-46db-9390-6fa3b4249971","Type":"ContainerDied","Data":"66979e9d114660976c72e7a016727c13e9e2e0c63f5901985b70686909f9e7fb"}
Sep 10 03:20:12.557006 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:12.556988    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c9cef5-d578999ff-7h2vd"
Sep 10 03:20:12.559665 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:12.559634    2558 generic.go:356] "Generic (PLEG): container finished" podID="3a678a31-a511-4f52-9890-ec6026611cbe" containerID="f2bf3f74ee089400488ab268dd21076acc4c1b0a8abd007eb69acf7699f6b4a3" exitCode=0
Sep 10 03:20:12.559830 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:12.559684    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9cef5-5bb46b7fb6-gc2m8" event={"ID":"3a678a31-a511-4f52-9890-ec6026611cbe","Type":"ContainerDied","Data":"f2bf3f74ee089400488ab268dd21076acc4c1b0a8abd007eb69acf7699f6b4a3"}
Sep 10 03:20:12.559830 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:12.559724    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9cef5-5bb46b7fb6-gc2m8" event={"ID":"3a678a31-a511-4f52-9890-ec6026611cbe","Type":"ContainerDied","Data":"9cc32c1da3e19b3f5a66dbe955e292ffad132aff1f200d0fcf63fd4647d055d7"}
Sep 10 03:20:12.559830 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:12.559807    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9cef5-5bb46b7fb6-gc2m8"
Sep 10 03:20:12.565992 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:12.565961    2558 scope.go:117] "RemoveContainer" containerID="5b0a6991436efc88fe51aef80644242e697376c16c8e2e9fd281e61cba801e22"
Sep 10 03:20:12.566402 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:20:12.566375    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b0a6991436efc88fe51aef80644242e697376c16c8e2e9fd281e61cba801e22\": container with ID starting with 5b0a6991436efc88fe51aef80644242e697376c16c8e2e9fd281e61cba801e22 not found: ID does not exist" containerID="5b0a6991436efc88fe51aef80644242e697376c16c8e2e9fd281e61cba801e22"
Sep 10 03:20:12.566516 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:12.566410    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b0a6991436efc88fe51aef80644242e697376c16c8e2e9fd281e61cba801e22"} err="failed to get container status \"5b0a6991436efc88fe51aef80644242e697376c16c8e2e9fd281e61cba801e22\": rpc error: code = NotFound desc = could not find container \"5b0a6991436efc88fe51aef80644242e697376c16c8e2e9fd281e61cba801e22\": container with ID starting with 5b0a6991436efc88fe51aef80644242e697376c16c8e2e9fd281e61cba801e22 not found: ID does not exist"
Sep 10 03:20:12.566516 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:12.566431    2558 scope.go:117] "RemoveContainer" containerID="ce7eeb800a9053781fcb6461fe9feac82703cc6fca195dfe9361b5654787e1ac"
Sep 10 03:20:12.573240 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:12.573205    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d"]
Sep 10 03:20:12.576560 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:12.576413    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9cef5-68454896ff-z724d"]
Sep 10 03:20:12.576637 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:12.576585    2558 scope.go:117] "RemoveContainer" containerID="ce7eeb800a9053781fcb6461fe9feac82703cc6fca195dfe9361b5654787e1ac"
Sep 10 03:20:12.576976 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:20:12.576950    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce7eeb800a9053781fcb6461fe9feac82703cc6fca195dfe9361b5654787e1ac\": container with ID starting with ce7eeb800a9053781fcb6461fe9feac82703cc6fca195dfe9361b5654787e1ac not found: ID does not exist" containerID="ce7eeb800a9053781fcb6461fe9feac82703cc6fca195dfe9361b5654787e1ac"
Sep 10 03:20:12.577089 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:12.576979    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce7eeb800a9053781fcb6461fe9feac82703cc6fca195dfe9361b5654787e1ac"} err="failed to get container status \"ce7eeb800a9053781fcb6461fe9feac82703cc6fca195dfe9361b5654787e1ac\": rpc error: code = NotFound desc = could not find container \"ce7eeb800a9053781fcb6461fe9feac82703cc6fca195dfe9361b5654787e1ac\": container with ID starting with ce7eeb800a9053781fcb6461fe9feac82703cc6fca195dfe9361b5654787e1ac not found: ID does not exist"
Sep 10 03:20:12.577089 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:12.576997    2558 scope.go:117] "RemoveContainer" containerID="f2bf3f74ee089400488ab268dd21076acc4c1b0a8abd007eb69acf7699f6b4a3"
Sep 10 03:20:12.586141 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:12.586092    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c9cef5-d578999ff-7h2vd"]
Sep 10 03:20:12.586561 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:12.586538    2558 scope.go:117] "RemoveContainer" containerID="f2bf3f74ee089400488ab268dd21076acc4c1b0a8abd007eb69acf7699f6b4a3"
Sep 10 03:20:12.587009 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:20:12.586974    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2bf3f74ee089400488ab268dd21076acc4c1b0a8abd007eb69acf7699f6b4a3\": container with ID starting with f2bf3f74ee089400488ab268dd21076acc4c1b0a8abd007eb69acf7699f6b4a3 not found: ID does not exist" containerID="f2bf3f74ee089400488ab268dd21076acc4c1b0a8abd007eb69acf7699f6b4a3"
Sep 10 03:20:12.587211 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:12.587179    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2bf3f74ee089400488ab268dd21076acc4c1b0a8abd007eb69acf7699f6b4a3"} err="failed to get container status \"f2bf3f74ee089400488ab268dd21076acc4c1b0a8abd007eb69acf7699f6b4a3\": rpc error: code = NotFound desc = could not find container \"f2bf3f74ee089400488ab268dd21076acc4c1b0a8abd007eb69acf7699f6b4a3\": container with ID starting with f2bf3f74ee089400488ab268dd21076acc4c1b0a8abd007eb69acf7699f6b4a3 not found: ID does not exist"
Sep 10 03:20:12.589559 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:12.589534    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c9cef5-d578999ff-7h2vd"]
Sep 10 03:20:12.598873 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:12.598839    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c9cef5-5bb46b7fb6-gc2m8"]
Sep 10 03:20:12.602361 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:12.602334    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c9cef5-5bb46b7fb6-gc2m8"]
Sep 10 03:20:14.068709 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:14.068676    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2de8d4b8-4fa7-46db-9390-6fa3b4249971" path="/var/lib/kubelet/pods/2de8d4b8-4fa7-46db-9390-6fa3b4249971/volumes"
Sep 10 03:20:14.069137 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:14.069046    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a678a31-a511-4f52-9890-ec6026611cbe" path="/var/lib/kubelet/pods/3a678a31-a511-4f52-9890-ec6026611cbe/volumes"
Sep 10 03:20:14.069456 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:14.069441    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6dc7868-800f-46fa-af8e-103b4ea82d36" path="/var/lib/kubelet/pods/c6dc7868-800f-46fa-af8e-103b4ea82d36/volumes"
Sep 10 03:20:17.096374 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:17.096336    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:20:52.639459 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:52.639422    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:20:52.739025 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:52.738992    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:20:53.303261 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:53.303215    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:20:54.277725 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:54.277680    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:20:59.766076 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:59.766034    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:20:59.836695 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:20:59.836659    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:21:00.036505 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:00.036417    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:21:00.574168 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:00.574127    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:21:10.849634 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:10.849576    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv"]
Sep 10 03:21:10.850666 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:10.850626    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv" podUID="7ad7c90c-9e2e-4bd3-b01f-00236dde69da" containerName="maas-api" containerID="cri-o://5fdf815835f5e47145893a958c387e80f6a08843f1e69b561902a591861afa2a" gracePeriod=30
Sep 10 03:21:10.874311 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:10.874257    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:21:10.914412 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:10.914359    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c970f3-6bf9c9c56b-74cb9"]
Sep 10 03:21:10.914689 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:10.914640    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c970f3-6bf9c9c56b-74cb9" podUID="6dd97aa1-a0fe-48d2-8b19-72f5769880a8" containerName="payload-pre-processing" containerID="cri-o://f7617243173c6a3f23a1eb94cc127821a7287343b43f444cdc6d436e3e7e4ae1" gracePeriod=30
Sep 10 03:21:10.975151 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:10.975117    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:21:11.076631 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.076591    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:21:11.146172 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.146138    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:21:11.168788 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.168757    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv"
Sep 10 03:21:11.205184 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.205147    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c970f3-6bf9c9c56b-74cb9"
Sep 10 03:21:11.357948 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.357901    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad7c90c-9e2e-4bd3-b01f-00236dde69da-openshift-service-ca\") pod \"7ad7c90c-9e2e-4bd3-b01f-00236dde69da\" (UID: \"7ad7c90c-9e2e-4bd3-b01f-00236dde69da\") "
Sep 10 03:21:11.358183 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.357971    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxwpk\" (UniqueName: \"kubernetes.io/projected/7ad7c90c-9e2e-4bd3-b01f-00236dde69da-kube-api-access-nxwpk\") pod \"7ad7c90c-9e2e-4bd3-b01f-00236dde69da\" (UID: \"7ad7c90c-9e2e-4bd3-b01f-00236dde69da\") "
Sep 10 03:21:11.358183 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.358038    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd97aa1-a0fe-48d2-8b19-72f5769880a8-plugins-config-volume\") pod \"6dd97aa1-a0fe-48d2-8b19-72f5769880a8\" (UID: \"6dd97aa1-a0fe-48d2-8b19-72f5769880a8\") "
Sep 10 03:21:11.358183 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.358088    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ad7c90c-9e2e-4bd3-b01f-00236dde69da-maas-api-tls\") pod \"7ad7c90c-9e2e-4bd3-b01f-00236dde69da\" (UID: \"7ad7c90c-9e2e-4bd3-b01f-00236dde69da\") "
Sep 10 03:21:11.358183 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.358157    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kftbb\" (UniqueName: \"kubernetes.io/projected/6dd97aa1-a0fe-48d2-8b19-72f5769880a8-kube-api-access-kftbb\") pod \"6dd97aa1-a0fe-48d2-8b19-72f5769880a8\" (UID: \"6dd97aa1-a0fe-48d2-8b19-72f5769880a8\") "
Sep 10 03:21:11.358491 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.358216    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ad7c90c-9e2e-4bd3-b01f-00236dde69da-metrics-certs\") pod \"7ad7c90c-9e2e-4bd3-b01f-00236dde69da\" (UID: \"7ad7c90c-9e2e-4bd3-b01f-00236dde69da\") "
Sep 10 03:21:11.358750 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.358380    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ad7c90c-9e2e-4bd3-b01f-00236dde69da-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7ad7c90c-9e2e-4bd3-b01f-00236dde69da" (UID: "7ad7c90c-9e2e-4bd3-b01f-00236dde69da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:21:11.359074 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.359028    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dd97aa1-a0fe-48d2-8b19-72f5769880a8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6dd97aa1-a0fe-48d2-8b19-72f5769880a8" (UID: "6dd97aa1-a0fe-48d2-8b19-72f5769880a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:21:11.360793 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.360759    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ad7c90c-9e2e-4bd3-b01f-00236dde69da-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7ad7c90c-9e2e-4bd3-b01f-00236dde69da" (UID: "7ad7c90c-9e2e-4bd3-b01f-00236dde69da"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:21:11.361006 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.360978    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad7c90c-9e2e-4bd3-b01f-00236dde69da-kube-api-access-nxwpk" (OuterVolumeSpecName: "kube-api-access-nxwpk") pod "7ad7c90c-9e2e-4bd3-b01f-00236dde69da" (UID: "7ad7c90c-9e2e-4bd3-b01f-00236dde69da"). InnerVolumeSpecName "kube-api-access-nxwpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:21:11.361154 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.361018    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dd97aa1-a0fe-48d2-8b19-72f5769880a8-kube-api-access-kftbb" (OuterVolumeSpecName: "kube-api-access-kftbb") pod "6dd97aa1-a0fe-48d2-8b19-72f5769880a8" (UID: "6dd97aa1-a0fe-48d2-8b19-72f5769880a8"). InnerVolumeSpecName "kube-api-access-kftbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:21:11.361154 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.361079    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ad7c90c-9e2e-4bd3-b01f-00236dde69da-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7ad7c90c-9e2e-4bd3-b01f-00236dde69da" (UID: "7ad7c90c-9e2e-4bd3-b01f-00236dde69da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:21:11.459116 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.459050    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd97aa1-a0fe-48d2-8b19-72f5769880a8-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:21:11.459314 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.459126    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ad7c90c-9e2e-4bd3-b01f-00236dde69da-maas-api-tls\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:21:11.459314 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.459144    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kftbb\" (UniqueName: \"kubernetes.io/projected/6dd97aa1-a0fe-48d2-8b19-72f5769880a8-kube-api-access-kftbb\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:21:11.459314 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.459157    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ad7c90c-9e2e-4bd3-b01f-00236dde69da-metrics-certs\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:21:11.459314 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.459170    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad7c90c-9e2e-4bd3-b01f-00236dde69da-openshift-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:21:11.459314 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.459181    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxwpk\" (UniqueName: \"kubernetes.io/projected/7ad7c90c-9e2e-4bd3-b01f-00236dde69da-kube-api-access-nxwpk\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:21:11.464174 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.464137    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:21:11.911970 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.911931    2558 generic.go:356] "Generic (PLEG): container finished" podID="6dd97aa1-a0fe-48d2-8b19-72f5769880a8" containerID="f7617243173c6a3f23a1eb94cc127821a7287343b43f444cdc6d436e3e7e4ae1" exitCode=0
Sep 10 03:21:11.912454 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.911981    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c970f3-6bf9c9c56b-74cb9"
Sep 10 03:21:11.912454 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.912014    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c970f3-6bf9c9c56b-74cb9" event={"ID":"6dd97aa1-a0fe-48d2-8b19-72f5769880a8","Type":"ContainerDied","Data":"f7617243173c6a3f23a1eb94cc127821a7287343b43f444cdc6d436e3e7e4ae1"}
Sep 10 03:21:11.912454 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.912073    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c970f3-6bf9c9c56b-74cb9" event={"ID":"6dd97aa1-a0fe-48d2-8b19-72f5769880a8","Type":"ContainerDied","Data":"8be09f88869141d6eb549917679ac215317b1979da2efd13f18238d56200d385"}
Sep 10 03:21:11.912454 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.912132    2558 scope.go:117] "RemoveContainer" containerID="f7617243173c6a3f23a1eb94cc127821a7287343b43f444cdc6d436e3e7e4ae1"
Sep 10 03:21:11.913626 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.913601    2558 generic.go:356] "Generic (PLEG): container finished" podID="7ad7c90c-9e2e-4bd3-b01f-00236dde69da" containerID="5fdf815835f5e47145893a958c387e80f6a08843f1e69b561902a591861afa2a" exitCode=0
Sep 10 03:21:11.913702 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.913645    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv" event={"ID":"7ad7c90c-9e2e-4bd3-b01f-00236dde69da","Type":"ContainerDied","Data":"5fdf815835f5e47145893a958c387e80f6a08843f1e69b561902a591861afa2a"}
Sep 10 03:21:11.913702 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.913653    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv"
Sep 10 03:21:11.913702 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.913693    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv" event={"ID":"7ad7c90c-9e2e-4bd3-b01f-00236dde69da","Type":"ContainerDied","Data":"03bfbd40d49768f2b8e7804c87ed09eef9dc85eb851647d79adb70405d713006"}
Sep 10 03:21:11.928931 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.928890    2558 scope.go:117] "RemoveContainer" containerID="f7617243173c6a3f23a1eb94cc127821a7287343b43f444cdc6d436e3e7e4ae1"
Sep 10 03:21:11.929390 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:21:11.929357    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7617243173c6a3f23a1eb94cc127821a7287343b43f444cdc6d436e3e7e4ae1\": container with ID starting with f7617243173c6a3f23a1eb94cc127821a7287343b43f444cdc6d436e3e7e4ae1 not found: ID does not exist" containerID="f7617243173c6a3f23a1eb94cc127821a7287343b43f444cdc6d436e3e7e4ae1"
Sep 10 03:21:11.929507 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.929393    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7617243173c6a3f23a1eb94cc127821a7287343b43f444cdc6d436e3e7e4ae1"} err="failed to get container status \"f7617243173c6a3f23a1eb94cc127821a7287343b43f444cdc6d436e3e7e4ae1\": rpc error: code = NotFound desc = could not find container \"f7617243173c6a3f23a1eb94cc127821a7287343b43f444cdc6d436e3e7e4ae1\": container with ID starting with f7617243173c6a3f23a1eb94cc127821a7287343b43f444cdc6d436e3e7e4ae1 not found: ID does not exist"
Sep 10 03:21:11.929507 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.929412    2558 scope.go:117] "RemoveContainer" containerID="5fdf815835f5e47145893a958c387e80f6a08843f1e69b561902a591861afa2a"
Sep 10 03:21:11.938754 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.938716    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c970f3-6bf9c9c56b-74cb9"]
Sep 10 03:21:11.939178 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.939160    2558 scope.go:117] "RemoveContainer" containerID="5fdf815835f5e47145893a958c387e80f6a08843f1e69b561902a591861afa2a"
Sep 10 03:21:11.939524 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:21:11.939495    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fdf815835f5e47145893a958c387e80f6a08843f1e69b561902a591861afa2a\": container with ID starting with 5fdf815835f5e47145893a958c387e80f6a08843f1e69b561902a591861afa2a not found: ID does not exist" containerID="5fdf815835f5e47145893a958c387e80f6a08843f1e69b561902a591861afa2a"
Sep 10 03:21:11.939664 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.939536    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fdf815835f5e47145893a958c387e80f6a08843f1e69b561902a591861afa2a"} err="failed to get container status \"5fdf815835f5e47145893a958c387e80f6a08843f1e69b561902a591861afa2a\": rpc error: code = NotFound desc = could not find container \"5fdf815835f5e47145893a958c387e80f6a08843f1e69b561902a591861afa2a\": container with ID starting with 5fdf815835f5e47145893a958c387e80f6a08843f1e69b561902a591861afa2a not found: ID does not exist"
Sep 10 03:21:11.942394 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.942369    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c970f3-6bf9c9c56b-74cb9"]
Sep 10 03:21:11.951647 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.951615    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv"]
Sep 10 03:21:11.955620 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:11.955572    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c970f3-86488c548f-bhsbv"]
Sep 10 03:21:12.069311 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:12.069276    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dd97aa1-a0fe-48d2-8b19-72f5769880a8" path="/var/lib/kubelet/pods/6dd97aa1-a0fe-48d2-8b19-72f5769880a8/volumes"
Sep 10 03:21:12.069827 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:12.069801    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ad7c90c-9e2e-4bd3-b01f-00236dde69da" path="/var/lib/kubelet/pods/7ad7c90c-9e2e-4bd3-b01f-00236dde69da/volumes"
Sep 10 03:21:17.334677 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:17.334640    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:21:17.399145 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:17.399090    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:21:17.601608 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:17.601527    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:21:17.939541 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:17.939447    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-af7727-6z7xw"]
Sep 10 03:21:17.939834 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:17.939821    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a678a31-a511-4f52-9890-ec6026611cbe" containerName="payload-pre-processing"
Sep 10 03:21:17.939882 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:17.939835    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a678a31-a511-4f52-9890-ec6026611cbe" containerName="payload-pre-processing"
Sep 10 03:21:17.939882 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:17.939847    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2de8d4b8-4fa7-46db-9390-6fa3b4249971" containerName="payload-processing"
Sep 10 03:21:17.939882 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:17.939854    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2de8d4b8-4fa7-46db-9390-6fa3b4249971" containerName="payload-processing"
Sep 10 03:21:17.939882 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:17.939861    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ad7c90c-9e2e-4bd3-b01f-00236dde69da" containerName="maas-api"
Sep 10 03:21:17.939882 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:17.939868    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ad7c90c-9e2e-4bd3-b01f-00236dde69da" containerName="maas-api"
Sep 10 03:21:17.939882 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:17.939881    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6dc7868-800f-46fa-af8e-103b4ea82d36" containerName="maas-api"
Sep 10 03:21:17.940093 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:17.939887    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6dc7868-800f-46fa-af8e-103b4ea82d36" containerName="maas-api"
Sep 10 03:21:17.940093 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:17.939893    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dd97aa1-a0fe-48d2-8b19-72f5769880a8" containerName="payload-pre-processing"
Sep 10 03:21:17.940093 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:17.939898    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dd97aa1-a0fe-48d2-8b19-72f5769880a8" containerName="payload-pre-processing"
Sep 10 03:21:17.940093 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:17.939907    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cb24320-6749-48e4-a6d4-70564b121f00" containerName="maas-api"
Sep 10 03:21:17.940093 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:17.939912    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cb24320-6749-48e4-a6d4-70564b121f00" containerName="maas-api"
Sep 10 03:21:17.940093 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:17.939921    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a66208ee-3ea7-477b-9142-9c1986dfa47a" containerName="payload-processing"
Sep 10 03:21:17.940093 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:17.939926    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a66208ee-3ea7-477b-9142-9c1986dfa47a" containerName="payload-processing"
Sep 10 03:21:17.940093 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:17.939975    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2de8d4b8-4fa7-46db-9390-6fa3b4249971" containerName="payload-processing"
Sep 10 03:21:17.940093 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:17.939983    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dd97aa1-a0fe-48d2-8b19-72f5769880a8" containerName="payload-pre-processing"
Sep 10 03:21:17.940093 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:17.939990    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a66208ee-3ea7-477b-9142-9c1986dfa47a" containerName="payload-processing"
Sep 10 03:21:17.940093 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:17.939996    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ad7c90c-9e2e-4bd3-b01f-00236dde69da" containerName="maas-api"
Sep 10 03:21:17.940093 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:17.940002    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cb24320-6749-48e4-a6d4-70564b121f00" containerName="maas-api"
Sep 10 03:21:17.940093 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:17.940007    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a678a31-a511-4f52-9890-ec6026611cbe" containerName="payload-pre-processing"
Sep 10 03:21:17.940093 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:17.940012    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6dc7868-800f-46fa-af8e-103b4ea82d36" containerName="maas-api"
Sep 10 03:21:17.944812 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:17.944777    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-af7727-6z7xw"
Sep 10 03:21:17.947446 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:17.947409    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8jjl\""
Sep 10 03:21:17.950995 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:17.950963    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-af7727-6z7xw"]
Sep 10 03:21:18.014232 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:18.014181    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64fddba5-ec12-436a-84ed-8a1eb9fba7c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-af7727-6z7xw\" (UID: \"64fddba5-ec12-436a-84ed-8a1eb9fba7c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-af7727-6z7xw"
Sep 10 03:21:18.115290 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:18.115238    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64fddba5-ec12-436a-84ed-8a1eb9fba7c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-af7727-6z7xw\" (UID: \"64fddba5-ec12-436a-84ed-8a1eb9fba7c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-af7727-6z7xw"
Sep 10 03:21:18.115937 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:18.115911    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64fddba5-ec12-436a-84ed-8a1eb9fba7c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-af7727-6z7xw\" (UID: \"64fddba5-ec12-436a-84ed-8a1eb9fba7c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-af7727-6z7xw"
Sep 10 03:21:18.175758 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:18.175710    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:21:18.257575 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:18.257539    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-af7727-6z7xw"
Sep 10 03:21:18.533363 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:18.533330    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-af7727-6z7xw"]
Sep 10 03:21:18.535450 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:21:18.535421    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64fddba5_ec12_436a_84ed_8a1eb9fba7c8.slice/crio-4651f30e60c4b949dfeb5339640f0ef8173ccc7007b061b755a5a3af64594aed WatchSource:0}: Error finding container 4651f30e60c4b949dfeb5339640f0ef8173ccc7007b061b755a5a3af64594aed: Status 404 returned error can't find the container with id 4651f30e60c4b949dfeb5339640f0ef8173ccc7007b061b755a5a3af64594aed
Sep 10 03:21:18.954508 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:18.954473    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-af7727-6z7xw" event={"ID":"64fddba5-ec12-436a-84ed-8a1eb9fba7c8","Type":"ContainerStarted","Data":"aea8d27e2a3155182459e4b4bb0ca988b47a1207be0c2ad995051f8faf8c922b"}
Sep 10 03:21:18.954708 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:18.954534    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-af7727-6z7xw" event={"ID":"64fddba5-ec12-436a-84ed-8a1eb9fba7c8","Type":"ContainerStarted","Data":"4651f30e60c4b949dfeb5339640f0ef8173ccc7007b061b755a5a3af64594aed"}
Sep 10 03:21:18.972112 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:18.972036    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-af7727-6z7xw" podStartSLOduration=1.972019502 podStartE2EDuration="1.972019502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:21:18.969161713 +0000 UTC m=+2069.344247635" watchObservedRunningTime="2026-09-10 03:21:18.972019502 +0000 UTC m=+2069.347105423"
Sep 10 03:21:19.962306 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:19.962265    2558 generic.go:356] "Generic (PLEG): container finished" podID="64fddba5-ec12-436a-84ed-8a1eb9fba7c8" containerID="aea8d27e2a3155182459e4b4bb0ca988b47a1207be0c2ad995051f8faf8c922b" exitCode=0
Sep 10 03:21:19.962719 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:19.962355    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-af7727-6z7xw" event={"ID":"64fddba5-ec12-436a-84ed-8a1eb9fba7c8","Type":"ContainerDied","Data":"aea8d27e2a3155182459e4b4bb0ca988b47a1207be0c2ad995051f8faf8c922b"}
Sep 10 03:21:21.105914 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:21.105880    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-af7727-6z7xw"
Sep 10 03:21:21.236491 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:21.236389    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64fddba5-ec12-436a-84ed-8a1eb9fba7c8-maas-api-service-ca\") pod \"64fddba5-ec12-436a-84ed-8a1eb9fba7c8\" (UID: \"64fddba5-ec12-436a-84ed-8a1eb9fba7c8\") "
Sep 10 03:21:21.236762 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:21.236736    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64fddba5-ec12-436a-84ed-8a1eb9fba7c8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "64fddba5-ec12-436a-84ed-8a1eb9fba7c8" (UID: "64fddba5-ec12-436a-84ed-8a1eb9fba7c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:21:21.336974 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:21.336925    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64fddba5-ec12-436a-84ed-8a1eb9fba7c8-maas-api-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:21:21.978744 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:21.978706    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-af7727-6z7xw" event={"ID":"64fddba5-ec12-436a-84ed-8a1eb9fba7c8","Type":"ContainerDied","Data":"4651f30e60c4b949dfeb5339640f0ef8173ccc7007b061b755a5a3af64594aed"}
Sep 10 03:21:21.978744 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:21.978736    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-af7727-6z7xw"
Sep 10 03:21:21.978744 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:21.978758    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4651f30e60c4b949dfeb5339640f0ef8173ccc7007b061b755a5a3af64594aed"
Sep 10 03:21:24.788933 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:24.788900    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:21:28.080374 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:28.080336    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:21:28.611637 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:28.611594    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:21:28.655013 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:28.654963    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7727-7dbbddd5d-mh874"]
Sep 10 03:21:28.655424 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:28.655366    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7727-7dbbddd5d-mh874" podUID="7590551e-6d05-4757-bf5e-7ef219553510" containerName="payload-pre-processing" containerID="cri-o://f8de70542e7af396c759073cd9f02ddf64e68ea4066461d7c61309a107dc7cd5" gracePeriod=30
Sep 10 03:21:28.685389 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:28.685357    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:21:28.786160 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:28.786122    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-af7727-6z7xw"]
Sep 10 03:21:28.789736 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:28.789685    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:21:28.789930 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:28.789907    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-af7727-6z7xw"]
Sep 10 03:21:28.850419 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:28.850383    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:21:28.940133 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:28.940083    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7727-7dbbddd5d-mh874"
Sep 10 03:21:29.019731 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:29.019691    2558 generic.go:356] "Generic (PLEG): container finished" podID="7590551e-6d05-4757-bf5e-7ef219553510" containerID="f8de70542e7af396c759073cd9f02ddf64e68ea4066461d7c61309a107dc7cd5" exitCode=0
Sep 10 03:21:29.019945 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:29.019761    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7727-7dbbddd5d-mh874"
Sep 10 03:21:29.019945 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:29.019777    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7727-7dbbddd5d-mh874" event={"ID":"7590551e-6d05-4757-bf5e-7ef219553510","Type":"ContainerDied","Data":"f8de70542e7af396c759073cd9f02ddf64e68ea4066461d7c61309a107dc7cd5"}
Sep 10 03:21:29.019945 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:29.019834    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7727-7dbbddd5d-mh874" event={"ID":"7590551e-6d05-4757-bf5e-7ef219553510","Type":"ContainerDied","Data":"b8fea9efa24624b92e20bf7189fdb362ea8da5a2af1c5c45c813cbd693d61028"}
Sep 10 03:21:29.019945 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:29.019861    2558 scope.go:117] "RemoveContainer" containerID="f8de70542e7af396c759073cd9f02ddf64e68ea4066461d7c61309a107dc7cd5"
Sep 10 03:21:29.029209 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:29.029182    2558 scope.go:117] "RemoveContainer" containerID="f8de70542e7af396c759073cd9f02ddf64e68ea4066461d7c61309a107dc7cd5"
Sep 10 03:21:29.029550 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:21:29.029527    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8de70542e7af396c759073cd9f02ddf64e68ea4066461d7c61309a107dc7cd5\": container with ID starting with f8de70542e7af396c759073cd9f02ddf64e68ea4066461d7c61309a107dc7cd5 not found: ID does not exist" containerID="f8de70542e7af396c759073cd9f02ddf64e68ea4066461d7c61309a107dc7cd5"
Sep 10 03:21:29.029634 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:29.029565    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8de70542e7af396c759073cd9f02ddf64e68ea4066461d7c61309a107dc7cd5"} err="failed to get container status \"f8de70542e7af396c759073cd9f02ddf64e68ea4066461d7c61309a107dc7cd5\": rpc error: code = NotFound desc = could not find container \"f8de70542e7af396c759073cd9f02ddf64e68ea4066461d7c61309a107dc7cd5\": container with ID starting with f8de70542e7af396c759073cd9f02ddf64e68ea4066461d7c61309a107dc7cd5 not found: ID does not exist"
Sep 10 03:21:29.090418 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:29.090379    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7590551e-6d05-4757-bf5e-7ef219553510-plugins-config-volume\") pod \"7590551e-6d05-4757-bf5e-7ef219553510\" (UID: \"7590551e-6d05-4757-bf5e-7ef219553510\") "
Sep 10 03:21:29.090938 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:29.090440    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55bj2\" (UniqueName: \"kubernetes.io/projected/7590551e-6d05-4757-bf5e-7ef219553510-kube-api-access-55bj2\") pod \"7590551e-6d05-4757-bf5e-7ef219553510\" (UID: \"7590551e-6d05-4757-bf5e-7ef219553510\") "
Sep 10 03:21:29.090938 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:29.090819    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7590551e-6d05-4757-bf5e-7ef219553510-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7590551e-6d05-4757-bf5e-7ef219553510" (UID: "7590551e-6d05-4757-bf5e-7ef219553510"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:21:29.094709 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:29.094672    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7590551e-6d05-4757-bf5e-7ef219553510-kube-api-access-55bj2" (OuterVolumeSpecName: "kube-api-access-55bj2") pod "7590551e-6d05-4757-bf5e-7ef219553510" (UID: "7590551e-6d05-4757-bf5e-7ef219553510"). InnerVolumeSpecName "kube-api-access-55bj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:21:29.184839 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:29.184721    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:21:29.191934 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:29.191906    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7590551e-6d05-4757-bf5e-7ef219553510-plugins-config-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:21:29.191934 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:29.191932    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55bj2\" (UniqueName: \"kubernetes.io/projected/7590551e-6d05-4757-bf5e-7ef219553510-kube-api-access-55bj2\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:21:29.342116 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:29.342064    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7727-7dbbddd5d-mh874"]
Sep 10 03:21:29.348599 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:29.348552    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7727-7dbbddd5d-mh874"]
Sep 10 03:21:30.070682 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:30.070637    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64fddba5-ec12-436a-84ed-8a1eb9fba7c8" path="/var/lib/kubelet/pods/64fddba5-ec12-436a-84ed-8a1eb9fba7c8/volumes"
Sep 10 03:21:30.071076 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:30.071049    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7590551e-6d05-4757-bf5e-7ef219553510" path="/var/lib/kubelet/pods/7590551e-6d05-4757-bf5e-7ef219553510/volumes"
Sep 10 03:21:35.071666 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:35.071623    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:21:35.171591 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:35.171529    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:21:35.339940 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:35.339838    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:21:35.666259 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:35.666053    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7daa14-ztbf6"]
Sep 10 03:21:35.666631 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:35.666597    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7590551e-6d05-4757-bf5e-7ef219553510" containerName="payload-pre-processing"
Sep 10 03:21:35.666631 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:35.666624    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7590551e-6d05-4757-bf5e-7ef219553510" containerName="payload-pre-processing"
Sep 10 03:21:35.666631 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:35.666636    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64fddba5-ec12-436a-84ed-8a1eb9fba7c8" containerName="revoke-keys"
Sep 10 03:21:35.666854 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:35.666643    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="64fddba5-ec12-436a-84ed-8a1eb9fba7c8" containerName="revoke-keys"
Sep 10 03:21:35.666854 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:35.666698    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7590551e-6d05-4757-bf5e-7ef219553510" containerName="payload-pre-processing"
Sep 10 03:21:35.666854 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:35.666708    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="64fddba5-ec12-436a-84ed-8a1eb9fba7c8" containerName="revoke-keys"
Sep 10 03:21:35.671345 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:35.671319    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7daa14-ztbf6"
Sep 10 03:21:35.673825 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:35.673801    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8jjl\""
Sep 10 03:21:35.676214 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:35.676179    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7daa14-ztbf6"]
Sep 10 03:21:35.745168 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:35.745084    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92a12b51-0b98-458e-9ba3-bbe7cbc88337-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7daa14-ztbf6\" (UID: \"92a12b51-0b98-458e-9ba3-bbe7cbc88337\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7daa14-ztbf6"
Sep 10 03:21:35.846416 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:35.846367    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92a12b51-0b98-458e-9ba3-bbe7cbc88337-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7daa14-ztbf6\" (UID: \"92a12b51-0b98-458e-9ba3-bbe7cbc88337\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7daa14-ztbf6"
Sep 10 03:21:35.847040 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:35.847017    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92a12b51-0b98-458e-9ba3-bbe7cbc88337-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7daa14-ztbf6\" (UID: \"92a12b51-0b98-458e-9ba3-bbe7cbc88337\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7daa14-ztbf6"
Sep 10 03:21:35.859405 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:35.859360    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:21:35.985740 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:35.985704    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7daa14-ztbf6"
Sep 10 03:21:36.258684 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:36.258657    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7daa14-ztbf6"]
Sep 10 03:21:36.261320 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:21:36.261283    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a12b51_0b98_458e_9ba3_bbe7cbc88337.slice/crio-0c2f571b2a26e2ea7557d20bc02504139aa20402f3038d126533ad04e851a4c5 WatchSource:0}: Error finding container 0c2f571b2a26e2ea7557d20bc02504139aa20402f3038d126533ad04e851a4c5: Status 404 returned error can't find the container with id 0c2f571b2a26e2ea7557d20bc02504139aa20402f3038d126533ad04e851a4c5
Sep 10 03:21:37.070516 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:37.070475    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7daa14-ztbf6" event={"ID":"92a12b51-0b98-458e-9ba3-bbe7cbc88337","Type":"ContainerStarted","Data":"34dd886ba7ecc32e3526c3dc5ce5f1d1c1097c5862098a06e17a3d5e3a02c8ae"}
Sep 10 03:21:37.070732 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:37.070537    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7daa14-ztbf6" event={"ID":"92a12b51-0b98-458e-9ba3-bbe7cbc88337","Type":"ContainerStarted","Data":"0c2f571b2a26e2ea7557d20bc02504139aa20402f3038d126533ad04e851a4c5"}
Sep 10 03:21:37.091175 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:37.086545    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7daa14-ztbf6" podStartSLOduration=2.086522648 podStartE2EDuration="2.086522648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:21:37.083721872 +0000 UTC m=+2087.458807793" watchObservedRunningTime="2026-09-10 03:21:37.086522648 +0000 UTC m=+2087.461608572"
Sep 10 03:21:38.078167 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:38.078127    2558 generic.go:356] "Generic (PLEG): container finished" podID="92a12b51-0b98-458e-9ba3-bbe7cbc88337" containerID="34dd886ba7ecc32e3526c3dc5ce5f1d1c1097c5862098a06e17a3d5e3a02c8ae" exitCode=0
Sep 10 03:21:38.078589 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:38.078270    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7daa14-ztbf6" event={"ID":"92a12b51-0b98-458e-9ba3-bbe7cbc88337","Type":"ContainerDied","Data":"34dd886ba7ecc32e3526c3dc5ce5f1d1c1097c5862098a06e17a3d5e3a02c8ae"}
Sep 10 03:21:39.225957 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:39.225927    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7daa14-ztbf6"
Sep 10 03:21:39.276870 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:39.276826    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92a12b51-0b98-458e-9ba3-bbe7cbc88337-maas-api-service-ca\") pod \"92a12b51-0b98-458e-9ba3-bbe7cbc88337\" (UID: \"92a12b51-0b98-458e-9ba3-bbe7cbc88337\") "
Sep 10 03:21:39.277307 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:39.277276    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92a12b51-0b98-458e-9ba3-bbe7cbc88337-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "92a12b51-0b98-458e-9ba3-bbe7cbc88337" (UID: "92a12b51-0b98-458e-9ba3-bbe7cbc88337"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:21:39.377986 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:39.377873    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92a12b51-0b98-458e-9ba3-bbe7cbc88337-maas-api-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:21:40.091144 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:40.091114    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7daa14-ztbf6"
Sep 10 03:21:40.091348 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:40.091114    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7daa14-ztbf6" event={"ID":"92a12b51-0b98-458e-9ba3-bbe7cbc88337","Type":"ContainerDied","Data":"0c2f571b2a26e2ea7557d20bc02504139aa20402f3038d126533ad04e851a4c5"}
Sep 10 03:21:40.091348 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:40.091248    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c2f571b2a26e2ea7557d20bc02504139aa20402f3038d126533ad04e851a4c5"
Sep 10 03:21:46.188545 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:46.188499    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:21:46.189040 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:46.188684    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7daa14-29816835-xb9td"]
Sep 10 03:21:46.192441 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:46.192406    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7daa14-29816835-xb9td"]
Sep 10 03:21:46.255076 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:46.255026    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:21:46.321638 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:46.321598    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:21:46.360052 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:46.359992    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7daa14-ztbf6"]
Sep 10 03:21:46.362268 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:46.362233    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7daa14-ztbf6"]
Sep 10 03:21:46.421325 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:46.421292    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:21:46.727752 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:46.727706    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:21:48.069805 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:48.069768    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31cc1407-9a71-49f2-b7f6-3bc16f3b14b7" path="/var/lib/kubelet/pods/31cc1407-9a71-49f2-b7f6-3bc16f3b14b7/volumes"
Sep 10 03:21:48.070228 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:48.070081    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92a12b51-0b98-458e-9ba3-bbe7cbc88337" path="/var/lib/kubelet/pods/92a12b51-0b98-458e-9ba3-bbe7cbc88337/volumes"
Sep 10 03:21:50.635166 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:50.635126    2558 scope.go:117] "RemoveContainer" containerID="5db4beec708fadc8c84cbdfdb10f720f008dc2f7f6024f0371b18e2ef977eeeb"
Sep 10 03:21:50.644018 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:50.643995    2558 scope.go:117] "RemoveContainer" containerID="da69f2c0d03efb46b2336c2c033b6116e8e4de14dc8eae3190cfe76615b4024d"
Sep 10 03:21:50.652692 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:50.652670    2558 scope.go:117] "RemoveContainer" containerID="8de4e785f7d00287490060b7c755bd84a3c7a0de1dac60d983135a272e9dfaf8"
Sep 10 03:21:52.572203 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:52.572167    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:21:52.672205 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:52.672175    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:21:52.867139 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:21:52.867002    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:22:29.495299 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:22:29.495255    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:22:29.546729 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:22:29.546694    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:22:41.970909 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:22:41.970868    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:23:19.190304 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:23:19.190263    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:23:19.471361 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:23:19.471315    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:23:21.918824 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:23:21.918781    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:23:42.689237 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:23:42.689194    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:24:31.696214 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:24:31.696166    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgc8x"]
Sep 10 03:24:31.696831 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:24:31.696520    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgc8x" podUID="6b1e6ef4-b258-4a0f-aa44-d8c049d8909e" containerName="manager" containerID="cri-o://ff2fafeed5751d3ed7fe416f5048d3e4e107d89353826e8b8afd52b05675cde2" gracePeriod=10
Sep 10 03:24:32.050731 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:24:32.050692    2558 generic.go:356] "Generic (PLEG): container finished" podID="6b1e6ef4-b258-4a0f-aa44-d8c049d8909e" containerID="ff2fafeed5751d3ed7fe416f5048d3e4e107d89353826e8b8afd52b05675cde2" exitCode=0
Sep 10 03:24:32.050872 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:24:32.050750    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgc8x" event={"ID":"6b1e6ef4-b258-4a0f-aa44-d8c049d8909e","Type":"ContainerDied","Data":"ff2fafeed5751d3ed7fe416f5048d3e4e107d89353826e8b8afd52b05675cde2"}
Sep 10 03:24:32.064547 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:24:32.064516    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgc8x"
Sep 10 03:24:32.191536 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:24:32.191488    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b1e6ef4-b258-4a0f-aa44-d8c049d8909e-extensions-socket-volume\") pod \"6b1e6ef4-b258-4a0f-aa44-d8c049d8909e\" (UID: \"6b1e6ef4-b258-4a0f-aa44-d8c049d8909e\") "
Sep 10 03:24:32.191737 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:24:32.191557    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nflss\" (UniqueName: \"kubernetes.io/projected/6b1e6ef4-b258-4a0f-aa44-d8c049d8909e-kube-api-access-nflss\") pod \"6b1e6ef4-b258-4a0f-aa44-d8c049d8909e\" (UID: \"6b1e6ef4-b258-4a0f-aa44-d8c049d8909e\") "
Sep 10 03:24:32.191998 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:24:32.191955    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b1e6ef4-b258-4a0f-aa44-d8c049d8909e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6b1e6ef4-b258-4a0f-aa44-d8c049d8909e" (UID: "6b1e6ef4-b258-4a0f-aa44-d8c049d8909e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 03:24:32.193790 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:24:32.193756    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b1e6ef4-b258-4a0f-aa44-d8c049d8909e-kube-api-access-nflss" (OuterVolumeSpecName: "kube-api-access-nflss") pod "6b1e6ef4-b258-4a0f-aa44-d8c049d8909e" (UID: "6b1e6ef4-b258-4a0f-aa44-d8c049d8909e"). InnerVolumeSpecName "kube-api-access-nflss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:24:32.292973 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:24:32.292877    2558 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b1e6ef4-b258-4a0f-aa44-d8c049d8909e-extensions-socket-volume\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:24:32.292973 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:24:32.292916    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nflss\" (UniqueName: \"kubernetes.io/projected/6b1e6ef4-b258-4a0f-aa44-d8c049d8909e-kube-api-access-nflss\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:24:33.058325 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:24:33.058293    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgc8x"
Sep 10 03:24:33.058793 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:24:33.058327    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgc8x" event={"ID":"6b1e6ef4-b258-4a0f-aa44-d8c049d8909e","Type":"ContainerDied","Data":"b50996cf4e676e299df257dfb0a267b68de8999e5aca1da435a3ccde22297ce5"}
Sep 10 03:24:33.058793 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:24:33.058396    2558 scope.go:117] "RemoveContainer" containerID="ff2fafeed5751d3ed7fe416f5048d3e4e107d89353826e8b8afd52b05675cde2"
Sep 10 03:24:33.080246 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:24:33.080181    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgc8x"]
Sep 10 03:24:33.084364 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:24:33.084332    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgc8x"]
Sep 10 03:24:34.069238 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:24:34.069201    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b1e6ef4-b258-4a0f-aa44-d8c049d8909e" path="/var/lib/kubelet/pods/6b1e6ef4-b258-4a0f-aa44-d8c049d8909e/volumes"
Sep 10 03:25:32.379293 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:25:32.379254    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frjj2"]
Sep 10 03:25:32.379932 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:25:32.379802    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92a12b51-0b98-458e-9ba3-bbe7cbc88337" containerName="revoke-keys"
Sep 10 03:25:32.379932 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:25:32.379825    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="92a12b51-0b98-458e-9ba3-bbe7cbc88337" containerName="revoke-keys"
Sep 10 03:25:32.379932 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:25:32.379861    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b1e6ef4-b258-4a0f-aa44-d8c049d8909e" containerName="manager"
Sep 10 03:25:32.379932 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:25:32.379870    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b1e6ef4-b258-4a0f-aa44-d8c049d8909e" containerName="manager"
Sep 10 03:25:32.380197 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:25:32.379941    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="92a12b51-0b98-458e-9ba3-bbe7cbc88337" containerName="revoke-keys"
Sep 10 03:25:32.380197 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:25:32.379957    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b1e6ef4-b258-4a0f-aa44-d8c049d8909e" containerName="manager"
Sep 10 03:25:32.383026 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:25:32.382992    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frjj2"
Sep 10 03:25:32.385228 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:25:32.385199    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-sp59z\""
Sep 10 03:25:32.395185 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:25:32.395155    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frjj2"]
Sep 10 03:25:32.541369 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:25:32.541331    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7wh7\" (UniqueName: \"kubernetes.io/projected/eafd907c-b015-46f9-b5f8-79f08dcc51f1-kube-api-access-x7wh7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-frjj2\" (UID: \"eafd907c-b015-46f9-b5f8-79f08dcc51f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frjj2"
Sep 10 03:25:32.541567 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:25:32.541397    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eafd907c-b015-46f9-b5f8-79f08dcc51f1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-frjj2\" (UID: \"eafd907c-b015-46f9-b5f8-79f08dcc51f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frjj2"
Sep 10 03:25:32.642861 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:25:32.642750    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eafd907c-b015-46f9-b5f8-79f08dcc51f1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-frjj2\" (UID: \"eafd907c-b015-46f9-b5f8-79f08dcc51f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frjj2"
Sep 10 03:25:32.642861 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:25:32.642839    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7wh7\" (UniqueName: \"kubernetes.io/projected/eafd907c-b015-46f9-b5f8-79f08dcc51f1-kube-api-access-x7wh7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-frjj2\" (UID: \"eafd907c-b015-46f9-b5f8-79f08dcc51f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frjj2"
Sep 10 03:25:32.643198 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:25:32.643176    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eafd907c-b015-46f9-b5f8-79f08dcc51f1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-frjj2\" (UID: \"eafd907c-b015-46f9-b5f8-79f08dcc51f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frjj2"
Sep 10 03:25:32.651595 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:25:32.651564    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7wh7\" (UniqueName: \"kubernetes.io/projected/eafd907c-b015-46f9-b5f8-79f08dcc51f1-kube-api-access-x7wh7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-frjj2\" (UID: \"eafd907c-b015-46f9-b5f8-79f08dcc51f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frjj2"
Sep 10 03:25:32.695185 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:25:32.695144    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frjj2"
Sep 10 03:25:32.974440 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:25:32.974412    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frjj2"]
Sep 10 03:25:32.976981 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:25:32.976945    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeafd907c_b015_46f9_b5f8_79f08dcc51f1.slice/crio-bb93ddbccc69a28d8eb2eb312f541d321db7c3a47020fb19cfcd7bacb7676ee8 WatchSource:0}: Error finding container bb93ddbccc69a28d8eb2eb312f541d321db7c3a47020fb19cfcd7bacb7676ee8: Status 404 returned error can't find the container with id bb93ddbccc69a28d8eb2eb312f541d321db7c3a47020fb19cfcd7bacb7676ee8
Sep 10 03:25:33.388831 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:25:33.388715    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frjj2" event={"ID":"eafd907c-b015-46f9-b5f8-79f08dcc51f1","Type":"ContainerStarted","Data":"01d6e3096be51dc3092a793178335e818a44f14380cbedcea5dacd124fae1330"}
Sep 10 03:25:33.388831 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:25:33.388781    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frjj2" event={"ID":"eafd907c-b015-46f9-b5f8-79f08dcc51f1","Type":"ContainerStarted","Data":"bb93ddbccc69a28d8eb2eb312f541d321db7c3a47020fb19cfcd7bacb7676ee8"}
Sep 10 03:25:33.388831 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:25:33.388803    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frjj2"
Sep 10 03:25:33.427085 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:25:33.427030    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frjj2" podStartSLOduration=1.427001875 podStartE2EDuration="1.427001875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:25:33.425083546 +0000 UTC m=+2323.800169467" watchObservedRunningTime="2026-09-10 03:25:33.427001875 +0000 UTC m=+2323.802087792"
Sep 10 03:25:44.397548 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:25:44.397465    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frjj2"
Sep 10 03:26:13.510411 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:26:13.510365    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:26:13.599366 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:26:13.599324    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:26:16.145152 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:26:16.145097    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:26:44.252259 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:26:44.252204    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:26:52.909422 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:26:52.909377    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:27:09.576307 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:09.576260    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:27:11.269374 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:11.269338    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:27:27.128720 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:27.128674    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-77d77d65f8-9th46"]
Sep 10 03:27:27.132381 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:27.132351    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77d77d65f8-9th46"
Sep 10 03:27:27.135209 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:27.135180    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 03:27:27.135360 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:27.135181    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 03:27:27.135360 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:27.135190    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fqv2s\""
Sep 10 03:27:27.143542 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:27.143511    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-77d77d65f8-9th46"]
Sep 10 03:27:27.233275 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:27.233227    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfr68\" (UniqueName: \"kubernetes.io/projected/ac4560af-2e9c-4897-80b4-fc5d2dc22be2-kube-api-access-kfr68\") pod \"maas-api-77d77d65f8-9th46\" (UID: \"ac4560af-2e9c-4897-80b4-fc5d2dc22be2\") " pod="odh-ai-gateway-infra/maas-api-77d77d65f8-9th46"
Sep 10 03:27:27.233513 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:27.233294    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac4560af-2e9c-4897-80b4-fc5d2dc22be2-openshift-service-ca\") pod \"maas-api-77d77d65f8-9th46\" (UID: \"ac4560af-2e9c-4897-80b4-fc5d2dc22be2\") " pod="odh-ai-gateway-infra/maas-api-77d77d65f8-9th46"
Sep 10 03:27:27.233513 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:27.233354    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac4560af-2e9c-4897-80b4-fc5d2dc22be2-maas-api-tls\") pod \"maas-api-77d77d65f8-9th46\" (UID: \"ac4560af-2e9c-4897-80b4-fc5d2dc22be2\") " pod="odh-ai-gateway-infra/maas-api-77d77d65f8-9th46"
Sep 10 03:27:27.233513 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:27.233377    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac4560af-2e9c-4897-80b4-fc5d2dc22be2-metrics-certs\") pod \"maas-api-77d77d65f8-9th46\" (UID: \"ac4560af-2e9c-4897-80b4-fc5d2dc22be2\") " pod="odh-ai-gateway-infra/maas-api-77d77d65f8-9th46"
Sep 10 03:27:27.333889 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:27.333850    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfr68\" (UniqueName: \"kubernetes.io/projected/ac4560af-2e9c-4897-80b4-fc5d2dc22be2-kube-api-access-kfr68\") pod \"maas-api-77d77d65f8-9th46\" (UID: \"ac4560af-2e9c-4897-80b4-fc5d2dc22be2\") " pod="odh-ai-gateway-infra/maas-api-77d77d65f8-9th46"
Sep 10 03:27:27.334137 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:27.333910    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac4560af-2e9c-4897-80b4-fc5d2dc22be2-openshift-service-ca\") pod \"maas-api-77d77d65f8-9th46\" (UID: \"ac4560af-2e9c-4897-80b4-fc5d2dc22be2\") " pod="odh-ai-gateway-infra/maas-api-77d77d65f8-9th46"
Sep 10 03:27:27.334137 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:27.333966    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac4560af-2e9c-4897-80b4-fc5d2dc22be2-maas-api-tls\") pod \"maas-api-77d77d65f8-9th46\" (UID: \"ac4560af-2e9c-4897-80b4-fc5d2dc22be2\") " pod="odh-ai-gateway-infra/maas-api-77d77d65f8-9th46"
Sep 10 03:27:27.334137 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:27.333993    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac4560af-2e9c-4897-80b4-fc5d2dc22be2-metrics-certs\") pod \"maas-api-77d77d65f8-9th46\" (UID: \"ac4560af-2e9c-4897-80b4-fc5d2dc22be2\") " pod="odh-ai-gateway-infra/maas-api-77d77d65f8-9th46"
Sep 10 03:27:27.335036 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:27.335000    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac4560af-2e9c-4897-80b4-fc5d2dc22be2-openshift-service-ca\") pod \"maas-api-77d77d65f8-9th46\" (UID: \"ac4560af-2e9c-4897-80b4-fc5d2dc22be2\") " pod="odh-ai-gateway-infra/maas-api-77d77d65f8-9th46"
Sep 10 03:27:27.337483 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:27.337418    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac4560af-2e9c-4897-80b4-fc5d2dc22be2-metrics-certs\") pod \"maas-api-77d77d65f8-9th46\" (UID: \"ac4560af-2e9c-4897-80b4-fc5d2dc22be2\") " pod="odh-ai-gateway-infra/maas-api-77d77d65f8-9th46"
Sep 10 03:27:27.337753 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:27.337727    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac4560af-2e9c-4897-80b4-fc5d2dc22be2-maas-api-tls\") pod \"maas-api-77d77d65f8-9th46\" (UID: \"ac4560af-2e9c-4897-80b4-fc5d2dc22be2\") " pod="odh-ai-gateway-infra/maas-api-77d77d65f8-9th46"
Sep 10 03:27:27.341524 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:27.341495    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfr68\" (UniqueName: \"kubernetes.io/projected/ac4560af-2e9c-4897-80b4-fc5d2dc22be2-kube-api-access-kfr68\") pod \"maas-api-77d77d65f8-9th46\" (UID: \"ac4560af-2e9c-4897-80b4-fc5d2dc22be2\") " pod="odh-ai-gateway-infra/maas-api-77d77d65f8-9th46"
Sep 10 03:27:27.445031 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:27.444929    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77d77d65f8-9th46"
Sep 10 03:27:27.721347 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:27.721069    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-77d77d65f8-9th46"]
Sep 10 03:27:27.723779 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:27:27.723741    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac4560af_2e9c_4897_80b4_fc5d2dc22be2.slice/crio-e4a8e38356bb775b9210c52a688205351d6d64782651fec44003bf76e8e30b70 WatchSource:0}: Error finding container e4a8e38356bb775b9210c52a688205351d6d64782651fec44003bf76e8e30b70: Status 404 returned error can't find the container with id e4a8e38356bb775b9210c52a688205351d6d64782651fec44003bf76e8e30b70
Sep 10 03:27:27.725278 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:27.725247    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 03:27:28.019414 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:28.019300    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77d77d65f8-9th46" event={"ID":"ac4560af-2e9c-4897-80b4-fc5d2dc22be2","Type":"ContainerStarted","Data":"e4a8e38356bb775b9210c52a688205351d6d64782651fec44003bf76e8e30b70"}
Sep 10 03:27:29.026052 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:29.026010    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77d77d65f8-9th46" event={"ID":"ac4560af-2e9c-4897-80b4-fc5d2dc22be2","Type":"ContainerStarted","Data":"ea915902d6e5308291fc56931fcfbb6f7d1b2977a963fa353ada60be631ef609"}
Sep 10 03:27:29.026528 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:29.026228    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-77d77d65f8-9th46"
Sep 10 03:27:29.045518 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:29.045456    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-77d77d65f8-9th46" podStartSLOduration=1.540659064 podStartE2EDuration="2.045437938s" totalImagesPullingTime="504.778874ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:27:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:27:29.044370175 +0000 UTC m=+2439.419456096" watchObservedRunningTime="2026-09-10 03:27:29.045437938 +0000 UTC m=+2439.420523858"
Sep 10 03:27:34.524902 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:34.524832    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-77d77d65f8-9th46"]
Sep 10 03:27:34.525432 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:34.525198    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-77d77d65f8-9th46" podUID="ac4560af-2e9c-4897-80b4-fc5d2dc22be2" containerName="maas-api" containerID="cri-o://ea915902d6e5308291fc56931fcfbb6f7d1b2977a963fa353ada60be631ef609" gracePeriod=30
Sep 10 03:27:34.531189 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:34.531161    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-77d77d65f8-9th46"
Sep 10 03:27:34.568176 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:34.566832    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-59f756556b-wwlzj"]
Sep 10 03:27:34.569188 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:27:34.569138    2558 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac4560af_2e9c_4897_80b4_fc5d2dc22be2.slice/crio-ea915902d6e5308291fc56931fcfbb6f7d1b2977a963fa353ada60be631ef609.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac4560af_2e9c_4897_80b4_fc5d2dc22be2.slice/crio-conmon-ea915902d6e5308291fc56931fcfbb6f7d1b2977a963fa353ada60be631ef609.scope\": RecentStats: unable to find data in memory cache]"
Sep 10 03:27:34.573384 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:34.573350    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f756556b-wwlzj"
Sep 10 03:27:34.576644 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:34.576614    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f756556b-wwlzj"]
Sep 10 03:27:34.599663 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:34.599636    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkv6v\" (UniqueName: \"kubernetes.io/projected/f6abe5c8-12f8-4f60-a2cf-ebd47371ae1e-kube-api-access-xkv6v\") pod \"maas-api-59f756556b-wwlzj\" (UID: \"f6abe5c8-12f8-4f60-a2cf-ebd47371ae1e\") " pod="odh-ai-gateway-infra/maas-api-59f756556b-wwlzj"
Sep 10 03:27:34.599811 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:34.599681    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6abe5c8-12f8-4f60-a2cf-ebd47371ae1e-metrics-certs\") pod \"maas-api-59f756556b-wwlzj\" (UID: \"f6abe5c8-12f8-4f60-a2cf-ebd47371ae1e\") " pod="odh-ai-gateway-infra/maas-api-59f756556b-wwlzj"
Sep 10 03:27:34.599811 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:34.599758    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6abe5c8-12f8-4f60-a2cf-ebd47371ae1e-openshift-service-ca\") pod \"maas-api-59f756556b-wwlzj\" (UID: \"f6abe5c8-12f8-4f60-a2cf-ebd47371ae1e\") " pod="odh-ai-gateway-infra/maas-api-59f756556b-wwlzj"
Sep 10 03:27:34.599896 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:34.599814    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6abe5c8-12f8-4f60-a2cf-ebd47371ae1e-maas-api-tls\") pod \"maas-api-59f756556b-wwlzj\" (UID: \"f6abe5c8-12f8-4f60-a2cf-ebd47371ae1e\") " pod="odh-ai-gateway-infra/maas-api-59f756556b-wwlzj"
Sep 10 03:27:34.700677 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:34.700643    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6abe5c8-12f8-4f60-a2cf-ebd47371ae1e-metrics-certs\") pod \"maas-api-59f756556b-wwlzj\" (UID: \"f6abe5c8-12f8-4f60-a2cf-ebd47371ae1e\") " pod="odh-ai-gateway-infra/maas-api-59f756556b-wwlzj"
Sep 10 03:27:34.700835 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:34.700699    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6abe5c8-12f8-4f60-a2cf-ebd47371ae1e-openshift-service-ca\") pod \"maas-api-59f756556b-wwlzj\" (UID: \"f6abe5c8-12f8-4f60-a2cf-ebd47371ae1e\") " pod="odh-ai-gateway-infra/maas-api-59f756556b-wwlzj"
Sep 10 03:27:34.700835 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:34.700746    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6abe5c8-12f8-4f60-a2cf-ebd47371ae1e-maas-api-tls\") pod \"maas-api-59f756556b-wwlzj\" (UID: \"f6abe5c8-12f8-4f60-a2cf-ebd47371ae1e\") " pod="odh-ai-gateway-infra/maas-api-59f756556b-wwlzj"
Sep 10 03:27:34.700835 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:34.700782    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkv6v\" (UniqueName: \"kubernetes.io/projected/f6abe5c8-12f8-4f60-a2cf-ebd47371ae1e-kube-api-access-xkv6v\") pod \"maas-api-59f756556b-wwlzj\" (UID: \"f6abe5c8-12f8-4f60-a2cf-ebd47371ae1e\") " pod="odh-ai-gateway-infra/maas-api-59f756556b-wwlzj"
Sep 10 03:27:34.701590 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:34.701554    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6abe5c8-12f8-4f60-a2cf-ebd47371ae1e-openshift-service-ca\") pod \"maas-api-59f756556b-wwlzj\" (UID: \"f6abe5c8-12f8-4f60-a2cf-ebd47371ae1e\") " pod="odh-ai-gateway-infra/maas-api-59f756556b-wwlzj"
Sep 10 03:27:34.703497 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:34.703473    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6abe5c8-12f8-4f60-a2cf-ebd47371ae1e-metrics-certs\") pod \"maas-api-59f756556b-wwlzj\" (UID: \"f6abe5c8-12f8-4f60-a2cf-ebd47371ae1e\") " pod="odh-ai-gateway-infra/maas-api-59f756556b-wwlzj"
Sep 10 03:27:34.704122 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:34.704073    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6abe5c8-12f8-4f60-a2cf-ebd47371ae1e-maas-api-tls\") pod \"maas-api-59f756556b-wwlzj\" (UID: \"f6abe5c8-12f8-4f60-a2cf-ebd47371ae1e\") " pod="odh-ai-gateway-infra/maas-api-59f756556b-wwlzj"
Sep 10 03:27:34.709522 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:34.709481    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkv6v\" (UniqueName: \"kubernetes.io/projected/f6abe5c8-12f8-4f60-a2cf-ebd47371ae1e-kube-api-access-xkv6v\") pod \"maas-api-59f756556b-wwlzj\" (UID: \"f6abe5c8-12f8-4f60-a2cf-ebd47371ae1e\") " pod="odh-ai-gateway-infra/maas-api-59f756556b-wwlzj"
Sep 10 03:27:34.790203 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:34.790131    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77d77d65f8-9th46"
Sep 10 03:27:34.801349 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:34.801311    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac4560af-2e9c-4897-80b4-fc5d2dc22be2-openshift-service-ca\") pod \"ac4560af-2e9c-4897-80b4-fc5d2dc22be2\" (UID: \"ac4560af-2e9c-4897-80b4-fc5d2dc22be2\") "
Sep 10 03:27:34.801543 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:34.801373    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac4560af-2e9c-4897-80b4-fc5d2dc22be2-metrics-certs\") pod \"ac4560af-2e9c-4897-80b4-fc5d2dc22be2\" (UID: \"ac4560af-2e9c-4897-80b4-fc5d2dc22be2\") "
Sep 10 03:27:34.801543 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:34.801409    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfr68\" (UniqueName: \"kubernetes.io/projected/ac4560af-2e9c-4897-80b4-fc5d2dc22be2-kube-api-access-kfr68\") pod \"ac4560af-2e9c-4897-80b4-fc5d2dc22be2\" (UID: \"ac4560af-2e9c-4897-80b4-fc5d2dc22be2\") "
Sep 10 03:27:34.801543 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:34.801430    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac4560af-2e9c-4897-80b4-fc5d2dc22be2-maas-api-tls\") pod \"ac4560af-2e9c-4897-80b4-fc5d2dc22be2\" (UID: \"ac4560af-2e9c-4897-80b4-fc5d2dc22be2\") "
Sep 10 03:27:34.801796 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:34.801768    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac4560af-2e9c-4897-80b4-fc5d2dc22be2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ac4560af-2e9c-4897-80b4-fc5d2dc22be2" (UID: "ac4560af-2e9c-4897-80b4-fc5d2dc22be2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 03:27:34.803665 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:34.803633    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac4560af-2e9c-4897-80b4-fc5d2dc22be2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ac4560af-2e9c-4897-80b4-fc5d2dc22be2" (UID: "ac4560af-2e9c-4897-80b4-fc5d2dc22be2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:27:34.803665 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:34.803649    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac4560af-2e9c-4897-80b4-fc5d2dc22be2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ac4560af-2e9c-4897-80b4-fc5d2dc22be2" (UID: "ac4560af-2e9c-4897-80b4-fc5d2dc22be2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 03:27:34.803810 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:34.803702    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac4560af-2e9c-4897-80b4-fc5d2dc22be2-kube-api-access-kfr68" (OuterVolumeSpecName: "kube-api-access-kfr68") pod "ac4560af-2e9c-4897-80b4-fc5d2dc22be2" (UID: "ac4560af-2e9c-4897-80b4-fc5d2dc22be2"). InnerVolumeSpecName "kube-api-access-kfr68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 03:27:34.901077 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:34.901032    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f756556b-wwlzj"
Sep 10 03:27:34.902310 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:34.902284    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac4560af-2e9c-4897-80b4-fc5d2dc22be2-openshift-service-ca\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:27:34.902381 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:34.902314    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac4560af-2e9c-4897-80b4-fc5d2dc22be2-metrics-certs\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:27:34.902381 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:34.902324    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfr68\" (UniqueName: \"kubernetes.io/projected/ac4560af-2e9c-4897-80b4-fc5d2dc22be2-kube-api-access-kfr68\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:27:34.902381 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:34.902333    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac4560af-2e9c-4897-80b4-fc5d2dc22be2-maas-api-tls\") on node \"ip-10-0-128-101.ec2.internal\" DevicePath \"\""
Sep 10 03:27:35.062243 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:35.062139    2558 generic.go:356] "Generic (PLEG): container finished" podID="ac4560af-2e9c-4897-80b4-fc5d2dc22be2" containerID="ea915902d6e5308291fc56931fcfbb6f7d1b2977a963fa353ada60be631ef609" exitCode=0
Sep 10 03:27:35.062439 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:35.062266    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77d77d65f8-9th46" event={"ID":"ac4560af-2e9c-4897-80b4-fc5d2dc22be2","Type":"ContainerDied","Data":"ea915902d6e5308291fc56931fcfbb6f7d1b2977a963fa353ada60be631ef609"}
Sep 10 03:27:35.062439 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:35.062313    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77d77d65f8-9th46"
Sep 10 03:27:35.062439 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:35.062326    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77d77d65f8-9th46" event={"ID":"ac4560af-2e9c-4897-80b4-fc5d2dc22be2","Type":"ContainerDied","Data":"e4a8e38356bb775b9210c52a688205351d6d64782651fec44003bf76e8e30b70"}
Sep 10 03:27:35.062439 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:35.062365    2558 scope.go:117] "RemoveContainer" containerID="ea915902d6e5308291fc56931fcfbb6f7d1b2977a963fa353ada60be631ef609"
Sep 10 03:27:35.072191 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:35.072169    2558 scope.go:117] "RemoveContainer" containerID="ea915902d6e5308291fc56931fcfbb6f7d1b2977a963fa353ada60be631ef609"
Sep 10 03:27:35.072615 ip-10-0-128-101 kubenswrapper[2558]: E0910 03:27:35.072569    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea915902d6e5308291fc56931fcfbb6f7d1b2977a963fa353ada60be631ef609\": container with ID starting with ea915902d6e5308291fc56931fcfbb6f7d1b2977a963fa353ada60be631ef609 not found: ID does not exist" containerID="ea915902d6e5308291fc56931fcfbb6f7d1b2977a963fa353ada60be631ef609"
Sep 10 03:27:35.072688 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:35.072626    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea915902d6e5308291fc56931fcfbb6f7d1b2977a963fa353ada60be631ef609"} err="failed to get container status \"ea915902d6e5308291fc56931fcfbb6f7d1b2977a963fa353ada60be631ef609\": rpc error: code = NotFound desc = could not find container \"ea915902d6e5308291fc56931fcfbb6f7d1b2977a963fa353ada60be631ef609\": container with ID starting with ea915902d6e5308291fc56931fcfbb6f7d1b2977a963fa353ada60be631ef609 not found: ID does not exist"
Sep 10 03:27:35.084321 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:35.084274    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-77d77d65f8-9th46"]
Sep 10 03:27:35.087207 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:35.087180    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-77d77d65f8-9th46"]
Sep 10 03:27:35.182550 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:35.182515    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f756556b-wwlzj"]
Sep 10 03:27:35.184832 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:27:35.184799    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6abe5c8_12f8_4f60_a2cf_ebd47371ae1e.slice/crio-826824184f70a343ce28e2e4a7a41b4071828eae890f416f4a127d73cf1a1ac0 WatchSource:0}: Error finding container 826824184f70a343ce28e2e4a7a41b4071828eae890f416f4a127d73cf1a1ac0: Status 404 returned error can't find the container with id 826824184f70a343ce28e2e4a7a41b4071828eae890f416f4a127d73cf1a1ac0
Sep 10 03:27:36.070480 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:36.070446    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac4560af-2e9c-4897-80b4-fc5d2dc22be2" path="/var/lib/kubelet/pods/ac4560af-2e9c-4897-80b4-fc5d2dc22be2/volumes"
Sep 10 03:27:36.072048 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:36.072015    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f756556b-wwlzj" event={"ID":"f6abe5c8-12f8-4f60-a2cf-ebd47371ae1e","Type":"ContainerStarted","Data":"56071ed13e828aba4c05f1655ffc6f40e9d277ae206658a46ebdb438ddb0f08d"}
Sep 10 03:27:36.072208 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:36.072084    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-59f756556b-wwlzj"
Sep 10 03:27:36.072208 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:36.072133    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f756556b-wwlzj" event={"ID":"f6abe5c8-12f8-4f60-a2cf-ebd47371ae1e","Type":"ContainerStarted","Data":"826824184f70a343ce28e2e4a7a41b4071828eae890f416f4a127d73cf1a1ac0"}
Sep 10 03:27:36.088550 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:36.088487    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-59f756556b-wwlzj" podStartSLOduration=1.734540071 podStartE2EDuration="2.08847016s" totalImagesPullingTime="353.930089ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 03:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:27:36.086690584 +0000 UTC m=+2446.461776507" watchObservedRunningTime="2026-09-10 03:27:36.08847016 +0000 UTC m=+2446.463556081"
Sep 10 03:27:42.083922 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:42.083885    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-59f756556b-wwlzj"
Sep 10 03:27:50.742138 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:50.742084    2558 scope.go:117] "RemoveContainer" containerID="34dd886ba7ecc32e3526c3dc5ce5f1d1c1097c5862098a06e17a3d5e3a02c8ae"
Sep 10 03:27:50.750654 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:27:50.750630    2558 scope.go:117] "RemoveContainer" containerID="aea8d27e2a3155182459e4b4bb0ca988b47a1207be0c2ad995051f8faf8c922b"
Sep 10 03:28:01.219390 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:01.219351    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5c96959ff4-44rtg_75ef62e9-cb32-463f-af86-685cc0890b21/authorino/0.log"
Sep 10 03:28:06.588532 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:06.588485    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-mxtj6_4b5dd34b-715d-469c-8e64-6406b64cd509/manager/0.log"
Sep 10 03:28:06.707798 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:06.707757    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-v8f8w_6a49b00d-e027-421f-883e-3cacf773af95/manager/0.log"
Sep 10 03:28:06.964622 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:06.964589    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-6dv7b_65045f32-f1e0-4756-b1dc-d36d99c20175/server/0.log"
Sep 10 03:28:07.085788 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:07.085749    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mzhlk_7eb27485-1656-4e2d-b38f-e59551e49c8b/manager/0.log"
Sep 10 03:28:07.443180 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:07.443063    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84d77ffc9d-tzqhc_eb6047a4-0717-4515-b95e-cbaea1712fe3/copy-manifests/0.log"
Sep 10 03:28:07.461187 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:07.461147    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84d77ffc9d-tzqhc_eb6047a4-0717-4515-b95e-cbaea1712fe3/manager/0.log"
Sep 10 03:28:07.904971 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:07.904935    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-59f756556b-wwlzj_f6abe5c8-12f8-4f60-a2cf-ebd47371ae1e/maas-api/0.log"
Sep 10 03:28:08.239030 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:08.238985    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-xfr2r_65b485c5-f43c-4925-9621-ab193d525ba3/postgres/0.log"
Sep 10 03:28:09.031607 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:09.031546    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5c96959ff4-44rtg_75ef62e9-cb32-463f-af86-685cc0890b21/authorino/0.log"
Sep 10 03:28:09.152688 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:09.152653    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-sv8sd_9f5cecb4-ee84-45f3-a3e9-5ef4367c02b6/manager/0.log"
Sep 10 03:28:09.266583 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:09.266553    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-2m2kv_55057b44-ed7c-40b2-8db8-3e7d2e26993d/manager/0.log"
Sep 10 03:28:09.375320 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:09.375208    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-52nqn_c41ef279-4fec-4d1b-a20d-17144bbf0b09/kuadrant-console-plugin/0.log"
Sep 10 03:28:09.503194 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:09.503154    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-frjj2_eafd907c-b015-46f9-b5f8-79f08dcc51f1/manager/0.log"
Sep 10 03:28:09.755671 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:09.755637    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-27xfc_d75ae269-1b0b-485b-97db-2d005740e830/manager/0.log"
Sep 10 03:28:10.451411 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:10.451378    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-759bbcf6c6-xgrqq_e8968a45-059c-4148-b1fc-195e51b13330/kube-auth-proxy/0.log"
Sep 10 03:28:10.691090 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:10.691051    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-tv2l8_8e7d2343-e490-443b-9e0f-21c7b051a445/payload-pre-processing/0.log"
Sep 10 03:28:10.807731 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:10.807624    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-twvbt_7d5508ae-a8b6-4aa6-a63f-112e565afc69/payload-processing/0.log"
Sep 10 03:28:11.481538 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:11.481492    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-g5gx8_413b0e1a-6827-4487-9a96-cdc23dbbfb24/main/0.log"
Sep 10 03:28:11.594233 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:11.594201    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-rptch_566cc6ae-630a-4ae3-abf9-8fd43be79e8e/main/0.log"
Sep 10 03:28:17.387722 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:17.387675    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:28:18.652271 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:18.652229    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log" limit=1024
Sep 10 03:28:20.896760 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:20.896703    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5qtg6_de3a2ed4-28df-4dea-8e14-c5dfe2981ba5/global-pull-secret-syncer/0.log"
Sep 10 03:28:21.053620 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:21.053573    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-296rx_82b3bbcd-1ae2-4bbc-88a7-61aff07f409a/konnectivity-agent/0.log"
Sep 10 03:28:21.156768 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:21.156661    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-101.ec2.internal_01f46892139817c71935091c0eabf5f5/haproxy/0.log"
Sep 10 03:28:25.637197 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:25.637154    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5c96959ff4-44rtg_75ef62e9-cb32-463f-af86-685cc0890b21/authorino/0.log"
Sep 10 03:28:25.677828 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:25.677790    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-sv8sd_9f5cecb4-ee84-45f3-a3e9-5ef4367c02b6/manager/0.log"
Sep 10 03:28:25.709888 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:25.709857    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-2m2kv_55057b44-ed7c-40b2-8db8-3e7d2e26993d/manager/0.log"
Sep 10 03:28:25.737619 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:25.737583    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-52nqn_c41ef279-4fec-4d1b-a20d-17144bbf0b09/kuadrant-console-plugin/0.log"
Sep 10 03:28:25.784862 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:25.784831    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-frjj2_eafd907c-b015-46f9-b5f8-79f08dcc51f1/manager/0.log"
Sep 10 03:28:25.903319 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:25.903211    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-27xfc_d75ae269-1b0b-485b-97db-2d005740e830/manager/0.log"
Sep 10 03:28:27.983971 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:27.983940    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4bt8z_9b99eefb-4544-4d4c-9087-2bd1723c04b6/node-exporter/0.log"
Sep 10 03:28:28.012850 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:28.012813    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4bt8z_9b99eefb-4544-4d4c-9087-2bd1723c04b6/kube-rbac-proxy/0.log"
Sep 10 03:28:28.042186 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:28.042156    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4bt8z_9b99eefb-4544-4d4c-9087-2bd1723c04b6/init-textfile/0.log"
Sep 10 03:28:30.242381 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:30.242336    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lrsp9/perf-node-gather-daemonset-2gnvf"]
Sep 10 03:28:30.242796 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:30.242777    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac4560af-2e9c-4897-80b4-fc5d2dc22be2" containerName="maas-api"
Sep 10 03:28:30.242796 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:30.242793    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac4560af-2e9c-4897-80b4-fc5d2dc22be2" containerName="maas-api"
Sep 10 03:28:30.242913 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:30.242902    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac4560af-2e9c-4897-80b4-fc5d2dc22be2" containerName="maas-api"
Sep 10 03:28:30.246030 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:30.246010    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-2gnvf"
Sep 10 03:28:30.248439 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:30.248412    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lrsp9\"/\"openshift-service-ca.crt\""
Sep 10 03:28:30.248580 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:30.248476    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lrsp9\"/\"kube-root-ca.crt\""
Sep 10 03:28:30.249069 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:30.249055    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lrsp9\"/\"default-dockercfg-6x58w\""
Sep 10 03:28:30.282390 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:30.282348    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lrsp9/perf-node-gather-daemonset-2gnvf"]
Sep 10 03:28:30.298267 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:30.298229    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqz72\" (UniqueName: \"kubernetes.io/projected/215035d2-e291-408c-95fe-469c44a619c4-kube-api-access-lqz72\") pod \"perf-node-gather-daemonset-2gnvf\" (UID: \"215035d2-e291-408c-95fe-469c44a619c4\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-2gnvf"
Sep 10 03:28:30.298495 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:30.298287    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/215035d2-e291-408c-95fe-469c44a619c4-sys\") pod \"perf-node-gather-daemonset-2gnvf\" (UID: \"215035d2-e291-408c-95fe-469c44a619c4\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-2gnvf"
Sep 10 03:28:30.298495 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:30.298316    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/215035d2-e291-408c-95fe-469c44a619c4-podres\") pod \"perf-node-gather-daemonset-2gnvf\" (UID: \"215035d2-e291-408c-95fe-469c44a619c4\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-2gnvf"
Sep 10 03:28:30.298495 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:30.298420    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/215035d2-e291-408c-95fe-469c44a619c4-lib-modules\") pod \"perf-node-gather-daemonset-2gnvf\" (UID: \"215035d2-e291-408c-95fe-469c44a619c4\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-2gnvf"
Sep 10 03:28:30.298495 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:30.298464    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/215035d2-e291-408c-95fe-469c44a619c4-proc\") pod \"perf-node-gather-daemonset-2gnvf\" (UID: \"215035d2-e291-408c-95fe-469c44a619c4\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-2gnvf"
Sep 10 03:28:30.343981 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:30.343941    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-tcjdj_39430887-0027-4101-86e6-be7bf438d62b/networking-console-plugin/0.log"
Sep 10 03:28:30.399290 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:30.399255    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/215035d2-e291-408c-95fe-469c44a619c4-lib-modules\") pod \"perf-node-gather-daemonset-2gnvf\" (UID: \"215035d2-e291-408c-95fe-469c44a619c4\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-2gnvf"
Sep 10 03:28:30.399290 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:30.399294    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/215035d2-e291-408c-95fe-469c44a619c4-proc\") pod \"perf-node-gather-daemonset-2gnvf\" (UID: \"215035d2-e291-408c-95fe-469c44a619c4\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-2gnvf"
Sep 10 03:28:30.399536 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:30.399386    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/215035d2-e291-408c-95fe-469c44a619c4-proc\") pod \"perf-node-gather-daemonset-2gnvf\" (UID: \"215035d2-e291-408c-95fe-469c44a619c4\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-2gnvf"
Sep 10 03:28:30.399536 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:30.399441    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqz72\" (UniqueName: \"kubernetes.io/projected/215035d2-e291-408c-95fe-469c44a619c4-kube-api-access-lqz72\") pod \"perf-node-gather-daemonset-2gnvf\" (UID: \"215035d2-e291-408c-95fe-469c44a619c4\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-2gnvf"
Sep 10 03:28:30.399536 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:30.399460    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/215035d2-e291-408c-95fe-469c44a619c4-lib-modules\") pod \"perf-node-gather-daemonset-2gnvf\" (UID: \"215035d2-e291-408c-95fe-469c44a619c4\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-2gnvf"
Sep 10 03:28:30.399536 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:30.399518    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/215035d2-e291-408c-95fe-469c44a619c4-sys\") pod \"perf-node-gather-daemonset-2gnvf\" (UID: \"215035d2-e291-408c-95fe-469c44a619c4\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-2gnvf"
Sep 10 03:28:30.399691 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:30.399538    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/215035d2-e291-408c-95fe-469c44a619c4-podres\") pod \"perf-node-gather-daemonset-2gnvf\" (UID: \"215035d2-e291-408c-95fe-469c44a619c4\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-2gnvf"
Sep 10 03:28:30.399691 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:30.399628    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/215035d2-e291-408c-95fe-469c44a619c4-sys\") pod \"perf-node-gather-daemonset-2gnvf\" (UID: \"215035d2-e291-408c-95fe-469c44a619c4\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-2gnvf"
Sep 10 03:28:30.399776 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:30.399707    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/215035d2-e291-408c-95fe-469c44a619c4-podres\") pod \"perf-node-gather-daemonset-2gnvf\" (UID: \"215035d2-e291-408c-95fe-469c44a619c4\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-2gnvf"
Sep 10 03:28:30.410080 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:30.410041    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqz72\" (UniqueName: \"kubernetes.io/projected/215035d2-e291-408c-95fe-469c44a619c4-kube-api-access-lqz72\") pod \"perf-node-gather-daemonset-2gnvf\" (UID: \"215035d2-e291-408c-95fe-469c44a619c4\") " pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-2gnvf"
Sep 10 03:28:30.556371 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:30.556256    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-2gnvf"
Sep 10 03:28:30.840193 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:30.840156    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lrsp9/perf-node-gather-daemonset-2gnvf"]
Sep 10 03:28:30.842093 ip-10-0-128-101 kubenswrapper[2558]: W0910 03:28:30.842061    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod215035d2_e291_408c_95fe_469c44a619c4.slice/crio-59db42db6891dfa15e6b30610a388b48e880d72d24eb6a03adfdb65134cc0a4f WatchSource:0}: Error finding container 59db42db6891dfa15e6b30610a388b48e880d72d24eb6a03adfdb65134cc0a4f: Status 404 returned error can't find the container with id 59db42db6891dfa15e6b30610a388b48e880d72d24eb6a03adfdb65134cc0a4f
Sep 10 03:28:31.388002 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:31.387962    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-2gnvf" event={"ID":"215035d2-e291-408c-95fe-469c44a619c4","Type":"ContainerStarted","Data":"e890fb864566fb2b6033d294ef4e77bf33761fa36b81f6feb76fbd9301212e3b"}
Sep 10 03:28:31.388530 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:31.388018    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-2gnvf" event={"ID":"215035d2-e291-408c-95fe-469c44a619c4","Type":"ContainerStarted","Data":"59db42db6891dfa15e6b30610a388b48e880d72d24eb6a03adfdb65134cc0a4f"}
Sep 10 03:28:31.388530 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:31.388132    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-2gnvf"
Sep 10 03:28:31.413154 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:31.413082    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-2gnvf" podStartSLOduration=1.413066337 podStartE2EDuration="1.413066337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 03:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 03:28:31.411741794 +0000 UTC m=+2501.786827715" watchObservedRunningTime="2026-09-10 03:28:31.413066337 +0000 UTC m=+2501.788152258"
Sep 10 03:28:33.679735 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:33.679697    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pwzr2_27053ebb-2737-4ae1-9aee-c36be0c9ec67/dns/0.log"
Sep 10 03:28:33.705764 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:33.705732    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pwzr2_27053ebb-2737-4ae1-9aee-c36be0c9ec67/kube-rbac-proxy/0.log"
Sep 10 03:28:33.763081 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:33.763048    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rl4nr_f90981c4-23c1-4950-8786-2f4bb0d6e3fb/dns-node-resolver/0.log"
Sep 10 03:28:34.357849 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:34.357810    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4xpxz_c68ab7ea-a147-44d5-81a1-91eca0da2366/node-ca/0.log"
Sep 10 03:28:35.653967 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:35.653935    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-759bbcf6c6-xgrqq_e8968a45-059c-4148-b1fc-195e51b13330/kube-auth-proxy/0.log"
Sep 10 03:28:35.727603 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:35.727565    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-tv2l8_8e7d2343-e490-443b-9e0f-21c7b051a445/payload-pre-processing/0.log"
Sep 10 03:28:35.770369 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:35.770335    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-twvbt_7d5508ae-a8b6-4aa6-a63f-112e565afc69/payload-processing/0.log"
Sep 10 03:28:36.723478 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:36.723444    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qqd6h_cae06c98-aa57-4981-9318-77f97d4aeba6/serve-healthcheck-canary/0.log"
Sep 10 03:28:37.401476 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:37.401438    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pfnd8_82088748-d4dc-4c0b-939c-62b49e6d19d7/kube-rbac-proxy/0.log"
Sep 10 03:28:37.404687 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:37.404657    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lrsp9/perf-node-gather-daemonset-2gnvf"
Sep 10 03:28:37.423466 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:37.423425    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pfnd8_82088748-d4dc-4c0b-939c-62b49e6d19d7/exporter/0.log"
Sep 10 03:28:37.453716 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:37.453680    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pfnd8_82088748-d4dc-4c0b-939c-62b49e6d19d7/extractor/0.log"
Sep 10 03:28:40.030871 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:40.030836    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-mxtj6_4b5dd34b-715d-469c-8e64-6406b64cd509/manager/0.log"
Sep 10 03:28:40.565210 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:40.565171    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-v8f8w_6a49b00d-e027-421f-883e-3cacf773af95/manager/0.log"
Sep 10 03:28:40.644506 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:40.644455    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-6dv7b_65045f32-f1e0-4756-b1dc-d36d99c20175/server/0.log"
Sep 10 03:28:40.807536 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:40.807482    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mzhlk_7eb27485-1656-4e2d-b38f-e59551e49c8b/manager/0.log"
Sep 10 03:28:41.035896 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:41.035851    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84d77ffc9d-tzqhc_eb6047a4-0717-4515-b95e-cbaea1712fe3/manager/0.log"
Sep 10 03:28:41.055182 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:41.055147    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84d77ffc9d-tzqhc_eb6047a4-0717-4515-b95e-cbaea1712fe3/copy-manifests/0.log"
Sep 10 03:28:41.140959 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:41.140917    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-759bbcf6c6-xgrqq_e8968a45-059c-4148-b1fc-195e51b13330/kube-auth-proxy/0.log"
Sep 10 03:28:41.196089 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:41.196050    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-tv2l8_8e7d2343-e490-443b-9e0f-21c7b051a445/payload-pre-processing/0.log"
Sep 10 03:28:41.235904 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:41.235867    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-twvbt_7d5508ae-a8b6-4aa6-a63f-112e565afc69/payload-processing/0.log"
Sep 10 03:28:42.346533 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:42.346501    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-62lbv_737e32f5-3b9d-4cf8-addd-70f13b10c6f4/openshift-lws-operator/0.log"
Sep 10 03:28:43.084528 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:43.084498    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-759bbcf6c6-xgrqq_e8968a45-059c-4148-b1fc-195e51b13330/kube-auth-proxy/0.log"
Sep 10 03:28:43.143806 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:43.143769    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-tv2l8_8e7d2343-e490-443b-9e0f-21c7b051a445/payload-pre-processing/0.log"
Sep 10 03:28:43.177567 ip-10-0-128-101 kubenswrapper[2558]: I0910 03:28:43.177531    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-twvbt_7d5508ae-a8b6-4aa6-a63f-112e565afc69/payload-processing/0.log"