Sep 18 17:12:41.421843 ip-10-0-139-122 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 18 17:12:41.421852 ip-10-0-139-122 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 18 17:12:41.421859 ip-10-0-139-122 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 18 17:12:41.422001 ip-10-0-139-122 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 18 17:12:51.564324 ip-10-0-139-122 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 18 17:12:51.564339 ip-10-0-139-122 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 6d1b319cff4e4266bad95e6e8898a8e4 --
Sep 18 17:14:47.198166 ip-10-0-139-122 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 17:14:47.548999 ip-10-0-139-122 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 17:14:47.548999 ip-10-0-139-122 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 17:14:47.548999 ip-10-0-139-122 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 17:14:47.548999 ip-10-0-139-122 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 17:14:47.548999 ip-10-0-139-122 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 17:14:47.550506 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.550418    2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 17:14:47.555097 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555081    2569 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 17:14:47.555097 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555097    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 17:14:47.555159 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555102    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 17:14:47.555159 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555105    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 17:14:47.555159 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555108    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 17:14:47.555159 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555111    2569 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 17:14:47.555159 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555114    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 17:14:47.555159 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555117    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 17:14:47.555159 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555120    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 17:14:47.555159 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555123    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 17:14:47.555159 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555126    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 17:14:47.555159 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555128    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 17:14:47.555159 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555131    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 17:14:47.555159 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555134    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 17:14:47.555159 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555137    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 17:14:47.555159 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555140    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 17:14:47.555159 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555142    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 17:14:47.555159 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555145    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 17:14:47.555159 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555148    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 17:14:47.555159 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555158    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 17:14:47.555159 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555161    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 17:14:47.555159 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555163    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 17:14:47.555641 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555166    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 17:14:47.555641 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555169    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 17:14:47.555641 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555172    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 17:14:47.555641 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555175    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 17:14:47.555641 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555177    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 17:14:47.555641 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555180    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 17:14:47.555641 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555182    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 17:14:47.555641 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555185    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 17:14:47.555641 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555187    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 17:14:47.555641 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555190    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 17:14:47.555641 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555193    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 17:14:47.555641 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555195    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 17:14:47.555641 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555198    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 17:14:47.555641 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555201    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 17:14:47.555641 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555203    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 17:14:47.555641 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555206    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 17:14:47.555641 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555208    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 17:14:47.555641 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555211    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 17:14:47.555641 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555213    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 17:14:47.555641 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555216    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 17:14:47.556146 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555219    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 17:14:47.556146 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555221    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 17:14:47.556146 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555225    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 17:14:47.556146 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555227    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 17:14:47.556146 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555230    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 17:14:47.556146 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555232    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 17:14:47.556146 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555235    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 17:14:47.556146 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555237    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 17:14:47.556146 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555240    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 17:14:47.556146 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555242    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 17:14:47.556146 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555245    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 17:14:47.556146 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555248    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 17:14:47.556146 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555250    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 17:14:47.556146 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555253    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 17:14:47.556146 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555256    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 17:14:47.556146 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555258    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 17:14:47.556146 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555261    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 17:14:47.556146 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555265    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 17:14:47.556146 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555269    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 17:14:47.556609 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555274    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 17:14:47.556609 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555278    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 17:14:47.556609 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555282    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 17:14:47.556609 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555286    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 17:14:47.556609 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555290    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 17:14:47.556609 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555293    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 17:14:47.556609 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555296    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 17:14:47.556609 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555298    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 17:14:47.556609 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555301    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 17:14:47.556609 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555304    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 17:14:47.556609 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555306    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 17:14:47.556609 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555309    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 17:14:47.556609 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555312    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 17:14:47.556609 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555314    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 17:14:47.556609 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555317    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 17:14:47.556609 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555319    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 17:14:47.556609 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555322    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 17:14:47.556609 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555325    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 17:14:47.556609 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555328    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 17:14:47.557085 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555330    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 17:14:47.557085 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555334    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 17:14:47.557085 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555337    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 17:14:47.557085 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555340    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 17:14:47.557085 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555342    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 17:14:47.557085 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555345    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 17:14:47.557085 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555348    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 17:14:47.557085 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555351    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 17:14:47.557085 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555354    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 17:14:47.557085 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555356    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 17:14:47.557085 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555359    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 17:14:47.557085 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555362    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 17:14:47.557085 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555364    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 17:14:47.557085 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555367    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 17:14:47.557085 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555369    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 17:14:47.557085 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555372    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 17:14:47.557085 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555375    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 17:14:47.557085 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555377    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 17:14:47.557085 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.555379    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 17:14:47.557085 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556856    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 17:14:47.557597 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556874    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 17:14:47.557597 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556878    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 17:14:47.557597 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556881    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 17:14:47.557597 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556884    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 17:14:47.557597 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556887    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 17:14:47.557597 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556890    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 17:14:47.557597 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556893    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 17:14:47.557597 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556898    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 17:14:47.557597 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556902    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 17:14:47.557597 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556905    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 17:14:47.557597 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556908    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 17:14:47.557597 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556911    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 17:14:47.557597 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556914    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 17:14:47.557597 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556918    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 17:14:47.557597 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556921    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 17:14:47.557597 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556923    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 17:14:47.557597 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556926    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 17:14:47.557597 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556929    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 17:14:47.557597 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556931    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 17:14:47.558081 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556934    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 17:14:47.558081 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556937    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 17:14:47.558081 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556940    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 17:14:47.558081 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556943    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 17:14:47.558081 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556945    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 17:14:47.558081 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556948    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 17:14:47.558081 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556950    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 17:14:47.558081 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556953    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 17:14:47.558081 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556955    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 17:14:47.558081 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556958    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 17:14:47.558081 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556961    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 17:14:47.558081 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556964    2569 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 17:14:47.558081 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556966    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 17:14:47.558081 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556969    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 17:14:47.558081 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556971    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 17:14:47.558081 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556974    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 17:14:47.558081 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556977    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 17:14:47.558081 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556979    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 17:14:47.558081 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556982    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 17:14:47.558081 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556984    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 17:14:47.558739 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556987    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 17:14:47.558739 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556990    2569 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 17:14:47.558739 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556992    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 17:14:47.558739 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556995    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 17:14:47.558739 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.556998    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 17:14:47.558739 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557001    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 17:14:47.558739 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557005    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 17:14:47.558739 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557009    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 17:14:47.558739 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557012    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 17:14:47.558739 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557015    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 17:14:47.558739 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557018    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 17:14:47.558739 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557022    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 17:14:47.558739 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557026    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 17:14:47.558739 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557029    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 17:14:47.558739 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557033    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 17:14:47.558739 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557036    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 17:14:47.558739 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557038    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 17:14:47.558739 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557041    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 17:14:47.558739 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557044    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 17:14:47.558739 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557047    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 17:14:47.559400 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557050    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 17:14:47.559400 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557052    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 17:14:47.559400 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557054    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 17:14:47.559400 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557057    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 17:14:47.559400 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557059    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 17:14:47.559400 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557062    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 17:14:47.559400 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557065    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 17:14:47.559400 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557068    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 17:14:47.559400 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557070    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 17:14:47.559400 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557073    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 17:14:47.559400 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557076    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 17:14:47.559400 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557078    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 17:14:47.559400 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557081    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 17:14:47.559400 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557085    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 17:14:47.559400 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557088    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 17:14:47.559400 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557091    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 17:14:47.559400 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557093    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 17:14:47.559400 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557097    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 17:14:47.559400 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557100    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 17:14:47.559881 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557102    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 17:14:47.559881 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557105    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 17:14:47.559881 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557108    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 17:14:47.559881 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557110    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 17:14:47.559881 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557113    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 17:14:47.559881 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557116    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 17:14:47.559881 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557119    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 17:14:47.559881 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557121    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 17:14:47.559881 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557124    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 17:14:47.559881 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557126    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 17:14:47.559881 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557129    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 17:14:47.559881 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557131    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 17:14:47.559881 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557134    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 17:14:47.559881 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557136    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 17:14:47.559881 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557139    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 17:14:47.559881 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557142    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 17:14:47.559881 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557144    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 17:14:47.559881 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557147    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 17:14:47.559881 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557149    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 17:14:47.559881 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557152    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 17:14:47.560373 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557532    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 17:14:47.560373 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557536    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 17:14:47.560373 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557540    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 17:14:47.560373 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557543    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 17:14:47.560373 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557546    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 17:14:47.560373 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557548    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 17:14:47.560373 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557553    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 17:14:47.560373 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557556    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 17:14:47.560373 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557559    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 17:14:47.560373 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557561    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 17:14:47.560373 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557565    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 17:14:47.560373 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557568    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 17:14:47.560373 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557571    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 17:14:47.560373 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557573    2569 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 17:14:47.560373 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557576    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 17:14:47.560373 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557579    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 17:14:47.560373 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557581    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 17:14:47.560373 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557585    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 17:14:47.560373 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557587    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 17:14:47.560373 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557590    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 17:14:47.560879 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557592    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 17:14:47.560879 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557595    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 17:14:47.560879 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557597    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 17:14:47.560879 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557600    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 17:14:47.560879 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557602    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 17:14:47.560879 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557605    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 17:14:47.560879 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557608    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 17:14:47.560879 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557611    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 17:14:47.560879 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557613    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 17:14:47.560879 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557616    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 17:14:47.560879 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557619    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 17:14:47.560879 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557621    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 17:14:47.560879 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557624    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 17:14:47.560879 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557626    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 17:14:47.560879 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557629    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 17:14:47.560879 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557631    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 17:14:47.560879 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557634    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 17:14:47.560879 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557636    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 17:14:47.560879 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557639    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 17:14:47.560879 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557641    2569 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 17:14:47.561374 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557644    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 17:14:47.561374 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557646    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 17:14:47.561374 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557649    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 17:14:47.561374 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557651    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 17:14:47.561374 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557655    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 17:14:47.561374 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557658    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 17:14:47.561374 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557662    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 17:14:47.561374 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557665    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 17:14:47.561374 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557667    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 17:14:47.561374 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557670    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 17:14:47.561374 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557673    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 17:14:47.561374 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557675    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 17:14:47.561374 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557678    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 17:14:47.561374 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557680    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 17:14:47.561374 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557682    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 17:14:47.561374 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557685    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 17:14:47.561374 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557688    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 17:14:47.561374 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557690    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 17:14:47.561374 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557693    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 17:14:47.561830 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557695    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 17:14:47.561830 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557698    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 17:14:47.561830 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557701    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 17:14:47.561830 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557703    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 17:14:47.561830 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557706    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 17:14:47.561830 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557708    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 17:14:47.561830 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557711    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 17:14:47.561830 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557713    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 17:14:47.561830 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557716    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 17:14:47.561830 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557718    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 17:14:47.561830 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557721    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 17:14:47.561830 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557724    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 17:14:47.561830 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557726    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 17:14:47.561830 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557729    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 17:14:47.561830 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557732    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 17:14:47.561830 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557734    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 17:14:47.561830 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557737    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 17:14:47.561830 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557740    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 17:14:47.561830 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557743    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 17:14:47.562336 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557745    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 17:14:47.562336 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557747    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 17:14:47.562336 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557750    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 17:14:47.562336 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557752    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 17:14:47.562336 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557755    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 17:14:47.562336 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557757    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 17:14:47.562336 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557761    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 17:14:47.562336 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557764    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 17:14:47.562336 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557766    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 17:14:47.562336 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557769    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 17:14:47.562336 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557771    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 17:14:47.562336 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557774    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 17:14:47.562336 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557777    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 17:14:47.562336 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557780    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 17:14:47.562336 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557783    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 17:14:47.562336 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557787    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 17:14:47.562336 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557790    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 17:14:47.562336 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557793    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 17:14:47.562336 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557796    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 17:14:47.562799 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557799    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 17:14:47.562799 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.557802    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 17:14:47.562799 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.557877    2569 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 17:14:47.562799 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.557908    2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 17:14:47.562799 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.557915    2569 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 17:14:47.562799 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.557920    2569 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 17:14:47.562799 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.557925    2569 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 17:14:47.562799 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.557928    2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 17:14:47.562799 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.557932    2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 17:14:47.562799 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.557937    2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 17:14:47.562799 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.557941    2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 17:14:47.562799 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.557944    2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 17:14:47.562799 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.557948    2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 17:14:47.562799 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.557951    2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 17:14:47.562799 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.557954    2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 17:14:47.562799 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.557958    2569 flags.go:64] FLAG: --cgroup-root=""
Sep 18 17:14:47.562799 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.557960    2569 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 17:14:47.562799 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.557964    2569 flags.go:64] FLAG: --client-ca-file=""
Sep 18 17:14:47.562799 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.557967    2569 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 17:14:47.562799 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.557969    2569 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 17:14:47.562799 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.557974    2569 flags.go:64] FLAG: --cluster-domain=""
Sep 18 17:14:47.562799 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.557977    2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 17:14:47.562799 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.557980    2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 17:14:47.562799 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.557983    2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 17:14:47.563373 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.557987    2569 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 17:14:47.563373 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.557993    2569 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 17:14:47.563373 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.557996    2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 17:14:47.563373 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.557999    2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 17:14:47.563373 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558002    2569 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 17:14:47.563373 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558005    2569 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 17:14:47.563373 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558008    2569 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 17:14:47.563373 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558011    2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 17:14:47.563373 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558015    2569 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 17:14:47.563373 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558017    2569 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 17:14:47.563373 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558021    2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 17:14:47.563373 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558024    2569 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 17:14:47.563373 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558027    2569 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 17:14:47.563373 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558030    2569 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 17:14:47.563373 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558033    2569 flags.go:64] FLAG: --enable-server="true"
Sep 18 17:14:47.563373 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558036    2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 17:14:47.563373 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558041    2569 flags.go:64] FLAG: --event-burst="100"
Sep 18 17:14:47.563373 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558044    2569 flags.go:64] FLAG: --event-qps="50"
Sep 18 17:14:47.563373 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558047    2569 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 17:14:47.563373 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558050    2569 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 17:14:47.563373 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558053    2569 flags.go:64] FLAG: --eviction-hard=""
Sep 18 17:14:47.563373 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558057    2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 17:14:47.563373 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558060    2569 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 17:14:47.563373 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558063    2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 17:14:47.563373 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558066    2569 flags.go:64] FLAG: --eviction-soft=""
Sep 18 17:14:47.563989 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558069    2569 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 17:14:47.563989 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558071    2569 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 17:14:47.563989 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558074    2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 17:14:47.563989 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558077    2569 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 17:14:47.563989 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558080    2569 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 17:14:47.563989 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558085    2569 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 17:14:47.563989 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558088    2569 flags.go:64] FLAG: --feature-gates=""
Sep 18 17:14:47.563989 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558092    2569 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 17:14:47.563989 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558096    2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 17:14:47.563989 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558099    2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 17:14:47.563989 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558103    2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 17:14:47.563989 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558106    2569 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 17:14:47.563989 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558109    2569 flags.go:64] FLAG: --help="false"
Sep 18 17:14:47.563989 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558112    2569 flags.go:64] FLAG: --hostname-override="ip-10-0-139-122.ec2.internal"
Sep 18 17:14:47.563989 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558115    2569 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 17:14:47.563989 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558118    2569 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 17:14:47.563989 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558121    2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 17:14:47.563989 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558124    2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 17:14:47.563989 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558128    2569 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 17:14:47.563989 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558131    2569 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 17:14:47.563989 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558133    2569 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 17:14:47.563989 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558136    2569 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 17:14:47.563989 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558139    2569 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 17:14:47.564568 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558142    2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 17:14:47.564568 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558145    2569 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 17:14:47.564568 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558444    2569 flags.go:64] FLAG: --kube-reserved=""
Sep 18 17:14:47.564568 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558457    2569 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 17:14:47.564568 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558463    2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 17:14:47.564568 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558469    2569 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 17:14:47.564568 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558485    2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 17:14:47.564568 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558491    2569 flags.go:64] FLAG: --lock-file=""
Sep 18 17:14:47.564568 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558496    2569 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 17:14:47.564568 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558501    2569 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 17:14:47.564568 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558513    2569 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 17:14:47.564568 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558523    2569 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 17:14:47.564568 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558528    2569 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 17:14:47.564568 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558532    2569 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 17:14:47.564568 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558539    2569 flags.go:64] FLAG: --logging-format="text"
Sep 18 17:14:47.564568 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558544    2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 17:14:47.564568 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558550    2569 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 17:14:47.564568 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558556    2569 flags.go:64] FLAG: --manifest-url=""
Sep 18 17:14:47.564568 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558561    2569 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 17:14:47.564568 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558572    2569 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 17:14:47.564568 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558577    2569 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 17:14:47.564568 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558584    2569 flags.go:64] FLAG: --max-pods="110"
Sep 18 17:14:47.564568 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558589    2569 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 17:14:47.564568 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558594    2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 17:14:47.564568 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558609    2569 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 17:14:47.565215 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558615    2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 17:14:47.565215 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558620    2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 17:14:47.565215 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558630    2569 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 17:14:47.565215 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558635    2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 17:14:47.565215 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558647    2569 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 17:14:47.565215 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558651    2569 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 17:14:47.565215 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558657    2569 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 17:14:47.565215 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558662    2569 flags.go:64] FLAG: --pod-cidr=""
Sep 18 17:14:47.565215 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558666    2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 17:14:47.565215 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558680    2569 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 17:14:47.565215 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558685    2569 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 17:14:47.565215 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558690    2569 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 17:14:47.565215 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558695    2569 flags.go:64] FLAG: --port="10250"
Sep 18 17:14:47.565215 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558700    2569 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 17:14:47.565215 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558706    2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03401a6c35a2559b5"
Sep 18 17:14:47.565215 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558711    2569 flags.go:64] FLAG: --qos-reserved=""
Sep 18 17:14:47.565215 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558717    2569 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 17:14:47.565215 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558721    2569 flags.go:64] FLAG: --register-node="true"
Sep 18 17:14:47.565215 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558731    2569 flags.go:64] FLAG: --register-with-taints=""
Sep 18 17:14:47.565215 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558737    2569 flags.go:64] FLAG: --registry-burst="10"
Sep 18 17:14:47.565215 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558742    2569 flags.go:64] FLAG: --registry-qps="5"
Sep 18 17:14:47.565215 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558748    2569 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 17:14:47.565215 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558754    2569 flags.go:64] FLAG: --reserved-memory=""
Sep 18 17:14:47.565215 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558760    2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 17:14:47.565215 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558767    2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 17:14:47.565849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558772    2569 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 17:14:47.565849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558782    2569 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 17:14:47.565849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558787    2569 flags.go:64] FLAG: --runonce="false"
Sep 18 17:14:47.565849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558792    2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 17:14:47.565849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558797    2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 17:14:47.565849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558802    2569 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 17:14:47.565849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558807    2569 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 17:14:47.565849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558813    2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 17:14:47.565849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558818    2569 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 17:14:47.565849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558823    2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 17:14:47.565849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558832    2569 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 17:14:47.565849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558837    2569 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 17:14:47.565849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558842    2569 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 17:14:47.565849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558847    2569 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 17:14:47.565849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558851    2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 17:14:47.565849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558856    2569 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 17:14:47.565849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558877    2569 flags.go:64] FLAG: --system-cgroups=""
Sep 18 17:14:47.565849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558882    2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 17:14:47.565849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558897    2569 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 17:14:47.565849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558902    2569 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 17:14:47.565849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558907    2569 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 17:14:47.565849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558914    2569 flags.go:64] FLAG: --tls-min-version=""
Sep 18 17:14:47.565849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558920    2569 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 17:14:47.565849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558925    2569 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 17:14:47.565849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558930    2569 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 17:14:47.566470 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558938    2569 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 17:14:47.566470 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558954    2569 flags.go:64] FLAG: --v="2"
Sep 18 17:14:47.566470 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558958    2569 flags.go:64] FLAG: --version="false"
Sep 18 17:14:47.566470 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558963    2569 flags.go:64] FLAG: --vmodule=""
Sep 18 17:14:47.566470 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558969    2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 17:14:47.566470 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.558973    2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 17:14:47.566470 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559385    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 17:14:47.566470 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559391    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 17:14:47.566470 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559394    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 17:14:47.566470 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559397    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 17:14:47.566470 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559400    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 17:14:47.566470 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559403    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 17:14:47.566470 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559406    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 17:14:47.566470 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559409    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 17:14:47.566470 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559412    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 17:14:47.566470 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559414    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 17:14:47.566470 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559417    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 17:14:47.566470 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559420    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 17:14:47.566470 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559423    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 17:14:47.566470 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559426    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 17:14:47.566470 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559428    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 17:14:47.567002 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559431    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 17:14:47.567002 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559434    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 17:14:47.567002 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559436    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 17:14:47.567002 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559439    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 17:14:47.567002 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559442    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 17:14:47.567002 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559444    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 17:14:47.567002 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559447    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 17:14:47.567002 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559450    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 17:14:47.567002 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559452    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 17:14:47.567002 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559455    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 17:14:47.567002 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559457    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 17:14:47.567002 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559460    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 17:14:47.567002 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559463    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 17:14:47.567002 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559466    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 17:14:47.567002 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559470    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 17:14:47.567002 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559476    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 17:14:47.567002 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559479    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 17:14:47.567002 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559483    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 17:14:47.567002 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559485    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 17:14:47.567554 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559489    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 17:14:47.567554 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559493    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 17:14:47.567554 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559496    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 17:14:47.567554 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559499    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 17:14:47.567554 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559502    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 17:14:47.567554 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559505    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 17:14:47.567554 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559508    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 17:14:47.567554 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559510    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 17:14:47.567554 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559513    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 17:14:47.567554 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559515    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 17:14:47.567554 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559518    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 17:14:47.567554 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559521    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 17:14:47.567554 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559524    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 17:14:47.567554 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559526    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 17:14:47.567554 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559529    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 17:14:47.567554 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559531    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 17:14:47.567554 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559534    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 17:14:47.567554 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559537    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 17:14:47.567554 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559540    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 17:14:47.568055 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559543    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 17:14:47.568055 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559545    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 17:14:47.568055 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559548    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 17:14:47.568055 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559551    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 17:14:47.568055 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559554    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 17:14:47.568055 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559556    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 17:14:47.568055 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559559    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 17:14:47.568055 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559562    2569 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 17:14:47.568055 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559565    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 17:14:47.568055 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559569    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 17:14:47.568055 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559572    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 17:14:47.568055 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559575    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 17:14:47.568055 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559578    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 17:14:47.568055 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559581    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 17:14:47.568055 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559583    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 17:14:47.568055 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559586    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 17:14:47.568055 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559588    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 17:14:47.568055 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559591    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 17:14:47.568055 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559594    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 17:14:47.568055 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559596    2569 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 17:14:47.568055 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559599    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 17:14:47.568572 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559601    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 17:14:47.568572 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559605    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 17:14:47.568572 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559609    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 17:14:47.568572 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559611    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 17:14:47.568572 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559614    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 17:14:47.568572 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559616    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 17:14:47.568572 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559619    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 17:14:47.568572 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559621    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 17:14:47.568572 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559625    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 17:14:47.568572 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559627    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 17:14:47.568572 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559630    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 17:14:47.568572 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559633    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 17:14:47.568572 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559635    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 17:14:47.568572 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559638    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 17:14:47.568572 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559641    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 17:14:47.568572 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559643    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 17:14:47.568572 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559645    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 17:14:47.568572 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559648    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 17:14:47.568572 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559651    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 17:14:47.568572 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559653    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 17:14:47.569088 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559657    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 17:14:47.569088 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559660    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 17:14:47.569088 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559664    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 17:14:47.569088 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559667    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 17:14:47.569088 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.559670    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 17:14:47.569088 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.560184    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 17:14:47.569088 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.566817    2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 17:14:47.569088 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.566831    2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 17:14:47.569088 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.567608    2569 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 17:14:47.569088 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.567615    2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 17:14:47.569088 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567673    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 17:14:47.569088 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567679    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 17:14:47.569088 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567683    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 17:14:47.569088 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567686    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 17:14:47.569088 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567689    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 17:14:47.569504 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567692    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 17:14:47.569504 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567695    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 17:14:47.569504 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567697    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 17:14:47.569504 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567700    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 17:14:47.569504 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567703    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 17:14:47.569504 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567705    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 17:14:47.569504 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567708    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 17:14:47.569504 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567711    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 17:14:47.569504 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567714    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 17:14:47.569504 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567716    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 17:14:47.569504 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567719    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 17:14:47.569504 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567722    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 17:14:47.569504 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567726    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 17:14:47.569504 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567730    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 17:14:47.569504 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567733    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 17:14:47.569504 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567736    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 17:14:47.569504 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567739    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 17:14:47.569504 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567742    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 17:14:47.569504 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567745    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 17:14:47.569984 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567748    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 17:14:47.569984 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567751    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 17:14:47.569984 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567754    2569 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 17:14:47.569984 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567756    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 17:14:47.569984 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567759    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 17:14:47.569984 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567762    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 17:14:47.569984 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567765    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 17:14:47.569984 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567768    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 17:14:47.569984 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567771    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 17:14:47.569984 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567773    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 17:14:47.569984 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567776    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 17:14:47.569984 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567779    2569 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 17:14:47.569984 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567781    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 17:14:47.569984 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567784    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 17:14:47.569984 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567786    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 17:14:47.569984 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567789    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 17:14:47.569984 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567791    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 17:14:47.569984 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567794    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 17:14:47.569984 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567797    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 17:14:47.569984 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567800    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 17:14:47.570463 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567802    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 17:14:47.570463 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567805    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 17:14:47.570463 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567807    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 17:14:47.570463 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567810    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 17:14:47.570463 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567812    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 17:14:47.570463 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567815    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 17:14:47.570463 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567819    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 17:14:47.570463 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567823    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 17:14:47.570463 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567825    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 17:14:47.570463 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567829    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 17:14:47.570463 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567831    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 17:14:47.570463 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567834    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 17:14:47.570463 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567837    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 17:14:47.570463 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567839    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 17:14:47.570463 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567842    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 17:14:47.570463 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567845    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 17:14:47.570463 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567847    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 17:14:47.570463 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567851    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 17:14:47.570463 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567854    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 17:14:47.570943 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567856    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 17:14:47.570943 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567875    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 17:14:47.570943 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567880    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 17:14:47.570943 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567885    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 17:14:47.570943 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567888    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 17:14:47.570943 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567890    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 17:14:47.570943 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567893    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 17:14:47.570943 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567896    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 17:14:47.570943 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567899    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 17:14:47.570943 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567902    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 17:14:47.570943 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567904    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 17:14:47.570943 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567907    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 17:14:47.570943 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567909    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 17:14:47.570943 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567912    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 17:14:47.570943 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567914    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 17:14:47.570943 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567917    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 17:14:47.570943 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567920    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 17:14:47.570943 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567922    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 17:14:47.570943 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567925    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 17:14:47.570943 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567927    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 17:14:47.571436 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567930    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 17:14:47.571436 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567933    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 17:14:47.571436 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567936    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 17:14:47.571436 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567941    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 17:14:47.571436 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567944    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 17:14:47.571436 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567947    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 17:14:47.571436 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567950    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 17:14:47.571436 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567953    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 17:14:47.571436 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567956    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 17:14:47.571436 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567959    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 17:14:47.571436 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567962    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 17:14:47.571436 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567965    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 17:14:47.571436 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567967    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 17:14:47.571436 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567970    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 17:14:47.571436 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567973    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 17:14:47.571436 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.567976    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 17:14:47.571818 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.567981    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 17:14:47.571818 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568094    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 17:14:47.571818 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568099    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 17:14:47.571818 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568102    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 17:14:47.571818 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568105    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 17:14:47.571818 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568108    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 17:14:47.571818 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568111    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 17:14:47.571818 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568114    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 17:14:47.571818 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568116    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 17:14:47.571818 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568119    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 17:14:47.571818 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568122    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 17:14:47.571818 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568124    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 17:14:47.571818 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568127    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 17:14:47.571818 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568130    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 17:14:47.571818 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568132    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 17:14:47.572337 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568136    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 17:14:47.572337 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568138    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 17:14:47.572337 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568141    2569 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 17:14:47.572337 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568144    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 17:14:47.572337 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568147    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 17:14:47.572337 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568150    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 17:14:47.572337 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568153    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 17:14:47.572337 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568155    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 17:14:47.572337 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568158    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 17:14:47.572337 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568161    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 17:14:47.572337 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568163    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 17:14:47.572337 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568166    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 17:14:47.572337 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568169    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 17:14:47.572337 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568172    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 17:14:47.572337 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568174    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 17:14:47.572337 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568178    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 17:14:47.572337 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568183    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 17:14:47.572337 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568186    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 17:14:47.572337 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568189    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 17:14:47.572337 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568192    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 17:14:47.572836 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568195    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 17:14:47.572836 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568197    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 17:14:47.572836 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568200    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 17:14:47.572836 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568202    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 17:14:47.572836 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568205    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 17:14:47.572836 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568208    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 17:14:47.572836 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568211    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 17:14:47.572836 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568214    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 17:14:47.572836 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568217    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 17:14:47.572836 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568220    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 17:14:47.572836 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568223    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 17:14:47.572836 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568226    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 17:14:47.572836 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568230    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 17:14:47.572836 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568232    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 17:14:47.572836 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568235    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 17:14:47.572836 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568238    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 17:14:47.572836 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568241    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 17:14:47.572836 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568243    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 17:14:47.572836 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568246    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 17:14:47.573301 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568249    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 17:14:47.573301 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568252    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 17:14:47.573301 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568254    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 17:14:47.573301 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568257    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 17:14:47.573301 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568260    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 17:14:47.573301 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568263    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 17:14:47.573301 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568265    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 17:14:47.573301 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568268    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 17:14:47.573301 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568271    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 17:14:47.573301 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568274    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 17:14:47.573301 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568277    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 17:14:47.573301 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568280    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 17:14:47.573301 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568282    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 17:14:47.573301 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568285    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 17:14:47.573301 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568287    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 17:14:47.573301 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568290    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 17:14:47.573301 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568292    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 17:14:47.573301 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568295    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 17:14:47.573301 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568298    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 17:14:47.573301 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568300    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 17:14:47.573790 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568303    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 17:14:47.573790 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568305    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 17:14:47.573790 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568308    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 17:14:47.573790 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568310    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 17:14:47.573790 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568313    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 17:14:47.573790 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568316    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 17:14:47.573790 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568318    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 17:14:47.573790 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568321    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 17:14:47.573790 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568324    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 17:14:47.573790 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568326    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 17:14:47.573790 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568329    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 17:14:47.573790 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568331    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 17:14:47.573790 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568334    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 17:14:47.573790 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568336    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 17:14:47.573790 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568340    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 17:14:47.573790 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568343    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 17:14:47.573790 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568346    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 17:14:47.573790 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568349    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 17:14:47.573790 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568351    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 17:14:47.574258 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568354    2569 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 17:14:47.574258 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568357    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 17:14:47.574258 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568360    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 17:14:47.574258 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568362    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 17:14:47.574258 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568365    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 17:14:47.574258 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568368    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 17:14:47.574258 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:47.568370    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 17:14:47.574258 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.568375    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 17:14:47.574258 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.568474    2569 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 17:14:47.574258 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.570259    2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 17:14:47.574258 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.571068    2569 server.go:1019] "Starting client certificate rotation"
Sep 18 17:14:47.574258 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.571173    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 17:14:47.574258 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.571205    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 17:14:47.574577 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.573200    2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 17:14:47.575968 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.575950    2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 17:14:47.586292 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.586274    2569 log.go:25] "Validated CRI v1 runtime API"
Sep 18 17:14:47.587796 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.587783    2569 log.go:25] "Validated CRI v1 image API"
Sep 18 17:14:47.589565 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.589493    2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 17:14:47.595785 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.595762    2569 fs.go:135] Filesystem UUIDs: map[58d9124b-2e89-4def-81bb-1cd2175471df:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 fdd925cc-b872-477f-a2cb-fde560dc2469:/dev/nvme0n1p4]
Sep 18 17:14:47.595852 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.595784    2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 17:14:47.600499 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.600391    2569 manager.go:216] Machine: {Timestamp:2026-09-18 17:14:47.599367365 +0000 UTC m=+0.293294907 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100037 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25de0513b7aac74d411e7f11ca1717 SystemUUID:ec25de05-13b7-aac7-4d41-1e7f11ca1717 BootID:6d1b319c-ff4e-4266-bad9-5e6e8898a8e4 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fc:80:d2:ed Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fc:80:d2:ed Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9a:b1:db:45:56:a3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 17:14:47.600499 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.600494    2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 17:14:47.600634 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.600582    2569 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 17:14:47.601607 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.601577    2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 17:14:47.601777 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.601609    2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-122.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 17:14:47.602672 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.602661    2569 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 17:14:47.602709 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.602675    2569 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 17:14:47.602709 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.602684    2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 17:14:47.603338 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.603328    2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 17:14:47.603416 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.603408    2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 17:14:47.606511 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.606494    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 17:14:47.606562 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.606545    2569 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 17:14:47.606667 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.606657    2569 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 17:14:47.606712 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.606688    2569 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 17:14:47.606712 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.606697    2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 17:14:47.606712 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.606711    2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 17:14:47.606809 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.606720    2569 kubelet.go:398] "Adding apiserver pod source"
Sep 18 17:14:47.606809 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.606729    2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 17:14:47.607132 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.607120    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 17:14:47.607164 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.607146    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 17:14:47.609689 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.609674    2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 17:14:47.611572 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.611559    2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 17:14:47.611610 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.611588    2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 17:14:47.612511 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.612500    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 17:14:47.612544 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.612517    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 17:14:47.612544 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.612524    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 17:14:47.612544 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.612529    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 17:14:47.612544 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.612538    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 17:14:47.612544 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.612544    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 17:14:47.612673 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.612559    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 17:14:47.612673 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.612564    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 17:14:47.612673 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.612570    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 17:14:47.612673 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.612575    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 17:14:47.612673 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.612592    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 17:14:47.612673 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.612597    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 17:14:47.613417 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.613408    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 17:14:47.613448 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.613419    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 17:14:47.617334 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.617323    2569 server.go:1268] "Started kubelet"
Sep 18 17:14:47.617394 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.617374    2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 17:14:47.617614 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.617529    2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 17:14:47.617698 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.617645    2569 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 17:14:47.617698 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:47.617641    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 17:14:47.617698 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:47.617665    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-122.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 17:14:47.617845 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.617806    2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-122.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 17:14:47.618426 ip-10-0-139-122 systemd[1]: Started Kubernetes Kubelet.
Sep 18 17:14:47.618665 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.618649    2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 17:14:47.619907 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.619893    2569 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 17:14:47.624692 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:47.623830    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-122.ec2.internal.18d679ad6895cd29  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-122.ec2.internal,UID:ip-10-0-139-122.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-122.ec2.internal,},FirstTimestamp:2026-09-18 17:14:47.617301801 +0000 UTC m=+0.311229320,LastTimestamp:2026-09-18 17:14:47.617301801 +0000 UTC m=+0.311229320,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-122.ec2.internal,}"
Sep 18 17:14:47.624882 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.624801    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 17:14:47.625878 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:47.625084    2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 17:14:47.625878 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.625200    2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 17:14:47.625878 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:47.625556    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-122.ec2.internal\" not found"
Sep 18 17:14:47.625878 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.625681    2569 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 17:14:47.625878 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.625693    2569 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 17:14:47.625878 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.625733    2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 17:14:47.626669 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.626649    2569 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 17:14:47.626669 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.626670    2569 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 17:14:47.627351 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.627332    2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 17:14:47.627460 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.627449    2569 factory.go:55] Registering systemd factory
Sep 18 17:14:47.627531 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.627522    2569 factory.go:223] Registration of the systemd container factory successfully
Sep 18 17:14:47.627926 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.627914    2569 factory.go:153] Registering CRI-O factory
Sep 18 17:14:47.628013 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.628005    2569 factory.go:223] Registration of the crio container factory successfully
Sep 18 17:14:47.628099 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.628091    2569 factory.go:103] Registering Raw factory
Sep 18 17:14:47.628187 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.628179    2569 manager.go:1196] Started watching for new ooms in manager
Sep 18 17:14:47.628878 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.628850    2569 manager.go:349] Starting recovery of all containers
Sep 18 17:14:47.630444 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:47.630407    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 17:14:47.630808 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:47.630788    2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-122.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 17:14:47.635070 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.635048    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ddn8d"
Sep 18 17:14:47.641344 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.641183    2569 manager.go:354] Recovery completed
Sep 18 17:14:47.643351 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.643333    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ddn8d"
Sep 18 17:14:47.645702 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.645689    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 17:14:47.647613 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.647595    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 17:14:47.647697 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.647622    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 17:14:47.647697 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.647638    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-122.ec2.internal" event="NodeHasSufficientPID"
Sep 18 17:14:47.649223 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.649207    2569 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 17:14:47.649223 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.649216    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 17:14:47.649223 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.649221    2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 17:14:47.649412 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.649237    2569 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 17:14:47.649620 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:47.649538    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-122.ec2.internal.18d679ad6a644e40  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-122.ec2.internal,UID:ip-10-0-139-122.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-122.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-122.ec2.internal,},FirstTimestamp:2026-09-18 17:14:47.64761248 +0000 UTC m=+0.341539998,LastTimestamp:2026-09-18 17:14:47.64761248 +0000 UTC m=+0.341539998,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-122.ec2.internal,}"
Sep 18 17:14:47.650124 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.650108    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 17:14:47.650198 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.650132    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 17:14:47.650198 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.650141    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-122.ec2.internal" event="NodeHasSufficientPID"
Sep 18 17:14:47.651128 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.651116    2569 policy_none.go:49] "None policy: Start"
Sep 18 17:14:47.651180 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.651132    2569 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 17:14:47.651180 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.651142    2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 17:14:47.652968 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.652957    2569 policy_none.go:47] "Start"
Sep 18 17:14:47.701595 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.676529    2569 manager.go:337] "Starting Device Plugin manager"
Sep 18 17:14:47.701595 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:47.676570    2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 17:14:47.701595 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.676583    2569 server.go:87] "Starting device plugin registration server"
Sep 18 17:14:47.701595 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.676957    2569 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 17:14:47.701595 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.676971    2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 17:14:47.701595 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.677074    2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 17:14:47.701595 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.677141    2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 17:14:47.701595 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.677149    2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 17:14:47.701595 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:47.677754    2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 17:14:47.701595 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:47.677803    2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-122.ec2.internal\" not found"
Sep 18 17:14:47.756795 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.756761    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 17:14:47.758113 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.758096    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 17:14:47.758211 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.758148    2569 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 17:14:47.758211 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.758177    2569 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 17:14:47.758291 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:47.758220    2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 17:14:47.760917 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.760896    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 17:14:47.777304 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.777280    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 17:14:47.778185 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.778166    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 17:14:47.778255 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.778196    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 17:14:47.778255 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.778207    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-122.ec2.internal" event="NodeHasSufficientPID"
Sep 18 17:14:47.778255 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.778229    2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-122.ec2.internal"
Sep 18 17:14:47.791098 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.791075    2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-122.ec2.internal"
Sep 18 17:14:47.791186 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:47.791103    2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-122.ec2.internal\": node \"ip-10-0-139-122.ec2.internal\" not found"
Sep 18 17:14:47.806327 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:47.806255    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-122.ec2.internal\" not found"
Sep 18 17:14:47.858798 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.858771    2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-122.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-122.ec2.internal"]
Sep 18 17:14:47.858937 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.858837    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 17:14:47.859818 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.859803    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 17:14:47.859883 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.859832    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 17:14:47.859883 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.859842    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-122.ec2.internal" event="NodeHasSufficientPID"
Sep 18 17:14:47.862399 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.862387    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 17:14:47.862521 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.862508    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-122.ec2.internal"
Sep 18 17:14:47.862568 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.862538    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 17:14:47.863153 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.863137    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 17:14:47.863234 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.863167    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 17:14:47.863234 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.863178    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-122.ec2.internal" event="NodeHasSufficientPID"
Sep 18 17:14:47.863234 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.863139    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 17:14:47.863234 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.863234    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 17:14:47.863364 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.863244    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-122.ec2.internal" event="NodeHasSufficientPID"
Sep 18 17:14:47.865337 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.865324    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-122.ec2.internal"
Sep 18 17:14:47.865397 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.865345    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 17:14:47.866434 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.866413    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 17:14:47.866524 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.866453    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 17:14:47.866524 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.866469    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-122.ec2.internal" event="NodeHasSufficientPID"
Sep 18 17:14:47.889434 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:47.889412    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-122.ec2.internal\" not found" node="ip-10-0-139-122.ec2.internal"
Sep 18 17:14:47.893785 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:47.893769    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-122.ec2.internal\" not found" node="ip-10-0-139-122.ec2.internal"
Sep 18 17:14:47.906823 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:47.906807    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-122.ec2.internal\" not found"
Sep 18 17:14:47.929327 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.929306    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e251745cdd5e519678a7024bcabf1fdb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-122.ec2.internal\" (UID: \"e251745cdd5e519678a7024bcabf1fdb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-122.ec2.internal"
Sep 18 17:14:47.929416 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.929336    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e251745cdd5e519678a7024bcabf1fdb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-122.ec2.internal\" (UID: \"e251745cdd5e519678a7024bcabf1fdb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-122.ec2.internal"
Sep 18 17:14:47.929416 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:47.929362    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/96da070bb6fafb62858754f170b042c5-config\") pod \"kube-apiserver-proxy-ip-10-0-139-122.ec2.internal\" (UID: \"96da070bb6fafb62858754f170b042c5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-122.ec2.internal"
Sep 18 17:14:48.006976 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:48.006948    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-122.ec2.internal\" not found"
Sep 18 17:14:48.030318 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:48.030294    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/96da070bb6fafb62858754f170b042c5-config\") pod \"kube-apiserver-proxy-ip-10-0-139-122.ec2.internal\" (UID: \"96da070bb6fafb62858754f170b042c5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-122.ec2.internal"
Sep 18 17:14:48.030398 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:48.030324    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e251745cdd5e519678a7024bcabf1fdb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-122.ec2.internal\" (UID: \"e251745cdd5e519678a7024bcabf1fdb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-122.ec2.internal"
Sep 18 17:14:48.030398 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:48.030341    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e251745cdd5e519678a7024bcabf1fdb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-122.ec2.internal\" (UID: \"e251745cdd5e519678a7024bcabf1fdb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-122.ec2.internal"
Sep 18 17:14:48.030499 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:48.030450    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/96da070bb6fafb62858754f170b042c5-config\") pod \"kube-apiserver-proxy-ip-10-0-139-122.ec2.internal\" (UID: \"96da070bb6fafb62858754f170b042c5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-122.ec2.internal"
Sep 18 17:14:48.030499 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:48.030480    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e251745cdd5e519678a7024bcabf1fdb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-122.ec2.internal\" (UID: \"e251745cdd5e519678a7024bcabf1fdb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-122.ec2.internal"
Sep 18 17:14:48.030569 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:48.030461    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e251745cdd5e519678a7024bcabf1fdb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-122.ec2.internal\" (UID: \"e251745cdd5e519678a7024bcabf1fdb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-122.ec2.internal"
Sep 18 17:14:48.107618 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:48.107558    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-122.ec2.internal\" not found"
Sep 18 17:14:48.192117 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:48.192079    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-122.ec2.internal"
Sep 18 17:14:48.197421 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:48.197186    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-122.ec2.internal"
Sep 18 17:14:48.208062 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:48.208041    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-122.ec2.internal\" not found"
Sep 18 17:14:48.310007 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:48.309975    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-122.ec2.internal\" not found"
Sep 18 17:14:48.410587 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:48.410496    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-122.ec2.internal\" not found"
Sep 18 17:14:48.501879 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:48.501851    2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 17:14:48.510911 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:48.510883    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-122.ec2.internal\" not found"
Sep 18 17:14:48.572943 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:48.572911    2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 17:14:48.573428 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:48.573090    2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 17:14:48.573428 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:48.573101    2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 17:14:48.611727 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:48.611700    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-122.ec2.internal\" not found"
Sep 18 17:14:48.648139 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:48.648104    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 17:09:48 +0000 UTC" deadline="2028-02-18 19:59:53.432388858 +0000 UTC"
Sep 18 17:14:48.648139 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:48.648131    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12434h45m4.784260969s"
Sep 18 17:14:48.712523 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:48.712492    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-122.ec2.internal\" not found"
Sep 18 17:14:48.749148 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:48.749120    2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 17:14:48.814616 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:48.814582    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-122.ec2.internal\" not found"
Sep 18 17:14:48.842682 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:48.842648    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96da070bb6fafb62858754f170b042c5.slice/crio-8b810636f5124303d42aefe768df2d0fd4605eb85b665628508a358a7a5803ba WatchSource:0}: Error finding container 8b810636f5124303d42aefe768df2d0fd4605eb85b665628508a358a7a5803ba: Status 404 returned error can't find the container with id 8b810636f5124303d42aefe768df2d0fd4605eb85b665628508a358a7a5803ba
Sep 18 17:14:48.842978 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:48.842948    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode251745cdd5e519678a7024bcabf1fdb.slice/crio-eb1e7366b14ab0d97db0c45ba3b3b570f86351c92acaeb9d2ad7bc193e4f0fda WatchSource:0}: Error finding container eb1e7366b14ab0d97db0c45ba3b3b570f86351c92acaeb9d2ad7bc193e4f0fda: Status 404 returned error can't find the container with id eb1e7366b14ab0d97db0c45ba3b3b570f86351c92acaeb9d2ad7bc193e4f0fda
Sep 18 17:14:48.849669 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:48.849654    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 17:14:48.915182 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:48.915148    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-122.ec2.internal\" not found"
Sep 18 17:14:49.021258 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:49.021186    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-122.ec2.internal\" not found"
Sep 18 17:14:49.121494 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:49.121459    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-122.ec2.internal\" not found"
Sep 18 17:14:49.193579 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.193552    2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 17:14:49.225879 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.225842    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-122.ec2.internal"
Sep 18 17:14:49.237109 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.237087    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 17:14:49.238021 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.238007    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-122.ec2.internal"
Sep 18 17:14:49.244876 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.244852    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 17:14:49.608394 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.608357    2569 apiserver.go:52] "Watching apiserver"
Sep 18 17:14:49.616043 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.616016    2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 17:14:49.616654 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.616627    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-l9knh","openshift-ovn-kubernetes/ovnkube-node-54gxw","kube-system/konnectivity-agent-blmjk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxh7t","openshift-cluster-node-tuning-operator/tuned-5j9tc","openshift-image-registry/node-ca-z6nxd","openshift-multus/multus-additional-cni-plugins-5zx6r","openshift-multus/multus-ps2t8","openshift-multus/network-metrics-daemon-ltwd4","openshift-network-operator/iptables-alerter-xzz8k","kube-system/global-pull-secret-syncer-7kwfm","kube-system/kube-apiserver-proxy-ip-10-0-139-122.ec2.internal","openshift-dns/node-resolver-zjhnk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-122.ec2.internal"]
Sep 18 17:14:49.619713 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.619693    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5zx6r"
Sep 18 17:14:49.622631 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.622610    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-54h9t\""
Sep 18 17:14:49.627324 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.622858    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 17:14:49.627324 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.623209    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 17:14:49.627324 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.623450    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 17:14:49.627324 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.623750    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 17:14:49.627324 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.625133    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 17:14:49.627324 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.625278    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zjhnk"
Sep 18 17:14:49.627324 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.625385    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-blmjk"
Sep 18 17:14:49.627324 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.625846    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 17:14:49.628145 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.628012    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 17:14:49.640041 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.628130    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 17:14:49.641317 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.640370    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxh7t"
Sep 18 17:14:49.641317 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.640473    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ptg44\""
Sep 18 17:14:49.641317 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.641040    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d9b8ea44-62cf-4b12-8a74-f30313e199b4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5zx6r\" (UID: \"d9b8ea44-62cf-4b12-8a74-f30313e199b4\") " pod="openshift-multus/multus-additional-cni-plugins-5zx6r"
Sep 18 17:14:49.641317 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.641084    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-c4zw5\""
Sep 18 17:14:49.641317 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.641225    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 17:14:49.641317 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.641089    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa3b7283-8f5d-4d10-a7ab-6bd10036542f-tmp-dir\") pod \"node-resolver-zjhnk\" (UID: \"fa3b7283-8f5d-4d10-a7ab-6bd10036542f\") " pod="openshift-dns/node-resolver-zjhnk"
Sep 18 17:14:49.641317 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.641294    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7e343c56-b6c8-42a6-bf92-dd46fe5d247b-agent-certs\") pod \"konnectivity-agent-blmjk\" (UID: \"7e343c56-b6c8-42a6-bf92-dd46fe5d247b\") " pod="kube-system/konnectivity-agent-blmjk"
Sep 18 17:14:49.641742 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.641333    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7e343c56-b6c8-42a6-bf92-dd46fe5d247b-konnectivity-ca\") pod \"konnectivity-agent-blmjk\" (UID: \"7e343c56-b6c8-42a6-bf92-dd46fe5d247b\") " pod="kube-system/konnectivity-agent-blmjk"
Sep 18 17:14:49.641742 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.641368    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a2bfa955-c3ee-4d5f-96fd-95f4038f00ee-sys-fs\") pod \"aws-ebs-csi-driver-node-pxh7t\" (UID: \"a2bfa955-c3ee-4d5f-96fd-95f4038f00ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxh7t"
Sep 18 17:14:49.641742 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.641400    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6qhq\" (UniqueName: \"kubernetes.io/projected/a2bfa955-c3ee-4d5f-96fd-95f4038f00ee-kube-api-access-k6qhq\") pod \"aws-ebs-csi-driver-node-pxh7t\" (UID: \"a2bfa955-c3ee-4d5f-96fd-95f4038f00ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxh7t"
Sep 18 17:14:49.641742 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.641432    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d9b8ea44-62cf-4b12-8a74-f30313e199b4-cni-binary-copy\") pod \"multus-additional-cni-plugins-5zx6r\" (UID: \"d9b8ea44-62cf-4b12-8a74-f30313e199b4\") " pod="openshift-multus/multus-additional-cni-plugins-5zx6r"
Sep 18 17:14:49.641742 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.641511    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d9b8ea44-62cf-4b12-8a74-f30313e199b4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5zx6r\" (UID: \"d9b8ea44-62cf-4b12-8a74-f30313e199b4\") " pod="openshift-multus/multus-additional-cni-plugins-5zx6r"
Sep 18 17:14:49.641742 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.641583    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w44p\" (UniqueName: \"kubernetes.io/projected/d9b8ea44-62cf-4b12-8a74-f30313e199b4-kube-api-access-2w44p\") pod \"multus-additional-cni-plugins-5zx6r\" (UID: \"d9b8ea44-62cf-4b12-8a74-f30313e199b4\") " pod="openshift-multus/multus-additional-cni-plugins-5zx6r"
Sep 18 17:14:49.641742 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.641660    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a2bfa955-c3ee-4d5f-96fd-95f4038f00ee-device-dir\") pod \"aws-ebs-csi-driver-node-pxh7t\" (UID: \"a2bfa955-c3ee-4d5f-96fd-95f4038f00ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxh7t"
Sep 18 17:14:49.641742 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.641676    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.641742 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.641692    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a2bfa955-c3ee-4d5f-96fd-95f4038f00ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-pxh7t\" (UID: \"a2bfa955-c3ee-4d5f-96fd-95f4038f00ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxh7t"
Sep 18 17:14:49.641742 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.641744    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9b8ea44-62cf-4b12-8a74-f30313e199b4-system-cni-dir\") pod \"multus-additional-cni-plugins-5zx6r\" (UID: \"d9b8ea44-62cf-4b12-8a74-f30313e199b4\") " pod="openshift-multus/multus-additional-cni-plugins-5zx6r"
Sep 18 17:14:49.644959 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.641783    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d9b8ea44-62cf-4b12-8a74-f30313e199b4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5zx6r\" (UID: \"d9b8ea44-62cf-4b12-8a74-f30313e199b4\") " pod="openshift-multus/multus-additional-cni-plugins-5zx6r"
Sep 18 17:14:49.644959 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.641827    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a2bfa955-c3ee-4d5f-96fd-95f4038f00ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pxh7t\" (UID: \"a2bfa955-c3ee-4d5f-96fd-95f4038f00ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxh7t"
Sep 18 17:14:49.644959 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.641856    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a2bfa955-c3ee-4d5f-96fd-95f4038f00ee-registration-dir\") pod \"aws-ebs-csi-driver-node-pxh7t\" (UID: \"a2bfa955-c3ee-4d5f-96fd-95f4038f00ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxh7t"
Sep 18 17:14:49.644959 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.641897    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d9b8ea44-62cf-4b12-8a74-f30313e199b4-cnibin\") pod \"multus-additional-cni-plugins-5zx6r\" (UID: \"d9b8ea44-62cf-4b12-8a74-f30313e199b4\") " pod="openshift-multus/multus-additional-cni-plugins-5zx6r"
Sep 18 17:14:49.644959 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.641927    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fa3b7283-8f5d-4d10-a7ab-6bd10036542f-hosts-file\") pod \"node-resolver-zjhnk\" (UID: \"fa3b7283-8f5d-4d10-a7ab-6bd10036542f\") " pod="openshift-dns/node-resolver-zjhnk"
Sep 18 17:14:49.644959 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.641960    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz77l\" (UniqueName: \"kubernetes.io/projected/fa3b7283-8f5d-4d10-a7ab-6bd10036542f-kube-api-access-pz77l\") pod \"node-resolver-zjhnk\" (UID: \"fa3b7283-8f5d-4d10-a7ab-6bd10036542f\") " pod="openshift-dns/node-resolver-zjhnk"
Sep 18 17:14:49.644959 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.642033    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a2bfa955-c3ee-4d5f-96fd-95f4038f00ee-socket-dir\") pod \"aws-ebs-csi-driver-node-pxh7t\" (UID: \"a2bfa955-c3ee-4d5f-96fd-95f4038f00ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxh7t"
Sep 18 17:14:49.644959 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.642600    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 17:14:49.644959 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.643671    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 17:14:49.644959 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.643688    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-z25s6\""
Sep 18 17:14:49.644959 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.644118    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 17:14:49.644959 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.644406    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 17:14:49.644959 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.644636    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z6nxd"
Sep 18 17:14:49.646740 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.646720    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-t4ft2\""
Sep 18 17:14:49.647127 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.647109    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltwd4"
Sep 18 17:14:49.647220 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:49.647179    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltwd4" podUID="91a9651b-affa-44e8-8996-1ea02fa0c2c4"
Sep 18 17:14:49.647278 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.647257    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 17:14:49.647901 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.647428    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 17:14:49.649417 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.649398    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9knh"
Sep 18 17:14:49.649510 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:49.649467    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9knh" podUID="3641bd80-d924-469d-a138-2300c1747d0f"
Sep 18 17:14:49.650562 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.650531    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 17:14:49.650903 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.650885    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5rn2p\""
Sep 18 17:14:49.651114 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.651097    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 17:14:49.651349 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.651333    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 17:14:49.651563 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.651541    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 17:14:49.653972 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.653865    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.654719 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.654702    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xzz8k"
Sep 18 17:14:49.656639 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.656618    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 17:14:49.656968 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.656933    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 17:14:49.657208 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.657188    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 17:14:49.657479 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.657460    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m626r\""
Sep 18 17:14:49.657702 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.657687    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:14:49.657767 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:49.657749    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kwfm" podUID="491a6ab9-f579-4c60-a0ea-6b6aaad27d52"
Sep 18 17:14:49.657824 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.657815    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 17:14:49.658067 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.657931    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 17:14:49.658256 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.658211    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 17:14:49.658634 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.658616    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 17:14:49.658886 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.658871    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sqknz\""
Sep 18 17:14:49.659103 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.659086    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 17:14:49.659502 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.659471    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.659983 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.659958    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 17:14:49.662204 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.662184    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 17:14:49.662297 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.662270    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5lxbz\""
Sep 18 17:14:49.672277 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.672252    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7jqqp"
Sep 18 17:14:49.681281 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.681259    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7jqqp"
Sep 18 17:14:49.727248 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.727161    2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 17:14:49.742551 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.742525    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-hostroot\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.742681 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.742566    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-etc-kubernetes\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.742681 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.742597    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvrq6\" (UniqueName: \"kubernetes.io/projected/b0a90b83-ded0-4f9d-9a60-c38f24a22144-kube-api-access-rvrq6\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.742681 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.742622    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-host-cni-netd\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.742843 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.742700    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-ovnkube-config\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.742843 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.742756    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9b8ea44-62cf-4b12-8a74-f30313e199b4-system-cni-dir\") pod \"multus-additional-cni-plugins-5zx6r\" (UID: \"d9b8ea44-62cf-4b12-8a74-f30313e199b4\") " pod="openshift-multus/multus-additional-cni-plugins-5zx6r"
Sep 18 17:14:49.742843 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.742789    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-host-var-lib-cni-multus\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.742843 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.742819    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b0a90b83-ded0-4f9d-9a60-c38f24a22144-multus-daemon-config\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.743077 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.742850    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9c52349-a5c2-4657-834d-e5e51ea9bef5-etc-kubernetes\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.743077 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.742898    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9b8ea44-62cf-4b12-8a74-f30313e199b4-system-cni-dir\") pod \"multus-additional-cni-plugins-5zx6r\" (UID: \"d9b8ea44-62cf-4b12-8a74-f30313e199b4\") " pod="openshift-multus/multus-additional-cni-plugins-5zx6r"
Sep 18 17:14:49.743077 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.742911    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b9c52349-a5c2-4657-834d-e5e51ea9bef5-etc-sysctl-d\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.743077 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.742951    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b9c52349-a5c2-4657-834d-e5e51ea9bef5-etc-sysctl-conf\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.743077 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.742976    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-host-slash\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.743077 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743004    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-run-systemd\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.743077 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743030    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-multus-cni-dir\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.743077 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743056    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-etc-openvswitch\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.743463 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743079    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-run-openvswitch\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.743463 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743104    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-run-ovn\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.743463 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743129    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-ovnkube-script-lib\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.743463 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743162    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d9b8ea44-62cf-4b12-8a74-f30313e199b4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5zx6r\" (UID: \"d9b8ea44-62cf-4b12-8a74-f30313e199b4\") " pod="openshift-multus/multus-additional-cni-plugins-5zx6r"
Sep 18 17:14:49.743463 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743188    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7e343c56-b6c8-42a6-bf92-dd46fe5d247b-konnectivity-ca\") pod \"konnectivity-agent-blmjk\" (UID: \"7e343c56-b6c8-42a6-bf92-dd46fe5d247b\") " pod="kube-system/konnectivity-agent-blmjk"
Sep 18 17:14:49.743463 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743212    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-node-log\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.743463 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743239    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d9b8ea44-62cf-4b12-8a74-f30313e199b4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5zx6r\" (UID: \"d9b8ea44-62cf-4b12-8a74-f30313e199b4\") " pod="openshift-multus/multus-additional-cni-plugins-5zx6r"
Sep 18 17:14:49.743463 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743265    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w44p\" (UniqueName: \"kubernetes.io/projected/d9b8ea44-62cf-4b12-8a74-f30313e199b4-kube-api-access-2w44p\") pod \"multus-additional-cni-plugins-5zx6r\" (UID: \"d9b8ea44-62cf-4b12-8a74-f30313e199b4\") " pod="openshift-multus/multus-additional-cni-plugins-5zx6r"
Sep 18 17:14:49.743463 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743291    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a2bfa955-c3ee-4d5f-96fd-95f4038f00ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-pxh7t\" (UID: \"a2bfa955-c3ee-4d5f-96fd-95f4038f00ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxh7t"
Sep 18 17:14:49.743463 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743317    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9c52349-a5c2-4657-834d-e5e51ea9bef5-sys\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.743463 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743341    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-var-lib-openvswitch\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.743463 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743365    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-host-cni-bin\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.743463 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743388    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-host-var-lib-cni-bin\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.743463 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743419    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b9c52349-a5c2-4657-834d-e5e51ea9bef5-etc-systemd\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.743463 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743442    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-host-run-netns\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.743463 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743464    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7a650e11-ff04-417e-bcb5-37123daa51b7-iptables-alerter-script\") pod \"iptables-alerter-xzz8k\" (UID: \"7a650e11-ff04-417e-bcb5-37123daa51b7\") " pod="openshift-network-operator/iptables-alerter-xzz8k"
Sep 18 17:14:49.744236 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743482    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fa3b7283-8f5d-4d10-a7ab-6bd10036542f-hosts-file\") pod \"node-resolver-zjhnk\" (UID: \"fa3b7283-8f5d-4d10-a7ab-6bd10036542f\") " pod="openshift-dns/node-resolver-zjhnk"
Sep 18 17:14:49.744236 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743497    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a2bfa955-c3ee-4d5f-96fd-95f4038f00ee-socket-dir\") pod \"aws-ebs-csi-driver-node-pxh7t\" (UID: \"a2bfa955-c3ee-4d5f-96fd-95f4038f00ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxh7t"
Sep 18 17:14:49.744236 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743512    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-host-var-lib-kubelet\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.744236 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743540    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9c52349-a5c2-4657-834d-e5e51ea9bef5-var-lib-kubelet\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.744236 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743556    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b9c52349-a5c2-4657-834d-e5e51ea9bef5-etc-tuned\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.744236 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743572    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v5fr\" (UniqueName: \"kubernetes.io/projected/3641bd80-d924-469d-a138-2300c1747d0f-kube-api-access-8v5fr\") pod \"network-check-target-l9knh\" (UID: \"3641bd80-d924-469d-a138-2300c1747d0f\") " pod="openshift-network-diagnostics/network-check-target-l9knh"
Sep 18 17:14:49.744236 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743587    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-env-overrides\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.744236 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743603    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np94r\" (UniqueName: \"kubernetes.io/projected/2af39eb4-05e3-4eca-87d1-87558b5ae1c1-kube-api-access-np94r\") pod \"node-ca-z6nxd\" (UID: \"2af39eb4-05e3-4eca-87d1-87558b5ae1c1\") " pod="openshift-image-registry/node-ca-z6nxd"
Sep 18 17:14:49.744236 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743622    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7e343c56-b6c8-42a6-bf92-dd46fe5d247b-agent-certs\") pod \"konnectivity-agent-blmjk\" (UID: \"7e343c56-b6c8-42a6-bf92-dd46fe5d247b\") " pod="kube-system/konnectivity-agent-blmjk"
Sep 18 17:14:49.744236 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743649    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-multus-conf-dir\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.744236 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743663    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b9c52349-a5c2-4657-834d-e5e51ea9bef5-etc-sysconfig\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.744236 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743678    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91a9651b-affa-44e8-8996-1ea02fa0c2c4-metrics-certs\") pod \"network-metrics-daemon-ltwd4\" (UID: \"91a9651b-affa-44e8-8996-1ea02fa0c2c4\") " pod="openshift-multus/network-metrics-daemon-ltwd4"
Sep 18 17:14:49.744236 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743692    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-log-socket\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.744236 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743706    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/491a6ab9-f579-4c60-a0ea-6b6aaad27d52-dbus\") pod \"global-pull-secret-syncer-7kwfm\" (UID: \"491a6ab9-f579-4c60-a0ea-6b6aaad27d52\") " pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:14:49.744236 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743725    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a2bfa955-c3ee-4d5f-96fd-95f4038f00ee-device-dir\") pod \"aws-ebs-csi-driver-node-pxh7t\" (UID: \"a2bfa955-c3ee-4d5f-96fd-95f4038f00ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxh7t"
Sep 18 17:14:49.744236 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743743    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45xql\" (UniqueName: \"kubernetes.io/projected/b9c52349-a5c2-4657-834d-e5e51ea9bef5-kube-api-access-45xql\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.744236 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743766    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d9b8ea44-62cf-4b12-8a74-f30313e199b4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5zx6r\" (UID: \"d9b8ea44-62cf-4b12-8a74-f30313e199b4\") " pod="openshift-multus/multus-additional-cni-plugins-5zx6r"
Sep 18 17:14:49.745196 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743769    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-ovn-node-metrics-cert\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.745196 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743841    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/491a6ab9-f579-4c60-a0ea-6b6aaad27d52-original-pull-secret\") pod \"global-pull-secret-syncer-7kwfm\" (UID: \"491a6ab9-f579-4c60-a0ea-6b6aaad27d52\") " pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:14:49.745196 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743846    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fa3b7283-8f5d-4d10-a7ab-6bd10036542f-hosts-file\") pod \"node-resolver-zjhnk\" (UID: \"fa3b7283-8f5d-4d10-a7ab-6bd10036542f\") " pod="openshift-dns/node-resolver-zjhnk"
Sep 18 17:14:49.745196 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743863    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d9b8ea44-62cf-4b12-8a74-f30313e199b4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5zx6r\" (UID: \"d9b8ea44-62cf-4b12-8a74-f30313e199b4\") " pod="openshift-multus/multus-additional-cni-plugins-5zx6r"
Sep 18 17:14:49.745196 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743881    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-host-run-multus-certs\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.745196 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743897    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-host-kubelet\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.745196 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743929    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d9b8ea44-62cf-4b12-8a74-f30313e199b4-cnibin\") pod \"multus-additional-cni-plugins-5zx6r\" (UID: \"d9b8ea44-62cf-4b12-8a74-f30313e199b4\") " pod="openshift-multus/multus-additional-cni-plugins-5zx6r"
Sep 18 17:14:49.745196 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743946    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz77l\" (UniqueName: \"kubernetes.io/projected/fa3b7283-8f5d-4d10-a7ab-6bd10036542f-kube-api-access-pz77l\") pod \"node-resolver-zjhnk\" (UID: \"fa3b7283-8f5d-4d10-a7ab-6bd10036542f\") " pod="openshift-dns/node-resolver-zjhnk"
Sep 18 17:14:49.745196 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743963    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-multus-socket-dir-parent\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.745196 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743969    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a2bfa955-c3ee-4d5f-96fd-95f4038f00ee-socket-dir\") pod \"aws-ebs-csi-driver-node-pxh7t\" (UID: \"a2bfa955-c3ee-4d5f-96fd-95f4038f00ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxh7t"
Sep 18 17:14:49.745196 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.743987    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2af39eb4-05e3-4eca-87d1-87558b5ae1c1-host\") pod \"node-ca-z6nxd\" (UID: \"2af39eb4-05e3-4eca-87d1-87558b5ae1c1\") " pod="openshift-image-registry/node-ca-z6nxd"
Sep 18 17:14:49.745196 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.744014    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b9c52349-a5c2-4657-834d-e5e51ea9bef5-run\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.745196 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.744041    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa3b7283-8f5d-4d10-a7ab-6bd10036542f-tmp-dir\") pod \"node-resolver-zjhnk\" (UID: \"fa3b7283-8f5d-4d10-a7ab-6bd10036542f\") " pod="openshift-dns/node-resolver-zjhnk"
Sep 18 17:14:49.745196 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.744066    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a2bfa955-c3ee-4d5f-96fd-95f4038f00ee-sys-fs\") pod \"aws-ebs-csi-driver-node-pxh7t\" (UID: \"a2bfa955-c3ee-4d5f-96fd-95f4038f00ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxh7t"
Sep 18 17:14:49.745196 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.744092    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6qhq\" (UniqueName: \"kubernetes.io/projected/a2bfa955-c3ee-4d5f-96fd-95f4038f00ee-kube-api-access-k6qhq\") pod \"aws-ebs-csi-driver-node-pxh7t\" (UID: \"a2bfa955-c3ee-4d5f-96fd-95f4038f00ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxh7t"
Sep 18 17:14:49.745196 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.744119    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-host-run-netns\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.745196 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.744128    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7e343c56-b6c8-42a6-bf92-dd46fe5d247b-konnectivity-ca\") pod \"konnectivity-agent-blmjk\" (UID: \"7e343c56-b6c8-42a6-bf92-dd46fe5d247b\") " pod="kube-system/konnectivity-agent-blmjk"
Sep 18 17:14:49.746107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.744145    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-systemd-units\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.746107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.744171    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqpg5\" (UniqueName: \"kubernetes.io/projected/91a9651b-affa-44e8-8996-1ea02fa0c2c4-kube-api-access-wqpg5\") pod \"network-metrics-daemon-ltwd4\" (UID: \"91a9651b-affa-44e8-8996-1ea02fa0c2c4\") " pod="openshift-multus/network-metrics-daemon-ltwd4"
Sep 18 17:14:49.746107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.744193    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a2bfa955-c3ee-4d5f-96fd-95f4038f00ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pxh7t\" (UID: \"a2bfa955-c3ee-4d5f-96fd-95f4038f00ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxh7t"
Sep 18 17:14:49.746107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.744221    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a2bfa955-c3ee-4d5f-96fd-95f4038f00ee-registration-dir\") pod \"aws-ebs-csi-driver-node-pxh7t\" (UID: \"a2bfa955-c3ee-4d5f-96fd-95f4038f00ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxh7t"
Sep 18 17:14:49.746107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.744245    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-cnibin\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.746107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.744280    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-host-run-k8s-cni-cncf-io\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.746107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.744307    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b9c52349-a5c2-4657-834d-e5e51ea9bef5-lib-modules\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.746107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.744309    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d9b8ea44-62cf-4b12-8a74-f30313e199b4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5zx6r\" (UID: \"d9b8ea44-62cf-4b12-8a74-f30313e199b4\") " pod="openshift-multus/multus-additional-cni-plugins-5zx6r"
Sep 18 17:14:49.746107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.744334    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.746107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.744395    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7a650e11-ff04-417e-bcb5-37123daa51b7-host-slash\") pod \"iptables-alerter-xzz8k\" (UID: \"7a650e11-ff04-417e-bcb5-37123daa51b7\") " pod="openshift-network-operator/iptables-alerter-xzz8k"
Sep 18 17:14:49.746107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.744427    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9c52349-a5c2-4657-834d-e5e51ea9bef5-host\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.746107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.744438    2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 17:14:49.746107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.744454    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-host-run-ovn-kubernetes\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.746107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.744481    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vvxh\" (UniqueName: \"kubernetes.io/projected/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-kube-api-access-7vvxh\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.746107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.744507    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd7zt\" (UniqueName: \"kubernetes.io/projected/7a650e11-ff04-417e-bcb5-37123daa51b7-kube-api-access-nd7zt\") pod \"iptables-alerter-xzz8k\" (UID: \"7a650e11-ff04-417e-bcb5-37123daa51b7\") " pod="openshift-network-operator/iptables-alerter-xzz8k"
Sep 18 17:14:49.746107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.744533    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/491a6ab9-f579-4c60-a0ea-6b6aaad27d52-kubelet-config\") pod \"global-pull-secret-syncer-7kwfm\" (UID: \"491a6ab9-f579-4c60-a0ea-6b6aaad27d52\") " pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:14:49.746107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.744562    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2af39eb4-05e3-4eca-87d1-87558b5ae1c1-serviceca\") pod \"node-ca-z6nxd\" (UID: \"2af39eb4-05e3-4eca-87d1-87558b5ae1c1\") " pod="openshift-image-registry/node-ca-z6nxd"
Sep 18 17:14:49.747066 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.744588    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-system-cni-dir\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.747066 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.744614    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b9c52349-a5c2-4657-834d-e5e51ea9bef5-etc-modprobe-d\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.747066 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.744641    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9c52349-a5c2-4657-834d-e5e51ea9bef5-tmp\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.747066 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.744671    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d9b8ea44-62cf-4b12-8a74-f30313e199b4-cni-binary-copy\") pod \"multus-additional-cni-plugins-5zx6r\" (UID: \"d9b8ea44-62cf-4b12-8a74-f30313e199b4\") " pod="openshift-multus/multus-additional-cni-plugins-5zx6r"
Sep 18 17:14:49.747066 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.744699    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b0a90b83-ded0-4f9d-9a60-c38f24a22144-cni-binary-copy\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.747066 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.744703    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a2bfa955-c3ee-4d5f-96fd-95f4038f00ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-pxh7t\" (UID: \"a2bfa955-c3ee-4d5f-96fd-95f4038f00ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxh7t"
Sep 18 17:14:49.747066 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.744796    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a2bfa955-c3ee-4d5f-96fd-95f4038f00ee-device-dir\") pod \"aws-ebs-csi-driver-node-pxh7t\" (UID: \"a2bfa955-c3ee-4d5f-96fd-95f4038f00ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxh7t"
Sep 18 17:14:49.747066 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.744875    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d9b8ea44-62cf-4b12-8a74-f30313e199b4-cnibin\") pod \"multus-additional-cni-plugins-5zx6r\" (UID: \"d9b8ea44-62cf-4b12-8a74-f30313e199b4\") " pod="openshift-multus/multus-additional-cni-plugins-5zx6r"
Sep 18 17:14:49.747066 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.744910    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a2bfa955-c3ee-4d5f-96fd-95f4038f00ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pxh7t\" (UID: \"a2bfa955-c3ee-4d5f-96fd-95f4038f00ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxh7t"
Sep 18 17:14:49.747066 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.744927    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a2bfa955-c3ee-4d5f-96fd-95f4038f00ee-sys-fs\") pod \"aws-ebs-csi-driver-node-pxh7t\" (UID: \"a2bfa955-c3ee-4d5f-96fd-95f4038f00ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxh7t"
Sep 18 17:14:49.747066 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.745008    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a2bfa955-c3ee-4d5f-96fd-95f4038f00ee-registration-dir\") pod \"aws-ebs-csi-driver-node-pxh7t\" (UID: \"a2bfa955-c3ee-4d5f-96fd-95f4038f00ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxh7t"
Sep 18 17:14:49.747066 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.745183    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa3b7283-8f5d-4d10-a7ab-6bd10036542f-tmp-dir\") pod \"node-resolver-zjhnk\" (UID: \"fa3b7283-8f5d-4d10-a7ab-6bd10036542f\") " pod="openshift-dns/node-resolver-zjhnk"
Sep 18 17:14:49.747066 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.745257    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d9b8ea44-62cf-4b12-8a74-f30313e199b4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5zx6r\" (UID: \"d9b8ea44-62cf-4b12-8a74-f30313e199b4\") " pod="openshift-multus/multus-additional-cni-plugins-5zx6r"
Sep 18 17:14:49.747066 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.745388    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d9b8ea44-62cf-4b12-8a74-f30313e199b4-cni-binary-copy\") pod \"multus-additional-cni-plugins-5zx6r\" (UID: \"d9b8ea44-62cf-4b12-8a74-f30313e199b4\") " pod="openshift-multus/multus-additional-cni-plugins-5zx6r"
Sep 18 17:14:49.750367 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.750348    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7e343c56-b6c8-42a6-bf92-dd46fe5d247b-agent-certs\") pod \"konnectivity-agent-blmjk\" (UID: \"7e343c56-b6c8-42a6-bf92-dd46fe5d247b\") " pod="kube-system/konnectivity-agent-blmjk"
Sep 18 17:14:49.754573 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.754510    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6qhq\" (UniqueName: \"kubernetes.io/projected/a2bfa955-c3ee-4d5f-96fd-95f4038f00ee-kube-api-access-k6qhq\") pod \"aws-ebs-csi-driver-node-pxh7t\" (UID: \"a2bfa955-c3ee-4d5f-96fd-95f4038f00ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxh7t"
Sep 18 17:14:49.759523 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.755307    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w44p\" (UniqueName: \"kubernetes.io/projected/d9b8ea44-62cf-4b12-8a74-f30313e199b4-kube-api-access-2w44p\") pod \"multus-additional-cni-plugins-5zx6r\" (UID: \"d9b8ea44-62cf-4b12-8a74-f30313e199b4\") " pod="openshift-multus/multus-additional-cni-plugins-5zx6r"
Sep 18 17:14:49.759523 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.758722    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz77l\" (UniqueName: \"kubernetes.io/projected/fa3b7283-8f5d-4d10-a7ab-6bd10036542f-kube-api-access-pz77l\") pod \"node-resolver-zjhnk\" (UID: \"fa3b7283-8f5d-4d10-a7ab-6bd10036542f\") " pod="openshift-dns/node-resolver-zjhnk"
Sep 18 17:14:49.765866 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.765802    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-122.ec2.internal" event={"ID":"e251745cdd5e519678a7024bcabf1fdb","Type":"ContainerStarted","Data":"eb1e7366b14ab0d97db0c45ba3b3b570f86351c92acaeb9d2ad7bc193e4f0fda"}
Sep 18 17:14:49.767972 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.767941    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-122.ec2.internal" event={"ID":"96da070bb6fafb62858754f170b042c5","Type":"ContainerStarted","Data":"8b810636f5124303d42aefe768df2d0fd4605eb85b665628508a358a7a5803ba"}
Sep 18 17:14:49.846093 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846057    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-node-log\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.846245 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846108    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9c52349-a5c2-4657-834d-e5e51ea9bef5-sys\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.846245 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846135    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-var-lib-openvswitch\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.846245 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846162    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-host-cni-bin\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.846245 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846186    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-host-var-lib-cni-bin\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.846245 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846210    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b9c52349-a5c2-4657-834d-e5e51ea9bef5-etc-systemd\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.846245 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846236    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-host-run-netns\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.846610 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846262    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7a650e11-ff04-417e-bcb5-37123daa51b7-iptables-alerter-script\") pod \"iptables-alerter-xzz8k\" (UID: \"7a650e11-ff04-417e-bcb5-37123daa51b7\") " pod="openshift-network-operator/iptables-alerter-xzz8k"
Sep 18 17:14:49.846610 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846289    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-host-var-lib-kubelet\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.846610 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846312    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9c52349-a5c2-4657-834d-e5e51ea9bef5-var-lib-kubelet\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.846610 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846336    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b9c52349-a5c2-4657-834d-e5e51ea9bef5-etc-tuned\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.846610 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846363    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v5fr\" (UniqueName: \"kubernetes.io/projected/3641bd80-d924-469d-a138-2300c1747d0f-kube-api-access-8v5fr\") pod \"network-check-target-l9knh\" (UID: \"3641bd80-d924-469d-a138-2300c1747d0f\") " pod="openshift-network-diagnostics/network-check-target-l9knh"
Sep 18 17:14:49.846610 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846387    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-env-overrides\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.846610 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846414    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np94r\" (UniqueName: \"kubernetes.io/projected/2af39eb4-05e3-4eca-87d1-87558b5ae1c1-kube-api-access-np94r\") pod \"node-ca-z6nxd\" (UID: \"2af39eb4-05e3-4eca-87d1-87558b5ae1c1\") " pod="openshift-image-registry/node-ca-z6nxd"
Sep 18 17:14:49.846610 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846440    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-multus-conf-dir\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.846610 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846464    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b9c52349-a5c2-4657-834d-e5e51ea9bef5-etc-sysconfig\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.846610 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846491    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91a9651b-affa-44e8-8996-1ea02fa0c2c4-metrics-certs\") pod \"network-metrics-daemon-ltwd4\" (UID: \"91a9651b-affa-44e8-8996-1ea02fa0c2c4\") " pod="openshift-multus/network-metrics-daemon-ltwd4"
Sep 18 17:14:49.846610 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846524    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-log-socket\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.846610 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846548    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/491a6ab9-f579-4c60-a0ea-6b6aaad27d52-dbus\") pod \"global-pull-secret-syncer-7kwfm\" (UID: \"491a6ab9-f579-4c60-a0ea-6b6aaad27d52\") " pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:14:49.846610 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846574    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45xql\" (UniqueName: \"kubernetes.io/projected/b9c52349-a5c2-4657-834d-e5e51ea9bef5-kube-api-access-45xql\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.846610 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846598    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-ovn-node-metrics-cert\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.847331 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846625    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/491a6ab9-f579-4c60-a0ea-6b6aaad27d52-original-pull-secret\") pod \"global-pull-secret-syncer-7kwfm\" (UID: \"491a6ab9-f579-4c60-a0ea-6b6aaad27d52\") " pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:14:49.847331 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846653    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-host-run-multus-certs\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.847331 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846677    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-host-kubelet\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.847331 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846707    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-multus-socket-dir-parent\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.847331 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846733    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2af39eb4-05e3-4eca-87d1-87558b5ae1c1-host\") pod \"node-ca-z6nxd\" (UID: \"2af39eb4-05e3-4eca-87d1-87558b5ae1c1\") " pod="openshift-image-registry/node-ca-z6nxd"
Sep 18 17:14:49.847331 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846757    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b9c52349-a5c2-4657-834d-e5e51ea9bef5-run\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.847331 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846789    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-host-run-netns\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.847331 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846812    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-systemd-units\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.847331 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846844    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqpg5\" (UniqueName: \"kubernetes.io/projected/91a9651b-affa-44e8-8996-1ea02fa0c2c4-kube-api-access-wqpg5\") pod \"network-metrics-daemon-ltwd4\" (UID: \"91a9651b-affa-44e8-8996-1ea02fa0c2c4\") " pod="openshift-multus/network-metrics-daemon-ltwd4"
Sep 18 17:14:49.847331 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846875    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-cnibin\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.847331 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846900    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-host-run-k8s-cni-cncf-io\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.847331 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846928    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b9c52349-a5c2-4657-834d-e5e51ea9bef5-lib-modules\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.847331 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846968    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.847331 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.846997    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7a650e11-ff04-417e-bcb5-37123daa51b7-host-slash\") pod \"iptables-alerter-xzz8k\" (UID: \"7a650e11-ff04-417e-bcb5-37123daa51b7\") " pod="openshift-network-operator/iptables-alerter-xzz8k"
Sep 18 17:14:49.847331 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.847021    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9c52349-a5c2-4657-834d-e5e51ea9bef5-host\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.847331 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.847046    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-host-run-ovn-kubernetes\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.847331 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.847074    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vvxh\" (UniqueName: \"kubernetes.io/projected/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-kube-api-access-7vvxh\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.848194 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.847159    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-host-var-lib-kubelet\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.848194 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.847164    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9c52349-a5c2-4657-834d-e5e51ea9bef5-var-lib-kubelet\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.848194 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.847218    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/491a6ab9-f579-4c60-a0ea-6b6aaad27d52-dbus\") pod \"global-pull-secret-syncer-7kwfm\" (UID: \"491a6ab9-f579-4c60-a0ea-6b6aaad27d52\") " pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:14:49.848194 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.847261    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd7zt\" (UniqueName: \"kubernetes.io/projected/7a650e11-ff04-417e-bcb5-37123daa51b7-kube-api-access-nd7zt\") pod \"iptables-alerter-xzz8k\" (UID: \"7a650e11-ff04-417e-bcb5-37123daa51b7\") " pod="openshift-network-operator/iptables-alerter-xzz8k"
Sep 18 17:14:49.848194 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.847265    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-node-log\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.848194 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.847294    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/491a6ab9-f579-4c60-a0ea-6b6aaad27d52-kubelet-config\") pod \"global-pull-secret-syncer-7kwfm\" (UID: \"491a6ab9-f579-4c60-a0ea-6b6aaad27d52\") " pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:14:49.848194 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.847321    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2af39eb4-05e3-4eca-87d1-87558b5ae1c1-serviceca\") pod \"node-ca-z6nxd\" (UID: \"2af39eb4-05e3-4eca-87d1-87558b5ae1c1\") " pod="openshift-image-registry/node-ca-z6nxd"
Sep 18 17:14:49.848194 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.847327    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9c52349-a5c2-4657-834d-e5e51ea9bef5-sys\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.848194 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.847350    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-system-cni-dir\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.848194 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.847374    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-var-lib-openvswitch\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.848194 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.847378    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b9c52349-a5c2-4657-834d-e5e51ea9bef5-etc-modprobe-d\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.848194 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.847420    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-host-cni-bin\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.848194 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.847545    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b9c52349-a5c2-4657-834d-e5e51ea9bef5-etc-modprobe-d\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.848194 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.847584    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-log-socket\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.848194 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.848061    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9c52349-a5c2-4657-834d-e5e51ea9bef5-tmp\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.848194 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:49.848077    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 17:14:49.848194 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.848105    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b0a90b83-ded0-4f9d-9a60-c38f24a22144-cni-binary-copy\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.848194 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.848135    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-hostroot\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.849107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.848135    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7a650e11-ff04-417e-bcb5-37123daa51b7-iptables-alerter-script\") pod \"iptables-alerter-xzz8k\" (UID: \"7a650e11-ff04-417e-bcb5-37123daa51b7\") " pod="openshift-network-operator/iptables-alerter-xzz8k"
Sep 18 17:14:49.849107 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:49.848175    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/491a6ab9-f579-4c60-a0ea-6b6aaad27d52-original-pull-secret podName:491a6ab9-f579-4c60-a0ea-6b6aaad27d52 nodeName:}" failed. No retries permitted until 2026-09-18 17:14:50.348134249 +0000 UTC m=+3.042061769 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/491a6ab9-f579-4c60-a0ea-6b6aaad27d52-original-pull-secret") pod "global-pull-secret-syncer-7kwfm" (UID: "491a6ab9-f579-4c60-a0ea-6b6aaad27d52") : object "kube-system"/"original-pull-secret" not registered
Sep 18 17:14:49.849107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.848180    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-hostroot\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.849107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.848207    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-host-var-lib-cni-bin\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.849107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.848224    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-host-run-multus-certs\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.849107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.848305    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-host-kubelet\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.849107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.848325    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b9c52349-a5c2-4657-834d-e5e51ea9bef5-run\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.849107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.848334    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-multus-socket-dir-parent\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.849107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.848360    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-systemd-units\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.849107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.848368    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-host-run-netns\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.849107 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:49.848427    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 17:14:49.849107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.848467    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-host-run-k8s-cni-cncf-io\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.849107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.848541    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-system-cni-dir\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.849107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.848621    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b9c52349-a5c2-4657-834d-e5e51ea9bef5-lib-modules\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.849107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.848700    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b9c52349-a5c2-4657-834d-e5e51ea9bef5-etc-sysconfig\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.849107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.848744    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.849107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.848756    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-host-run-ovn-kubernetes\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.850010 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.848796    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-etc-kubernetes\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.850010 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.848804    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b0a90b83-ded0-4f9d-9a60-c38f24a22144-cni-binary-copy\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.850010 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.848825    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvrq6\" (UniqueName: \"kubernetes.io/projected/b0a90b83-ded0-4f9d-9a60-c38f24a22144-kube-api-access-rvrq6\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.850010 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.848855    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-host-cni-netd\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.850010 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.848862    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-cnibin\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.850010 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.848882    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-ovnkube-config\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.850010 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.848910    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-host-var-lib-cni-multus\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.850010 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.848952    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b0a90b83-ded0-4f9d-9a60-c38f24a22144-multus-daemon-config\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.850010 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.848824    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/491a6ab9-f579-4c60-a0ea-6b6aaad27d52-kubelet-config\") pod \"global-pull-secret-syncer-7kwfm\" (UID: \"491a6ab9-f579-4c60-a0ea-6b6aaad27d52\") " pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:14:49.850010 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.849033    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9c52349-a5c2-4657-834d-e5e51ea9bef5-etc-kubernetes\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.850010 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.849062    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b9c52349-a5c2-4657-834d-e5e51ea9bef5-etc-sysctl-d\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.850010 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.849067    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-multus-conf-dir\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.850010 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.849088    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b9c52349-a5c2-4657-834d-e5e51ea9bef5-etc-sysctl-conf\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.850010 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.849114    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-host-slash\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.850010 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.849143    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-run-systemd\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.850010 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.849151    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-host-cni-netd\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.850010 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.849171    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-multus-cni-dir\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.850010 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.849199    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-etc-openvswitch\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.850880 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.849226    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-run-openvswitch\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.850880 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.849255    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-run-ovn\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.850880 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.849285    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-env-overrides\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.850880 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.849306    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2af39eb4-05e3-4eca-87d1-87558b5ae1c1-serviceca\") pod \"node-ca-z6nxd\" (UID: \"2af39eb4-05e3-4eca-87d1-87558b5ae1c1\") " pod="openshift-image-registry/node-ca-z6nxd"
Sep 18 17:14:49.850880 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.849117    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7a650e11-ff04-417e-bcb5-37123daa51b7-host-slash\") pod \"iptables-alerter-xzz8k\" (UID: \"7a650e11-ff04-417e-bcb5-37123daa51b7\") " pod="openshift-network-operator/iptables-alerter-xzz8k"
Sep 18 17:14:49.850880 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.849347    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-ovnkube-script-lib\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.850880 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.849361    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-host-slash\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.850880 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.849390    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9c52349-a5c2-4657-834d-e5e51ea9bef5-host\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.850880 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.849394    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-run-ovn\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.850880 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.849348    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-host-run-netns\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.850880 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.849429    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-run-openvswitch\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.850880 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.849441    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-run-systemd\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.850880 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.849475    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-etc-openvswitch\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.850880 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:49.849489    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91a9651b-affa-44e8-8996-1ea02fa0c2c4-metrics-certs podName:91a9651b-affa-44e8-8996-1ea02fa0c2c4 nodeName:}" failed. No retries permitted until 2026-09-18 17:14:50.349472524 +0000 UTC m=+3.043400032 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91a9651b-affa-44e8-8996-1ea02fa0c2c4-metrics-certs") pod "network-metrics-daemon-ltwd4" (UID: "91a9651b-affa-44e8-8996-1ea02fa0c2c4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 17:14:49.851622 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.849522    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-multus-cni-dir\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.851622 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.849536    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b9c52349-a5c2-4657-834d-e5e51ea9bef5-etc-systemd\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.851622 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.849651    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b0a90b83-ded0-4f9d-9a60-c38f24a22144-multus-daemon-config\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.851622 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.849690    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-etc-kubernetes\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.851622 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.849898    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-ovnkube-config\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.851622 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.849964    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9c52349-a5c2-4657-834d-e5e51ea9bef5-etc-kubernetes\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.851622 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.850018    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-ovnkube-script-lib\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.851622 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.850033    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b9c52349-a5c2-4657-834d-e5e51ea9bef5-etc-sysctl-d\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.851622 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.850055    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b0a90b83-ded0-4f9d-9a60-c38f24a22144-host-var-lib-cni-multus\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.851622 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.850067    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-ovn-node-metrics-cert\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.851622 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.850103    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b9c52349-a5c2-4657-834d-e5e51ea9bef5-etc-sysctl-conf\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.851622 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.850196    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2af39eb4-05e3-4eca-87d1-87558b5ae1c1-host\") pod \"node-ca-z6nxd\" (UID: \"2af39eb4-05e3-4eca-87d1-87558b5ae1c1\") " pod="openshift-image-registry/node-ca-z6nxd"
Sep 18 17:14:49.854771 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.854441    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9c52349-a5c2-4657-834d-e5e51ea9bef5-tmp\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.856120 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:49.856095    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 17:14:49.856120 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:49.856120    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 17:14:49.856271 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:49.856134    2569 projected.go:196] Error preparing data for projected volume kube-api-access-8v5fr for pod openshift-network-diagnostics/network-check-target-l9knh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 17:14:49.856271 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:49.856199    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3641bd80-d924-469d-a138-2300c1747d0f-kube-api-access-8v5fr podName:3641bd80-d924-469d-a138-2300c1747d0f nodeName:}" failed. No retries permitted until 2026-09-18 17:14:50.35618326 +0000 UTC m=+3.050110786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8v5fr" (UniqueName: "kubernetes.io/projected/3641bd80-d924-469d-a138-2300c1747d0f-kube-api-access-8v5fr") pod "network-check-target-l9knh" (UID: "3641bd80-d924-469d-a138-2300c1747d0f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 17:14:49.857734 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.857688    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45xql\" (UniqueName: \"kubernetes.io/projected/b9c52349-a5c2-4657-834d-e5e51ea9bef5-kube-api-access-45xql\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.858432 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.858275    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b9c52349-a5c2-4657-834d-e5e51ea9bef5-etc-tuned\") pod \"tuned-5j9tc\" (UID: \"b9c52349-a5c2-4657-834d-e5e51ea9bef5\") " pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.858432 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.858339    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vvxh\" (UniqueName: \"kubernetes.io/projected/7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445-kube-api-access-7vvxh\") pod \"ovnkube-node-54gxw\" (UID: \"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445\") " pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:49.860260 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.860233    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd7zt\" (UniqueName: \"kubernetes.io/projected/7a650e11-ff04-417e-bcb5-37123daa51b7-kube-api-access-nd7zt\") pod \"iptables-alerter-xzz8k\" (UID: \"7a650e11-ff04-417e-bcb5-37123daa51b7\") " pod="openshift-network-operator/iptables-alerter-xzz8k"
Sep 18 17:14:49.860534 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.860507    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvrq6\" (UniqueName: \"kubernetes.io/projected/b0a90b83-ded0-4f9d-9a60-c38f24a22144-kube-api-access-rvrq6\") pod \"multus-ps2t8\" (UID: \"b0a90b83-ded0-4f9d-9a60-c38f24a22144\") " pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:49.861237 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.861194    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqpg5\" (UniqueName: \"kubernetes.io/projected/91a9651b-affa-44e8-8996-1ea02fa0c2c4-kube-api-access-wqpg5\") pod \"network-metrics-daemon-ltwd4\" (UID: \"91a9651b-affa-44e8-8996-1ea02fa0c2c4\") " pod="openshift-multus/network-metrics-daemon-ltwd4"
Sep 18 17:14:49.863227 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.863181    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-np94r\" (UniqueName: \"kubernetes.io/projected/2af39eb4-05e3-4eca-87d1-87558b5ae1c1-kube-api-access-np94r\") pod \"node-ca-z6nxd\" (UID: \"2af39eb4-05e3-4eca-87d1-87558b5ae1c1\") " pod="openshift-image-registry/node-ca-z6nxd"
Sep 18 17:14:49.944723 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.944687    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5zx6r"
Sep 18 17:14:49.956486 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.956460    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zjhnk"
Sep 18 17:14:49.967426 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.967405    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-blmjk"
Sep 18 17:14:49.975564 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.975543    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxh7t"
Sep 18 17:14:49.983513 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.983491    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5j9tc"
Sep 18 17:14:49.994438 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:49.994417    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z6nxd"
Sep 18 17:14:50.002463 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:50.002428    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:14:50.013242 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:50.013210    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xzz8k"
Sep 18 17:14:50.019145 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:50.019128    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ps2t8"
Sep 18 17:14:50.094463 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:50.094425    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 17:14:50.354238 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:50.354152    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91a9651b-affa-44e8-8996-1ea02fa0c2c4-metrics-certs\") pod \"network-metrics-daemon-ltwd4\" (UID: \"91a9651b-affa-44e8-8996-1ea02fa0c2c4\") " pod="openshift-multus/network-metrics-daemon-ltwd4"
Sep 18 17:14:50.354238 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:50.354193    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/491a6ab9-f579-4c60-a0ea-6b6aaad27d52-original-pull-secret\") pod \"global-pull-secret-syncer-7kwfm\" (UID: \"491a6ab9-f579-4c60-a0ea-6b6aaad27d52\") " pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:14:50.354475 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:50.354311    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 17:14:50.354475 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:50.354318    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 17:14:50.354475 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:50.354371    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/491a6ab9-f579-4c60-a0ea-6b6aaad27d52-original-pull-secret podName:491a6ab9-f579-4c60-a0ea-6b6aaad27d52 nodeName:}" failed. No retries permitted until 2026-09-18 17:14:51.354358027 +0000 UTC m=+4.048285536 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/491a6ab9-f579-4c60-a0ea-6b6aaad27d52-original-pull-secret") pod "global-pull-secret-syncer-7kwfm" (UID: "491a6ab9-f579-4c60-a0ea-6b6aaad27d52") : object "kube-system"/"original-pull-secret" not registered
Sep 18 17:14:50.354475 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:50.354388    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91a9651b-affa-44e8-8996-1ea02fa0c2c4-metrics-certs podName:91a9651b-affa-44e8-8996-1ea02fa0c2c4 nodeName:}" failed. No retries permitted until 2026-09-18 17:14:51.354381088 +0000 UTC m=+4.048308595 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91a9651b-affa-44e8-8996-1ea02fa0c2c4-metrics-certs") pod "network-metrics-daemon-ltwd4" (UID: "91a9651b-affa-44e8-8996-1ea02fa0c2c4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 17:14:50.454841 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:50.454806    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v5fr\" (UniqueName: \"kubernetes.io/projected/3641bd80-d924-469d-a138-2300c1747d0f-kube-api-access-8v5fr\") pod \"network-check-target-l9knh\" (UID: \"3641bd80-d924-469d-a138-2300c1747d0f\") " pod="openshift-network-diagnostics/network-check-target-l9knh"
Sep 18 17:14:50.455050 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:50.455013    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 17:14:50.455050 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:50.455045    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 17:14:50.455157 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:50.455059    2569 projected.go:196] Error preparing data for projected volume kube-api-access-8v5fr for pod openshift-network-diagnostics/network-check-target-l9knh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 17:14:50.455157 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:50.455128    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3641bd80-d924-469d-a138-2300c1747d0f-kube-api-access-8v5fr podName:3641bd80-d924-469d-a138-2300c1747d0f nodeName:}" failed. No retries permitted until 2026-09-18 17:14:51.455107939 +0000 UTC m=+4.149035450 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8v5fr" (UniqueName: "kubernetes.io/projected/3641bd80-d924-469d-a138-2300c1747d0f-kube-api-access-8v5fr") pod "network-check-target-l9knh" (UID: "3641bd80-d924-469d-a138-2300c1747d0f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 17:14:50.684621 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:50.684533    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 17:09:50 +0000 UTC" deadline="2028-02-13 11:51:01.700795828 +0000 UTC"
Sep 18 17:14:50.684621 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:50.684563    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12306h36m11.01623606s"
Sep 18 17:14:50.758462 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:50.758430    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltwd4"
Sep 18 17:14:50.758616 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:50.758548    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltwd4" podUID="91a9651b-affa-44e8-8996-1ea02fa0c2c4"
Sep 18 17:14:50.932881 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:50.932823    2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 17:14:51.002269 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:51.002235    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2af39eb4_05e3_4eca_87d1_87558b5ae1c1.slice/crio-f0fb90141ed2dfa4da7b83518a1b7518a78d94955174a53b5e5b77c34302065b WatchSource:0}: Error finding container f0fb90141ed2dfa4da7b83518a1b7518a78d94955174a53b5e5b77c34302065b: Status 404 returned error can't find the container with id f0fb90141ed2dfa4da7b83518a1b7518a78d94955174a53b5e5b77c34302065b
Sep 18 17:14:51.004470 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:51.004197    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a90b83_ded0_4f9d_9a60_c38f24a22144.slice/crio-6072027d158f298cc73773491a98dcb5a65ae51296d1735db05d3f1c435afc9b WatchSource:0}: Error finding container 6072027d158f298cc73773491a98dcb5a65ae51296d1735db05d3f1c435afc9b: Status 404 returned error can't find the container with id 6072027d158f298cc73773491a98dcb5a65ae51296d1735db05d3f1c435afc9b
Sep 18 17:14:51.005594 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:51.005542    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c52349_a5c2_4657_834d_e5e51ea9bef5.slice/crio-a29b893df34bfb09283aed60c9a4306130cab156f4db14256a9bd6337d498854 WatchSource:0}: Error finding container a29b893df34bfb09283aed60c9a4306130cab156f4db14256a9bd6337d498854: Status 404 returned error can't find the container with id a29b893df34bfb09283aed60c9a4306130cab156f4db14256a9bd6337d498854
Sep 18 17:14:51.033385 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:51.033351    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa3b7283_8f5d_4d10_a7ab_6bd10036542f.slice/crio-f669baad54dcc3f9b97ffa4cd17c88c550ede1f07037fd2c3e52e605c08163ad WatchSource:0}: Error finding container f669baad54dcc3f9b97ffa4cd17c88c550ede1f07037fd2c3e52e605c08163ad: Status 404 returned error can't find the container with id f669baad54dcc3f9b97ffa4cd17c88c550ede1f07037fd2c3e52e605c08163ad
Sep 18 17:14:51.034249 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:51.034216    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7da7bff1_ba0a_4078_9ef3_f3fb7f2d7445.slice/crio-63841ac6dd49cd023d50bea24792789039c51c0de3183492a0375fe47e04f152 WatchSource:0}: Error finding container 63841ac6dd49cd023d50bea24792789039c51c0de3183492a0375fe47e04f152: Status 404 returned error can't find the container with id 63841ac6dd49cd023d50bea24792789039c51c0de3183492a0375fe47e04f152
Sep 18 17:14:51.035090 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:51.035057    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2bfa955_c3ee_4d5f_96fd_95f4038f00ee.slice/crio-1ce024955ac1c1b9642d6091237fd4d6043bb2f39b07ceea66f0a0e68641177e WatchSource:0}: Error finding container 1ce024955ac1c1b9642d6091237fd4d6043bb2f39b07ceea66f0a0e68641177e: Status 404 returned error can't find the container with id 1ce024955ac1c1b9642d6091237fd4d6043bb2f39b07ceea66f0a0e68641177e
Sep 18 17:14:51.035757 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:51.035730    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e343c56_b6c8_42a6_bf92_dd46fe5d247b.slice/crio-72d241888aa417c5c805c87333869fbcaa67110a5498f6d869259cc78c329ac4 WatchSource:0}: Error finding container 72d241888aa417c5c805c87333869fbcaa67110a5498f6d869259cc78c329ac4: Status 404 returned error can't find the container with id 72d241888aa417c5c805c87333869fbcaa67110a5498f6d869259cc78c329ac4
Sep 18 17:14:51.037151 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:51.037047    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9b8ea44_62cf_4b12_8a74_f30313e199b4.slice/crio-8aec6d56cac75c1f71fd97089aa6b58cbca31c3f7e647ac353cfedfe34a9d7a7 WatchSource:0}: Error finding container 8aec6d56cac75c1f71fd97089aa6b58cbca31c3f7e647ac353cfedfe34a9d7a7: Status 404 returned error can't find the container with id 8aec6d56cac75c1f71fd97089aa6b58cbca31c3f7e647ac353cfedfe34a9d7a7
Sep 18 17:14:51.037862 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:14:51.037838    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a650e11_ff04_417e_bcb5_37123daa51b7.slice/crio-0ea9eb9279cec39ad17dd8bd9d75b0d6da6a1590d84013b846b0bde185d54c88 WatchSource:0}: Error finding container 0ea9eb9279cec39ad17dd8bd9d75b0d6da6a1590d84013b846b0bde185d54c88: Status 404 returned error can't find the container with id 0ea9eb9279cec39ad17dd8bd9d75b0d6da6a1590d84013b846b0bde185d54c88
Sep 18 17:14:51.362512 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:51.362466    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91a9651b-affa-44e8-8996-1ea02fa0c2c4-metrics-certs\") pod \"network-metrics-daemon-ltwd4\" (UID: \"91a9651b-affa-44e8-8996-1ea02fa0c2c4\") " pod="openshift-multus/network-metrics-daemon-ltwd4"
Sep 18 17:14:51.362512 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:51.362508    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/491a6ab9-f579-4c60-a0ea-6b6aaad27d52-original-pull-secret\") pod \"global-pull-secret-syncer-7kwfm\" (UID: \"491a6ab9-f579-4c60-a0ea-6b6aaad27d52\") " pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:14:51.362774 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:51.362648    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 17:14:51.362774 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:51.362696    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 17:14:51.362774 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:51.362731    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91a9651b-affa-44e8-8996-1ea02fa0c2c4-metrics-certs podName:91a9651b-affa-44e8-8996-1ea02fa0c2c4 nodeName:}" failed. No retries permitted until 2026-09-18 17:14:53.36271037 +0000 UTC m=+6.056637897 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91a9651b-affa-44e8-8996-1ea02fa0c2c4-metrics-certs") pod "network-metrics-daemon-ltwd4" (UID: "91a9651b-affa-44e8-8996-1ea02fa0c2c4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 17:14:51.362774 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:51.362753    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/491a6ab9-f579-4c60-a0ea-6b6aaad27d52-original-pull-secret podName:491a6ab9-f579-4c60-a0ea-6b6aaad27d52 nodeName:}" failed. No retries permitted until 2026-09-18 17:14:53.362742408 +0000 UTC m=+6.056669936 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/491a6ab9-f579-4c60-a0ea-6b6aaad27d52-original-pull-secret") pod "global-pull-secret-syncer-7kwfm" (UID: "491a6ab9-f579-4c60-a0ea-6b6aaad27d52") : object "kube-system"/"original-pull-secret" not registered
Sep 18 17:14:51.463614 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:51.463578    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v5fr\" (UniqueName: \"kubernetes.io/projected/3641bd80-d924-469d-a138-2300c1747d0f-kube-api-access-8v5fr\") pod \"network-check-target-l9knh\" (UID: \"3641bd80-d924-469d-a138-2300c1747d0f\") " pod="openshift-network-diagnostics/network-check-target-l9knh"
Sep 18 17:14:51.463784 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:51.463730    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 17:14:51.463784 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:51.463746    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 17:14:51.463784 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:51.463756    2569 projected.go:196] Error preparing data for projected volume kube-api-access-8v5fr for pod openshift-network-diagnostics/network-check-target-l9knh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 17:14:51.463943 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:51.463808    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3641bd80-d924-469d-a138-2300c1747d0f-kube-api-access-8v5fr podName:3641bd80-d924-469d-a138-2300c1747d0f nodeName:}" failed. No retries permitted until 2026-09-18 17:14:53.463791147 +0000 UTC m=+6.157718656 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8v5fr" (UniqueName: "kubernetes.io/projected/3641bd80-d924-469d-a138-2300c1747d0f-kube-api-access-8v5fr") pod "network-check-target-l9knh" (UID: "3641bd80-d924-469d-a138-2300c1747d0f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 17:14:51.686367 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:51.686203    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 17:09:50 +0000 UTC" deadline="2028-05-11 06:36:45.591037903 +0000 UTC"
Sep 18 17:14:51.686367 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:51.686261    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14413h21m53.904781327s"
Sep 18 17:14:51.760100 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:51.760069    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9knh"
Sep 18 17:14:51.760288 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:51.760213    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9knh" podUID="3641bd80-d924-469d-a138-2300c1747d0f"
Sep 18 17:14:51.760288 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:51.760265    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:14:51.760446 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:51.760400    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kwfm" podUID="491a6ab9-f579-4c60-a0ea-6b6aaad27d52"
Sep 18 17:14:51.777971 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:51.777920    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ps2t8" event={"ID":"b0a90b83-ded0-4f9d-9a60-c38f24a22144","Type":"ContainerStarted","Data":"6072027d158f298cc73773491a98dcb5a65ae51296d1735db05d3f1c435afc9b"}
Sep 18 17:14:51.781354 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:51.781297    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z6nxd" event={"ID":"2af39eb4-05e3-4eca-87d1-87558b5ae1c1","Type":"ContainerStarted","Data":"f0fb90141ed2dfa4da7b83518a1b7518a78d94955174a53b5e5b77c34302065b"}
Sep 18 17:14:51.786259 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:51.786231    2569 generic.go:356] "Generic (PLEG): container finished" podID="e251745cdd5e519678a7024bcabf1fdb" containerID="f537bc28f79e7e51c2d0b41a531aaff8587aa541cc341a37d4be8c9b4d7a88e7" exitCode=0
Sep 18 17:14:51.786569 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:51.786544    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-122.ec2.internal" event={"ID":"e251745cdd5e519678a7024bcabf1fdb","Type":"ContainerDied","Data":"f537bc28f79e7e51c2d0b41a531aaff8587aa541cc341a37d4be8c9b4d7a88e7"}
Sep 18 17:14:51.796982 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:51.796953    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-122.ec2.internal" event={"ID":"96da070bb6fafb62858754f170b042c5","Type":"ContainerStarted","Data":"088ebc8a15c97f1c0c8c9bca1d989063b441a12f06841e4ba23be499e407058d"}
Sep 18 17:14:51.801485 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:51.801408    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xzz8k" event={"ID":"7a650e11-ff04-417e-bcb5-37123daa51b7","Type":"ContainerStarted","Data":"0ea9eb9279cec39ad17dd8bd9d75b0d6da6a1590d84013b846b0bde185d54c88"}
Sep 18 17:14:51.805648 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:51.805574    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5zx6r" event={"ID":"d9b8ea44-62cf-4b12-8a74-f30313e199b4","Type":"ContainerStarted","Data":"8aec6d56cac75c1f71fd97089aa6b58cbca31c3f7e647ac353cfedfe34a9d7a7"}
Sep 18 17:14:51.816630 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:51.816550    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-blmjk" event={"ID":"7e343c56-b6c8-42a6-bf92-dd46fe5d247b","Type":"ContainerStarted","Data":"72d241888aa417c5c805c87333869fbcaa67110a5498f6d869259cc78c329ac4"}
Sep 18 17:14:51.819097 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:51.819070    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxh7t" event={"ID":"a2bfa955-c3ee-4d5f-96fd-95f4038f00ee","Type":"ContainerStarted","Data":"1ce024955ac1c1b9642d6091237fd4d6043bb2f39b07ceea66f0a0e68641177e"}
Sep 18 17:14:51.827235 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:51.827145    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54gxw" event={"ID":"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445","Type":"ContainerStarted","Data":"63841ac6dd49cd023d50bea24792789039c51c0de3183492a0375fe47e04f152"}
Sep 18 17:14:51.838270 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:51.838241    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zjhnk" event={"ID":"fa3b7283-8f5d-4d10-a7ab-6bd10036542f","Type":"ContainerStarted","Data":"f669baad54dcc3f9b97ffa4cd17c88c550ede1f07037fd2c3e52e605c08163ad"}
Sep 18 17:14:51.853280 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:51.853247    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5j9tc" event={"ID":"b9c52349-a5c2-4657-834d-e5e51ea9bef5","Type":"ContainerStarted","Data":"a29b893df34bfb09283aed60c9a4306130cab156f4db14256a9bd6337d498854"}
Sep 18 17:14:52.759761 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:52.759232    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltwd4"
Sep 18 17:14:52.759761 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:52.759392    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltwd4" podUID="91a9651b-affa-44e8-8996-1ea02fa0c2c4"
Sep 18 17:14:52.879446 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:52.879409    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-122.ec2.internal" event={"ID":"e251745cdd5e519678a7024bcabf1fdb","Type":"ContainerStarted","Data":"079ea3864f11f6f605237c18b88fe0b403c01c1555751acf1457897b270005f8"}
Sep 18 17:14:52.897818 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:52.897759    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-122.ec2.internal" podStartSLOduration=2.89773822 podStartE2EDuration="2.89773822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:14:51.820549077 +0000 UTC m=+4.514476630" watchObservedRunningTime="2026-09-18 17:14:52.89773822 +0000 UTC m=+5.591665743"
Sep 18 17:14:53.379191 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:53.379146    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91a9651b-affa-44e8-8996-1ea02fa0c2c4-metrics-certs\") pod \"network-metrics-daemon-ltwd4\" (UID: \"91a9651b-affa-44e8-8996-1ea02fa0c2c4\") " pod="openshift-multus/network-metrics-daemon-ltwd4"
Sep 18 17:14:53.379377 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:53.379210    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/491a6ab9-f579-4c60-a0ea-6b6aaad27d52-original-pull-secret\") pod \"global-pull-secret-syncer-7kwfm\" (UID: \"491a6ab9-f579-4c60-a0ea-6b6aaad27d52\") " pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:14:53.379377 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:53.379360    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 17:14:53.379494 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:53.379425    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/491a6ab9-f579-4c60-a0ea-6b6aaad27d52-original-pull-secret podName:491a6ab9-f579-4c60-a0ea-6b6aaad27d52 nodeName:}" failed. No retries permitted until 2026-09-18 17:14:57.37940627 +0000 UTC m=+10.073333784 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/491a6ab9-f579-4c60-a0ea-6b6aaad27d52-original-pull-secret") pod "global-pull-secret-syncer-7kwfm" (UID: "491a6ab9-f579-4c60-a0ea-6b6aaad27d52") : object "kube-system"/"original-pull-secret" not registered
Sep 18 17:14:53.380117 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:53.379956    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 17:14:53.380117 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:53.380019    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91a9651b-affa-44e8-8996-1ea02fa0c2c4-metrics-certs podName:91a9651b-affa-44e8-8996-1ea02fa0c2c4 nodeName:}" failed. No retries permitted until 2026-09-18 17:14:57.380002402 +0000 UTC m=+10.073929916 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91a9651b-affa-44e8-8996-1ea02fa0c2c4-metrics-certs") pod "network-metrics-daemon-ltwd4" (UID: "91a9651b-affa-44e8-8996-1ea02fa0c2c4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 17:14:53.480558 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:53.479850    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v5fr\" (UniqueName: \"kubernetes.io/projected/3641bd80-d924-469d-a138-2300c1747d0f-kube-api-access-8v5fr\") pod \"network-check-target-l9knh\" (UID: \"3641bd80-d924-469d-a138-2300c1747d0f\") " pod="openshift-network-diagnostics/network-check-target-l9knh"
Sep 18 17:14:53.480558 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:53.480090    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 17:14:53.480558 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:53.480115    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 17:14:53.480558 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:53.480129    2569 projected.go:196] Error preparing data for projected volume kube-api-access-8v5fr for pod openshift-network-diagnostics/network-check-target-l9knh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 17:14:53.480558 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:53.480193    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3641bd80-d924-469d-a138-2300c1747d0f-kube-api-access-8v5fr podName:3641bd80-d924-469d-a138-2300c1747d0f nodeName:}" failed. No retries permitted until 2026-09-18 17:14:57.480172819 +0000 UTC m=+10.174100353 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8v5fr" (UniqueName: "kubernetes.io/projected/3641bd80-d924-469d-a138-2300c1747d0f-kube-api-access-8v5fr") pod "network-check-target-l9knh" (UID: "3641bd80-d924-469d-a138-2300c1747d0f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 17:14:53.760749 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:53.760224    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:14:53.760749 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:53.760242    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9knh"
Sep 18 17:14:53.760749 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:53.760351    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9knh" podUID="3641bd80-d924-469d-a138-2300c1747d0f"
Sep 18 17:14:53.760749 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:53.760423    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kwfm" podUID="491a6ab9-f579-4c60-a0ea-6b6aaad27d52"
Sep 18 17:14:54.759768 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:54.759221    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltwd4"
Sep 18 17:14:54.759768 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:54.759383    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltwd4" podUID="91a9651b-affa-44e8-8996-1ea02fa0c2c4"
Sep 18 17:14:55.759541 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:55.759503    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:14:55.760067 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:55.759656    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kwfm" podUID="491a6ab9-f579-4c60-a0ea-6b6aaad27d52"
Sep 18 17:14:55.760067 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:55.759501    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9knh"
Sep 18 17:14:55.760067 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:55.759908    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9knh" podUID="3641bd80-d924-469d-a138-2300c1747d0f"
Sep 18 17:14:56.759069 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:56.758981    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltwd4"
Sep 18 17:14:56.759233 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:56.759144    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltwd4" podUID="91a9651b-affa-44e8-8996-1ea02fa0c2c4"
Sep 18 17:14:57.416792 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:57.416746    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91a9651b-affa-44e8-8996-1ea02fa0c2c4-metrics-certs\") pod \"network-metrics-daemon-ltwd4\" (UID: \"91a9651b-affa-44e8-8996-1ea02fa0c2c4\") " pod="openshift-multus/network-metrics-daemon-ltwd4"
Sep 18 17:14:57.417314 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:57.416800    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/491a6ab9-f579-4c60-a0ea-6b6aaad27d52-original-pull-secret\") pod \"global-pull-secret-syncer-7kwfm\" (UID: \"491a6ab9-f579-4c60-a0ea-6b6aaad27d52\") " pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:14:57.417314 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:57.416963    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 17:14:57.417314 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:57.417034    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/491a6ab9-f579-4c60-a0ea-6b6aaad27d52-original-pull-secret podName:491a6ab9-f579-4c60-a0ea-6b6aaad27d52 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:05.417013881 +0000 UTC m=+18.110941389 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/491a6ab9-f579-4c60-a0ea-6b6aaad27d52-original-pull-secret") pod "global-pull-secret-syncer-7kwfm" (UID: "491a6ab9-f579-4c60-a0ea-6b6aaad27d52") : object "kube-system"/"original-pull-secret" not registered
Sep 18 17:14:57.417314 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:57.416964    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 17:14:57.417314 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:57.417121    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91a9651b-affa-44e8-8996-1ea02fa0c2c4-metrics-certs podName:91a9651b-affa-44e8-8996-1ea02fa0c2c4 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:05.417099998 +0000 UTC m=+18.111027512 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91a9651b-affa-44e8-8996-1ea02fa0c2c4-metrics-certs") pod "network-metrics-daemon-ltwd4" (UID: "91a9651b-affa-44e8-8996-1ea02fa0c2c4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 17:14:57.518013 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:57.517967    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v5fr\" (UniqueName: \"kubernetes.io/projected/3641bd80-d924-469d-a138-2300c1747d0f-kube-api-access-8v5fr\") pod \"network-check-target-l9knh\" (UID: \"3641bd80-d924-469d-a138-2300c1747d0f\") " pod="openshift-network-diagnostics/network-check-target-l9knh"
Sep 18 17:14:57.518193 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:57.518174    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 17:14:57.518245 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:57.518201    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 17:14:57.518245 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:57.518215    2569 projected.go:196] Error preparing data for projected volume kube-api-access-8v5fr for pod openshift-network-diagnostics/network-check-target-l9knh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 17:14:57.518336 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:57.518283    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3641bd80-d924-469d-a138-2300c1747d0f-kube-api-access-8v5fr podName:3641bd80-d924-469d-a138-2300c1747d0f nodeName:}" failed. No retries permitted until 2026-09-18 17:15:05.518264158 +0000 UTC m=+18.212191676 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8v5fr" (UniqueName: "kubernetes.io/projected/3641bd80-d924-469d-a138-2300c1747d0f-kube-api-access-8v5fr") pod "network-check-target-l9knh" (UID: "3641bd80-d924-469d-a138-2300c1747d0f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 17:14:57.760509 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:57.760428    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9knh"
Sep 18 17:14:57.760686 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:57.760550    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9knh" podUID="3641bd80-d924-469d-a138-2300c1747d0f"
Sep 18 17:14:57.761008 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:57.760983    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:14:57.761114 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:57.761092    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kwfm" podUID="491a6ab9-f579-4c60-a0ea-6b6aaad27d52"
Sep 18 17:14:58.759503 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:58.759465    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltwd4"
Sep 18 17:14:58.759987 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:58.759644    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltwd4" podUID="91a9651b-affa-44e8-8996-1ea02fa0c2c4"
Sep 18 17:14:59.759243 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:59.759147    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:14:59.759425 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:14:59.759148    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9knh"
Sep 18 17:14:59.759425 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:59.759322    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kwfm" podUID="491a6ab9-f579-4c60-a0ea-6b6aaad27d52"
Sep 18 17:14:59.759820 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:14:59.759771    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9knh" podUID="3641bd80-d924-469d-a138-2300c1747d0f"
Sep 18 17:15:00.758900 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:00.758871    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltwd4"
Sep 18 17:15:00.759068 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:00.759002    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltwd4" podUID="91a9651b-affa-44e8-8996-1ea02fa0c2c4"
Sep 18 17:15:01.758630 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:01.758596    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9knh"
Sep 18 17:15:01.759099 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:01.758597    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:15:01.759099 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:01.758714    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9knh" podUID="3641bd80-d924-469d-a138-2300c1747d0f"
Sep 18 17:15:01.759099 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:01.758816    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kwfm" podUID="491a6ab9-f579-4c60-a0ea-6b6aaad27d52"
Sep 18 17:15:02.758968 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:02.758940    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltwd4"
Sep 18 17:15:02.759373 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:02.759076    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltwd4" podUID="91a9651b-affa-44e8-8996-1ea02fa0c2c4"
Sep 18 17:15:03.758713 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:03.758642    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9knh"
Sep 18 17:15:03.758911 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:03.758785    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9knh" podUID="3641bd80-d924-469d-a138-2300c1747d0f"
Sep 18 17:15:03.759127 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:03.759101    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:15:03.759471 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:03.759223    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kwfm" podUID="491a6ab9-f579-4c60-a0ea-6b6aaad27d52"
Sep 18 17:15:04.758570 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:04.758526    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltwd4"
Sep 18 17:15:04.758756 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:04.758641    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltwd4" podUID="91a9651b-affa-44e8-8996-1ea02fa0c2c4"
Sep 18 17:15:05.475594 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:05.475413    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91a9651b-affa-44e8-8996-1ea02fa0c2c4-metrics-certs\") pod \"network-metrics-daemon-ltwd4\" (UID: \"91a9651b-affa-44e8-8996-1ea02fa0c2c4\") " pod="openshift-multus/network-metrics-daemon-ltwd4"
Sep 18 17:15:05.476014 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:05.475607    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/491a6ab9-f579-4c60-a0ea-6b6aaad27d52-original-pull-secret\") pod \"global-pull-secret-syncer-7kwfm\" (UID: \"491a6ab9-f579-4c60-a0ea-6b6aaad27d52\") " pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:15:05.476014 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:05.475550    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 17:15:05.476014 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:05.475703    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91a9651b-affa-44e8-8996-1ea02fa0c2c4-metrics-certs podName:91a9651b-affa-44e8-8996-1ea02fa0c2c4 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:21.475687364 +0000 UTC m=+34.169614875 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91a9651b-affa-44e8-8996-1ea02fa0c2c4-metrics-certs") pod "network-metrics-daemon-ltwd4" (UID: "91a9651b-affa-44e8-8996-1ea02fa0c2c4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 17:15:05.476014 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:05.475748    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 17:15:05.476014 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:05.475804    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/491a6ab9-f579-4c60-a0ea-6b6aaad27d52-original-pull-secret podName:491a6ab9-f579-4c60-a0ea-6b6aaad27d52 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:21.475788395 +0000 UTC m=+34.169715915 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/491a6ab9-f579-4c60-a0ea-6b6aaad27d52-original-pull-secret") pod "global-pull-secret-syncer-7kwfm" (UID: "491a6ab9-f579-4c60-a0ea-6b6aaad27d52") : object "kube-system"/"original-pull-secret" not registered
Sep 18 17:15:05.576702 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:05.576670    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v5fr\" (UniqueName: \"kubernetes.io/projected/3641bd80-d924-469d-a138-2300c1747d0f-kube-api-access-8v5fr\") pod \"network-check-target-l9knh\" (UID: \"3641bd80-d924-469d-a138-2300c1747d0f\") " pod="openshift-network-diagnostics/network-check-target-l9knh"
Sep 18 17:15:05.576872 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:05.576845    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 17:15:05.576920 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:05.576879    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 17:15:05.576920 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:05.576890    2569 projected.go:196] Error preparing data for projected volume kube-api-access-8v5fr for pod openshift-network-diagnostics/network-check-target-l9knh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 17:15:05.576996 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:05.576942    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3641bd80-d924-469d-a138-2300c1747d0f-kube-api-access-8v5fr podName:3641bd80-d924-469d-a138-2300c1747d0f nodeName:}" failed. No retries permitted until 2026-09-18 17:15:21.576927695 +0000 UTC m=+34.270855200 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8v5fr" (UniqueName: "kubernetes.io/projected/3641bd80-d924-469d-a138-2300c1747d0f-kube-api-access-8v5fr") pod "network-check-target-l9knh" (UID: "3641bd80-d924-469d-a138-2300c1747d0f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 17:15:05.758922 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:05.758836    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9knh"
Sep 18 17:15:05.759071 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:05.758835    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:15:05.759071 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:05.758988    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9knh" podUID="3641bd80-d924-469d-a138-2300c1747d0f"
Sep 18 17:15:05.759071 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:05.759036    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kwfm" podUID="491a6ab9-f579-4c60-a0ea-6b6aaad27d52"
Sep 18 17:15:06.758544 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:06.758511    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltwd4"
Sep 18 17:15:06.758942 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:06.758622    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltwd4" podUID="91a9651b-affa-44e8-8996-1ea02fa0c2c4"
Sep 18 17:15:07.759880 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:07.759840    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9knh"
Sep 18 17:15:07.760318 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:07.759957    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9knh" podUID="3641bd80-d924-469d-a138-2300c1747d0f"
Sep 18 17:15:07.760318 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:07.760022    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:15:07.760318 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:07.760133    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kwfm" podUID="491a6ab9-f579-4c60-a0ea-6b6aaad27d52"
Sep 18 17:15:08.759030 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:08.758990    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltwd4"
Sep 18 17:15:08.759216 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:08.759120    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltwd4" podUID="91a9651b-affa-44e8-8996-1ea02fa0c2c4"
Sep 18 17:15:09.758781 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:09.758614    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9knh"
Sep 18 17:15:09.759304 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:09.758670    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:15:09.759304 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:09.758854    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9knh" podUID="3641bd80-d924-469d-a138-2300c1747d0f"
Sep 18 17:15:09.759304 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:09.758957    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kwfm" podUID="491a6ab9-f579-4c60-a0ea-6b6aaad27d52"
Sep 18 17:15:09.921373 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:09.921336    2569 generic.go:356] "Generic (PLEG): container finished" podID="d9b8ea44-62cf-4b12-8a74-f30313e199b4" containerID="3ceb0108c4a75865c979d7f1f84c22f62f5644a32c28f52b4b43425c4fefb773" exitCode=0
Sep 18 17:15:09.921522 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:09.921410    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5zx6r" event={"ID":"d9b8ea44-62cf-4b12-8a74-f30313e199b4","Type":"ContainerDied","Data":"3ceb0108c4a75865c979d7f1f84c22f62f5644a32c28f52b4b43425c4fefb773"}
Sep 18 17:15:09.923018 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:09.922989    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-blmjk" event={"ID":"7e343c56-b6c8-42a6-bf92-dd46fe5d247b","Type":"ContainerStarted","Data":"2d8e0dadf173828267eb17ecb546b41f6716254fdef8c5de50d0d3e2d7a506e5"}
Sep 18 17:15:09.924570 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:09.924547    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxh7t" event={"ID":"a2bfa955-c3ee-4d5f-96fd-95f4038f00ee","Type":"ContainerStarted","Data":"1e72d3276486e5d017e693bebbfd83a618661f3935cce384a7145d2106fa0f7f"}
Sep 18 17:15:09.928422 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:09.928398    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54gxw" event={"ID":"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445","Type":"ContainerStarted","Data":"9b357a3805c9729b5a33a708a99cdcdcf861847c25b2474fb852b634083a70f9"}
Sep 18 17:15:09.928515 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:09.928473    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54gxw" event={"ID":"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445","Type":"ContainerStarted","Data":"61c0bbc7518d378a1a9c24d909dd80ab058f25884b5bb4a6821754ffab7fdffd"}
Sep 18 17:15:09.928558 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:09.928525    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54gxw" event={"ID":"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445","Type":"ContainerStarted","Data":"b949df55670dc1519efc2086c3d93adfad33157d8f1a2a0941dfc0308fc83ca1"}
Sep 18 17:15:09.928604 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:09.928577    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54gxw" event={"ID":"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445","Type":"ContainerStarted","Data":"bf9622ac5a7c5047da02a476936cc2d458c053ed525254633be5c59cad4cf551"}
Sep 18 17:15:09.928655 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:09.928628    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54gxw" event={"ID":"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445","Type":"ContainerStarted","Data":"2ee5df30b972b5546832b9ea9bcc2203f574650b81388f5ac4cf193633579e45"}
Sep 18 17:15:09.928708 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:09.928680    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54gxw" event={"ID":"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445","Type":"ContainerStarted","Data":"acf880f05e8d8008a3166fecedd21e8f2e9dad36e53f9016c8277d3592e0977e"}
Sep 18 17:15:09.929922 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:09.929895    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zjhnk" event={"ID":"fa3b7283-8f5d-4d10-a7ab-6bd10036542f","Type":"ContainerStarted","Data":"cbbf7cf118991dc5e843411325a3ad6e270b07702c1b391bd26f4f04dac90a4f"}
Sep 18 17:15:09.931370 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:09.931349    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5j9tc" event={"ID":"b9c52349-a5c2-4657-834d-e5e51ea9bef5","Type":"ContainerStarted","Data":"7bc610edd05826877e5d206373255ecb99bc62692217cdca07333778491df9b7"}
Sep 18 17:15:09.932699 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:09.932672    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ps2t8" event={"ID":"b0a90b83-ded0-4f9d-9a60-c38f24a22144","Type":"ContainerStarted","Data":"1d95652a205ddac582915bec6d926e27573359c5b43910bc8af3146404fece0d"}
Sep 18 17:15:09.934119 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:09.934085    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z6nxd" event={"ID":"2af39eb4-05e3-4eca-87d1-87558b5ae1c1","Type":"ContainerStarted","Data":"6cf95f7bd60f5b49fc1a62a979386c026eb722e881bbfdc0209776ade986bde6"}
Sep 18 17:15:09.943885 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:09.943834    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-122.ec2.internal" podStartSLOduration=19.943820731 podStartE2EDuration="19.943820731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:14:52.899124708 +0000 UTC m=+5.593052241" watchObservedRunningTime="2026-09-18 17:15:09.943820731 +0000 UTC m=+22.637748258"
Sep 18 17:15:09.960735 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:09.960654    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ps2t8" podStartSLOduration=3.730473231 podStartE2EDuration="21.960641279s" totalImagesPullingTime="18.230168048s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:14:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:15:09.960563072 +0000 UTC m=+22.654490619" watchObservedRunningTime="2026-09-18 17:15:09.960641279 +0000 UTC m=+22.654568806"
Sep 18 17:15:09.975293 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:09.975252    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5j9tc" podStartSLOduration=3.8073993059999998 podStartE2EDuration="21.975239775s" totalImagesPullingTime="18.167840469s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:14:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:15:09.974917978 +0000 UTC m=+22.668845526" watchObservedRunningTime="2026-09-18 17:15:09.975239775 +0000 UTC m=+22.669167303"
Sep 18 17:15:09.994029 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:09.993988    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zjhnk" podStartSLOduration=3.837581207 podStartE2EDuration="21.993970201s" totalImagesPullingTime="18.156388994s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:14:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:15:09.993656919 +0000 UTC m=+22.687584458" watchObservedRunningTime="2026-09-18 17:15:09.993970201 +0000 UTC m=+22.687897730"
Sep 18 17:15:10.023206 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:10.023160    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-blmjk" podStartSLOduration=7.704668262 podStartE2EDuration="22.023145607s" totalImagesPullingTime="14.318477345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:14:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:15:10.008483949 +0000 UTC m=+22.702411476" watchObservedRunningTime="2026-09-18 17:15:10.023145607 +0000 UTC m=+22.717073135"
Sep 18 17:15:10.023373 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:10.023351    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-z6nxd" podStartSLOduration=3.835675095 podStartE2EDuration="22.023344771s" totalImagesPullingTime="18.187669676s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:14:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:15:10.02326153 +0000 UTC m=+22.717189079" watchObservedRunningTime="2026-09-18 17:15:10.023344771 +0000 UTC m=+22.717272299"
Sep 18 17:15:10.759263 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:10.758723    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltwd4"
Sep 18 17:15:10.759263 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:10.758901    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltwd4" podUID="91a9651b-affa-44e8-8996-1ea02fa0c2c4"
Sep 18 17:15:10.864269 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:10.864235    2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 17:15:10.938835 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:10.938799    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xzz8k" event={"ID":"7a650e11-ff04-417e-bcb5-37123daa51b7","Type":"ContainerStarted","Data":"a7ef15ef60117e6a7090ffc51a11df6492f97df89d41d8e2f22285a511741679"}
Sep 18 17:15:10.940959 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:10.940930    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxh7t" event={"ID":"a2bfa955-c3ee-4d5f-96fd-95f4038f00ee","Type":"ContainerStarted","Data":"ae5c8e54c8b94380cfea20be103d224e1e18360e40df3e176170de3d3a1fe3d2"}
Sep 18 17:15:10.955648 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:10.955607    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xzz8k" podStartSLOduration=4.802828645 podStartE2EDuration="22.9555931s" totalImagesPullingTime="18.152764455s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:14:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:15:10.954917223 +0000 UTC m=+23.648844768" watchObservedRunningTime="2026-09-18 17:15:10.9555931 +0000 UTC m=+23.649520627"
Sep 18 17:15:11.691916 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:11.691804    2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T17:15:10.864260244Z","UUID":"84b23589-d4f9-4e95-900d-57501418613f","Handler":null,"Name":"","Endpoint":""}
Sep 18 17:15:11.693203 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:11.693183    2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 17:15:11.693315 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:11.693212    2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 17:15:11.739432 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:11.739119    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-blmjk"
Sep 18 17:15:11.739783 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:11.739762    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-blmjk"
Sep 18 17:15:11.758970 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:11.758944    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9knh"
Sep 18 17:15:11.759085 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:11.758944    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:15:11.759085 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:11.759061    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9knh" podUID="3641bd80-d924-469d-a138-2300c1747d0f"
Sep 18 17:15:11.759177 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:11.759153    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kwfm" podUID="491a6ab9-f579-4c60-a0ea-6b6aaad27d52"
Sep 18 17:15:11.944893 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:11.944814    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxh7t" event={"ID":"a2bfa955-c3ee-4d5f-96fd-95f4038f00ee","Type":"ContainerStarted","Data":"332e0fcd62818c67ba52281cd9b9fe7acf0d5d8ec2519b36d6007677db6eb5e4"}
Sep 18 17:15:11.945233 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:11.945212    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-blmjk"
Sep 18 17:15:11.945717 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:11.945699    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-blmjk"
Sep 18 17:15:11.967415 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:11.967358    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pxh7t" podStartSLOduration=3.651130416 podStartE2EDuration="23.967333795s" totalImagesPullingTime="20.316203379s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:14:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:15:11.965749541 +0000 UTC m=+24.659677091" watchObservedRunningTime="2026-09-18 17:15:11.967333795 +0000 UTC m=+24.661261323"
Sep 18 17:15:12.758643 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:12.758612    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltwd4"
Sep 18 17:15:12.758826 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:12.758720    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltwd4" podUID="91a9651b-affa-44e8-8996-1ea02fa0c2c4"
Sep 18 17:15:12.951269 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:12.951232    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54gxw" event={"ID":"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445","Type":"ContainerStarted","Data":"90e87706aba945f1dd31f3b9ebdd9c16ad67d2b1e45c447c7211a4a12676da60"}
Sep 18 17:15:13.759074 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:13.759039    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9knh"
Sep 18 17:15:13.759278 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:13.759161    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9knh" podUID="3641bd80-d924-469d-a138-2300c1747d0f"
Sep 18 17:15:13.759278 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:13.759225    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:15:13.759393 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:13.759323    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kwfm" podUID="491a6ab9-f579-4c60-a0ea-6b6aaad27d52"
Sep 18 17:15:14.758936 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:14.758777    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltwd4"
Sep 18 17:15:14.759448 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:14.758932    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltwd4" podUID="91a9651b-affa-44e8-8996-1ea02fa0c2c4"
Sep 18 17:15:14.957734 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:14.957405    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5zx6r" event={"ID":"d9b8ea44-62cf-4b12-8a74-f30313e199b4","Type":"ContainerStarted","Data":"160f4c653337573c76b53d6507fa5a5578a5b02d14edd60b024eac63114b50de"}
Sep 18 17:15:14.961425 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:14.961395    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54gxw" event={"ID":"7da7bff1-ba0a-4078-9ef3-f3fb7f2d7445","Type":"ContainerStarted","Data":"b4c54d06bd64740c18fd3a1587ccced9533388ddb945a42326c42768c68dcd81"}
Sep 18 17:15:14.961678 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:14.961656    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:15:14.961779 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:14.961719    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:15:14.975924 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:14.975901    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:15:15.014677 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:15.014626    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-54gxw" podStartSLOduration=8.796757979 podStartE2EDuration="27.014610067s" totalImagesPullingTime="18.217852088s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:14:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:15:15.014214582 +0000 UTC m=+27.708142108" watchObservedRunningTime="2026-09-18 17:15:15.014610067 +0000 UTC m=+27.708537595"
Sep 18 17:15:15.758452 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:15.758421    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:15:15.758647 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:15.758542    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kwfm" podUID="491a6ab9-f579-4c60-a0ea-6b6aaad27d52"
Sep 18 17:15:15.758647 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:15.758607    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9knh"
Sep 18 17:15:15.758765 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:15.758717    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9knh" podUID="3641bd80-d924-469d-a138-2300c1747d0f"
Sep 18 17:15:15.966993 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:15.966955    2569 generic.go:356] "Generic (PLEG): container finished" podID="d9b8ea44-62cf-4b12-8a74-f30313e199b4" containerID="160f4c653337573c76b53d6507fa5a5578a5b02d14edd60b024eac63114b50de" exitCode=0
Sep 18 17:15:15.967695 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:15.967670    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5zx6r" event={"ID":"d9b8ea44-62cf-4b12-8a74-f30313e199b4","Type":"ContainerDied","Data":"160f4c653337573c76b53d6507fa5a5578a5b02d14edd60b024eac63114b50de"}
Sep 18 17:15:15.968367 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:15.968335    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:15:15.990920 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:15.989745    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:15:16.598602 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:16.598564    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7kwfm"]
Sep 18 17:15:16.598750 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:16.598680    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:15:16.598818 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:16.598791    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kwfm" podUID="491a6ab9-f579-4c60-a0ea-6b6aaad27d52"
Sep 18 17:15:16.602539 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:16.602398    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ltwd4"]
Sep 18 17:15:16.602650 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:16.602550    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltwd4"
Sep 18 17:15:16.602716 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:16.602666    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltwd4" podUID="91a9651b-affa-44e8-8996-1ea02fa0c2c4"
Sep 18 17:15:16.603093 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:16.603070    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l9knh"]
Sep 18 17:15:16.603206 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:16.603153    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9knh"
Sep 18 17:15:16.603285 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:16.603255    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9knh" podUID="3641bd80-d924-469d-a138-2300c1747d0f"
Sep 18 17:15:16.971274 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:16.971160    2569 generic.go:356] "Generic (PLEG): container finished" podID="d9b8ea44-62cf-4b12-8a74-f30313e199b4" containerID="5206655d4482e3f0f0ea127edd98edd498b69880c1c0d0299f6e52fb95870fa9" exitCode=0
Sep 18 17:15:16.971274 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:16.971224    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5zx6r" event={"ID":"d9b8ea44-62cf-4b12-8a74-f30313e199b4","Type":"ContainerDied","Data":"5206655d4482e3f0f0ea127edd98edd498b69880c1c0d0299f6e52fb95870fa9"}
Sep 18 17:15:18.758889 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:18.758791    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltwd4"
Sep 18 17:15:18.758889 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:18.758789    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9knh"
Sep 18 17:15:18.759328 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:18.758919    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:15:18.759328 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:18.758928    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltwd4" podUID="91a9651b-affa-44e8-8996-1ea02fa0c2c4"
Sep 18 17:15:18.759328 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:18.758997    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kwfm" podUID="491a6ab9-f579-4c60-a0ea-6b6aaad27d52"
Sep 18 17:15:18.759328 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:18.759054    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9knh" podUID="3641bd80-d924-469d-a138-2300c1747d0f"
Sep 18 17:15:18.977533 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:18.977500    2569 generic.go:356] "Generic (PLEG): container finished" podID="d9b8ea44-62cf-4b12-8a74-f30313e199b4" containerID="3f304d32747f454e5a7ef7ec15da87345e6200b9e23a63b1a15a37f665136dd9" exitCode=0
Sep 18 17:15:18.977673 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:18.977561    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5zx6r" event={"ID":"d9b8ea44-62cf-4b12-8a74-f30313e199b4","Type":"ContainerDied","Data":"3f304d32747f454e5a7ef7ec15da87345e6200b9e23a63b1a15a37f665136dd9"}
Sep 18 17:15:20.759153 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:20.759119    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:15:20.759589 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:20.759169    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltwd4"
Sep 18 17:15:20.759589 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:20.759230    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9knh"
Sep 18 17:15:20.759589 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:20.759249    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kwfm" podUID="491a6ab9-f579-4c60-a0ea-6b6aaad27d52"
Sep 18 17:15:20.759589 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:20.759319    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9knh" podUID="3641bd80-d924-469d-a138-2300c1747d0f"
Sep 18 17:15:20.759589 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:20.759396    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltwd4" podUID="91a9651b-affa-44e8-8996-1ea02fa0c2c4"
Sep 18 17:15:21.494647 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:21.494612    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91a9651b-affa-44e8-8996-1ea02fa0c2c4-metrics-certs\") pod \"network-metrics-daemon-ltwd4\" (UID: \"91a9651b-affa-44e8-8996-1ea02fa0c2c4\") " pod="openshift-multus/network-metrics-daemon-ltwd4"
Sep 18 17:15:21.494965 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:21.494655    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/491a6ab9-f579-4c60-a0ea-6b6aaad27d52-original-pull-secret\") pod \"global-pull-secret-syncer-7kwfm\" (UID: \"491a6ab9-f579-4c60-a0ea-6b6aaad27d52\") " pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:15:21.494965 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:21.494778    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 17:15:21.494965 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:21.494777    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 17:15:21.494965 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:21.494856    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/491a6ab9-f579-4c60-a0ea-6b6aaad27d52-original-pull-secret podName:491a6ab9-f579-4c60-a0ea-6b6aaad27d52 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:53.494837706 +0000 UTC m=+66.188765231 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/491a6ab9-f579-4c60-a0ea-6b6aaad27d52-original-pull-secret") pod "global-pull-secret-syncer-7kwfm" (UID: "491a6ab9-f579-4c60-a0ea-6b6aaad27d52") : object "kube-system"/"original-pull-secret" not registered
Sep 18 17:15:21.494965 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:21.494891    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91a9651b-affa-44e8-8996-1ea02fa0c2c4-metrics-certs podName:91a9651b-affa-44e8-8996-1ea02fa0c2c4 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:53.494879501 +0000 UTC m=+66.188807007 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91a9651b-affa-44e8-8996-1ea02fa0c2c4-metrics-certs") pod "network-metrics-daemon-ltwd4" (UID: "91a9651b-affa-44e8-8996-1ea02fa0c2c4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 17:15:21.595539 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:21.595503    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v5fr\" (UniqueName: \"kubernetes.io/projected/3641bd80-d924-469d-a138-2300c1747d0f-kube-api-access-8v5fr\") pod \"network-check-target-l9knh\" (UID: \"3641bd80-d924-469d-a138-2300c1747d0f\") " pod="openshift-network-diagnostics/network-check-target-l9knh"
Sep 18 17:15:21.595723 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:21.595677    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 17:15:21.595723 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:21.595703    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 17:15:21.595723 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:21.595715    2569 projected.go:196] Error preparing data for projected volume kube-api-access-8v5fr for pod openshift-network-diagnostics/network-check-target-l9knh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 17:15:21.595910 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:21.595777    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3641bd80-d924-469d-a138-2300c1747d0f-kube-api-access-8v5fr podName:3641bd80-d924-469d-a138-2300c1747d0f nodeName:}" failed. No retries permitted until 2026-09-18 17:15:53.595758494 +0000 UTC m=+66.289686014 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8v5fr" (UniqueName: "kubernetes.io/projected/3641bd80-d924-469d-a138-2300c1747d0f-kube-api-access-8v5fr") pod "network-check-target-l9knh" (UID: "3641bd80-d924-469d-a138-2300c1747d0f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 17:15:22.759229 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:22.759198    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9knh"
Sep 18 17:15:22.759647 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:22.759238    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:15:22.759647 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:22.759303    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltwd4"
Sep 18 17:15:22.759647 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:22.759320    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9knh" podUID="3641bd80-d924-469d-a138-2300c1747d0f"
Sep 18 17:15:22.759647 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:22.759408    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kwfm" podUID="491a6ab9-f579-4c60-a0ea-6b6aaad27d52"
Sep 18 17:15:22.759647 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:22.759505    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltwd4" podUID="91a9651b-affa-44e8-8996-1ea02fa0c2c4"
Sep 18 17:15:23.076316 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:23.076249    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-122.ec2.internal" event="NodeReady"
Sep 18 17:15:23.076470 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:23.076400    2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 17:15:23.120415 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:23.120382    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-s566m"]
Sep 18 17:15:23.150128 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:23.150083    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nwncv"]
Sep 18 17:15:23.150284 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:23.150214    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-s566m"
Sep 18 17:15:23.152818 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:23.152768    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 17:15:23.152991 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:23.152956    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-99knh\""
Sep 18 17:15:23.152991 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:23.152968    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 17:15:23.165997 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:23.165975    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-s566m"]
Sep 18 17:15:23.165997 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:23.166017    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nwncv"]
Sep 18 17:15:23.166220 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:23.166132    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nwncv"
Sep 18 17:15:23.168768 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:23.168743    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ddjzh\""
Sep 18 17:15:23.168891 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:23.168781    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 17:15:23.168891 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:23.168820    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 17:15:23.169239 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:23.169217    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 17:15:23.309772 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:23.309569    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8bcf\" (UniqueName: \"kubernetes.io/projected/80573a4b-9369-42f3-94b4-c0673f122e62-kube-api-access-m8bcf\") pod \"dns-default-s566m\" (UID: \"80573a4b-9369-42f3-94b4-c0673f122e62\") " pod="openshift-dns/dns-default-s566m"
Sep 18 17:15:23.309966 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:23.309784    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/80573a4b-9369-42f3-94b4-c0673f122e62-metrics-tls\") pod \"dns-default-s566m\" (UID: \"80573a4b-9369-42f3-94b4-c0673f122e62\") " pod="openshift-dns/dns-default-s566m"
Sep 18 17:15:23.309966 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:23.309841    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/80573a4b-9369-42f3-94b4-c0673f122e62-config-volume\") pod \"dns-default-s566m\" (UID: \"80573a4b-9369-42f3-94b4-c0673f122e62\") " pod="openshift-dns/dns-default-s566m"
Sep 18 17:15:23.309966 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:23.309937    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6xgb\" (UniqueName: \"kubernetes.io/projected/1549ca4a-cf93-4295-8f95-c869e2b6a1c6-kube-api-access-x6xgb\") pod \"ingress-canary-nwncv\" (UID: \"1549ca4a-cf93-4295-8f95-c869e2b6a1c6\") " pod="openshift-ingress-canary/ingress-canary-nwncv"
Sep 18 17:15:23.310131 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:23.309969    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80573a4b-9369-42f3-94b4-c0673f122e62-tmp-dir\") pod \"dns-default-s566m\" (UID: \"80573a4b-9369-42f3-94b4-c0673f122e62\") " pod="openshift-dns/dns-default-s566m"
Sep 18 17:15:23.310131 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:23.309997    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1549ca4a-cf93-4295-8f95-c869e2b6a1c6-cert\") pod \"ingress-canary-nwncv\" (UID: \"1549ca4a-cf93-4295-8f95-c869e2b6a1c6\") " pod="openshift-ingress-canary/ingress-canary-nwncv"
Sep 18 17:15:23.411077 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:23.410991    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8bcf\" (UniqueName: \"kubernetes.io/projected/80573a4b-9369-42f3-94b4-c0673f122e62-kube-api-access-m8bcf\") pod \"dns-default-s566m\" (UID: \"80573a4b-9369-42f3-94b4-c0673f122e62\") " pod="openshift-dns/dns-default-s566m"
Sep 18 17:15:23.411077 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:23.411041    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/80573a4b-9369-42f3-94b4-c0673f122e62-metrics-tls\") pod \"dns-default-s566m\" (UID: \"80573a4b-9369-42f3-94b4-c0673f122e62\") " pod="openshift-dns/dns-default-s566m"
Sep 18 17:15:23.411297 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:23.411241    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 17:15:23.411297 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:23.411288    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80573a4b-9369-42f3-94b4-c0673f122e62-metrics-tls podName:80573a4b-9369-42f3-94b4-c0673f122e62 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:23.911274892 +0000 UTC m=+36.605202398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/80573a4b-9369-42f3-94b4-c0673f122e62-metrics-tls") pod "dns-default-s566m" (UID: "80573a4b-9369-42f3-94b4-c0673f122e62") : secret "dns-default-metrics-tls" not found
Sep 18 17:15:23.411297 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:23.411236    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/80573a4b-9369-42f3-94b4-c0673f122e62-config-volume\") pod \"dns-default-s566m\" (UID: \"80573a4b-9369-42f3-94b4-c0673f122e62\") " pod="openshift-dns/dns-default-s566m"
Sep 18 17:15:23.411445 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:23.411333    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6xgb\" (UniqueName: \"kubernetes.io/projected/1549ca4a-cf93-4295-8f95-c869e2b6a1c6-kube-api-access-x6xgb\") pod \"ingress-canary-nwncv\" (UID: \"1549ca4a-cf93-4295-8f95-c869e2b6a1c6\") " pod="openshift-ingress-canary/ingress-canary-nwncv"
Sep 18 17:15:23.411445 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:23.411356    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80573a4b-9369-42f3-94b4-c0673f122e62-tmp-dir\") pod \"dns-default-s566m\" (UID: \"80573a4b-9369-42f3-94b4-c0673f122e62\") " pod="openshift-dns/dns-default-s566m"
Sep 18 17:15:23.411445 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:23.411390    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1549ca4a-cf93-4295-8f95-c869e2b6a1c6-cert\") pod \"ingress-canary-nwncv\" (UID: \"1549ca4a-cf93-4295-8f95-c869e2b6a1c6\") " pod="openshift-ingress-canary/ingress-canary-nwncv"
Sep 18 17:15:23.411723 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:23.411697    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80573a4b-9369-42f3-94b4-c0673f122e62-tmp-dir\") pod \"dns-default-s566m\" (UID: \"80573a4b-9369-42f3-94b4-c0673f122e62\") " pod="openshift-dns/dns-default-s566m"
Sep 18 17:15:23.411914 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:23.411896    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/80573a4b-9369-42f3-94b4-c0673f122e62-config-volume\") pod \"dns-default-s566m\" (UID: \"80573a4b-9369-42f3-94b4-c0673f122e62\") " pod="openshift-dns/dns-default-s566m"
Sep 18 17:15:23.415858 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:23.415836    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1549ca4a-cf93-4295-8f95-c869e2b6a1c6-cert\") pod \"ingress-canary-nwncv\" (UID: \"1549ca4a-cf93-4295-8f95-c869e2b6a1c6\") " pod="openshift-ingress-canary/ingress-canary-nwncv"
Sep 18 17:15:23.420367 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:23.420343    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8bcf\" (UniqueName: \"kubernetes.io/projected/80573a4b-9369-42f3-94b4-c0673f122e62-kube-api-access-m8bcf\") pod \"dns-default-s566m\" (UID: \"80573a4b-9369-42f3-94b4-c0673f122e62\") " pod="openshift-dns/dns-default-s566m"
Sep 18 17:15:23.420603 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:23.420586    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6xgb\" (UniqueName: \"kubernetes.io/projected/1549ca4a-cf93-4295-8f95-c869e2b6a1c6-kube-api-access-x6xgb\") pod \"ingress-canary-nwncv\" (UID: \"1549ca4a-cf93-4295-8f95-c869e2b6a1c6\") " pod="openshift-ingress-canary/ingress-canary-nwncv"
Sep 18 17:15:23.476431 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:23.476394    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nwncv"
Sep 18 17:15:23.914627 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:23.914587    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/80573a4b-9369-42f3-94b4-c0673f122e62-metrics-tls\") pod \"dns-default-s566m\" (UID: \"80573a4b-9369-42f3-94b4-c0673f122e62\") " pod="openshift-dns/dns-default-s566m"
Sep 18 17:15:23.915187 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:23.914709    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 17:15:23.915187 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:23.914785    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80573a4b-9369-42f3-94b4-c0673f122e62-metrics-tls podName:80573a4b-9369-42f3-94b4-c0673f122e62 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:24.914768395 +0000 UTC m=+37.608695900 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/80573a4b-9369-42f3-94b4-c0673f122e62-metrics-tls") pod "dns-default-s566m" (UID: "80573a4b-9369-42f3-94b4-c0673f122e62") : secret "dns-default-metrics-tls" not found
Sep 18 17:15:24.033545 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:24.033522    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nwncv"]
Sep 18 17:15:24.036789 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:15:24.036760    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1549ca4a_cf93_4295_8f95_c869e2b6a1c6.slice/crio-3be8b3017d66c2a075213e50f7aa57b64b90975a99ba7f042595503e2267a1a3 WatchSource:0}: Error finding container 3be8b3017d66c2a075213e50f7aa57b64b90975a99ba7f042595503e2267a1a3: Status 404 returned error can't find the container with id 3be8b3017d66c2a075213e50f7aa57b64b90975a99ba7f042595503e2267a1a3
Sep 18 17:15:24.759269 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:24.759234    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9knh"
Sep 18 17:15:24.759448 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:24.759281    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:15:24.759944 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:24.759732    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltwd4"
Sep 18 17:15:24.761904 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:24.761878    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 17:15:24.763523 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:24.763066    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-f2lgv\""
Sep 18 17:15:24.763523 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:24.763112    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 17:15:24.763523 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:24.763150    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 17:15:24.763523 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:24.763348    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2h6n2\""
Sep 18 17:15:24.763523 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:24.763402    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 17:15:24.920638 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:24.920603    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/80573a4b-9369-42f3-94b4-c0673f122e62-metrics-tls\") pod \"dns-default-s566m\" (UID: \"80573a4b-9369-42f3-94b4-c0673f122e62\") " pod="openshift-dns/dns-default-s566m"
Sep 18 17:15:24.921336 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:24.920776    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 17:15:24.921336 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:24.920855    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80573a4b-9369-42f3-94b4-c0673f122e62-metrics-tls podName:80573a4b-9369-42f3-94b4-c0673f122e62 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:26.920834007 +0000 UTC m=+39.614761515 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/80573a4b-9369-42f3-94b4-c0673f122e62-metrics-tls") pod "dns-default-s566m" (UID: "80573a4b-9369-42f3-94b4-c0673f122e62") : secret "dns-default-metrics-tls" not found
Sep 18 17:15:24.995888 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:24.995822    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nwncv" event={"ID":"1549ca4a-cf93-4295-8f95-c869e2b6a1c6","Type":"ContainerStarted","Data":"3be8b3017d66c2a075213e50f7aa57b64b90975a99ba7f042595503e2267a1a3"}
Sep 18 17:15:24.999054 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:24.999024    2569 generic.go:356] "Generic (PLEG): container finished" podID="d9b8ea44-62cf-4b12-8a74-f30313e199b4" containerID="bbe4951db5092f8e174c4b1f038ad3350893a290629df676d27b09217c1f4c8b" exitCode=0
Sep 18 17:15:24.999286 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:24.999065    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5zx6r" event={"ID":"d9b8ea44-62cf-4b12-8a74-f30313e199b4","Type":"ContainerDied","Data":"bbe4951db5092f8e174c4b1f038ad3350893a290629df676d27b09217c1f4c8b"}
Sep 18 17:15:26.008216 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:26.007968    2569 generic.go:356] "Generic (PLEG): container finished" podID="d9b8ea44-62cf-4b12-8a74-f30313e199b4" containerID="e0d91b4e284602f9478e7eec6170a6f39b87014f203d51c1d7a3ee74506afa8c" exitCode=0
Sep 18 17:15:26.008622 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:26.008051    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5zx6r" event={"ID":"d9b8ea44-62cf-4b12-8a74-f30313e199b4","Type":"ContainerDied","Data":"e0d91b4e284602f9478e7eec6170a6f39b87014f203d51c1d7a3ee74506afa8c"}
Sep 18 17:15:26.936799 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:26.936708    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/80573a4b-9369-42f3-94b4-c0673f122e62-metrics-tls\") pod \"dns-default-s566m\" (UID: \"80573a4b-9369-42f3-94b4-c0673f122e62\") " pod="openshift-dns/dns-default-s566m"
Sep 18 17:15:26.936960 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:26.936879    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 17:15:26.937004 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:26.936962    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80573a4b-9369-42f3-94b4-c0673f122e62-metrics-tls podName:80573a4b-9369-42f3-94b4-c0673f122e62 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:30.93694573 +0000 UTC m=+43.630873236 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/80573a4b-9369-42f3-94b4-c0673f122e62-metrics-tls") pod "dns-default-s566m" (UID: "80573a4b-9369-42f3-94b4-c0673f122e62") : secret "dns-default-metrics-tls" not found
Sep 18 17:15:27.014136 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:27.014104    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5zx6r" event={"ID":"d9b8ea44-62cf-4b12-8a74-f30313e199b4","Type":"ContainerStarted","Data":"9e8fc2e3946a4e6efba4c33f80d4e377e30cb5413b5022c7dd919627be91602e"}
Sep 18 17:15:27.015626 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:27.015602    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nwncv" event={"ID":"1549ca4a-cf93-4295-8f95-c869e2b6a1c6","Type":"ContainerStarted","Data":"1c7989cfa593a4e4897e184b4f69e950fad7f1b6137691fc1f8cb3376cbc7166"}
Sep 18 17:15:27.037849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:27.037805    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5zx6r" podStartSLOduration=8.732596509 podStartE2EDuration="39.037789508s" totalImagesPullingTime="30.305192999s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:14:48 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:15:27.036347316 +0000 UTC m=+39.730274863" watchObservedRunningTime="2026-09-18 17:15:27.037789508 +0000 UTC m=+39.731717035"
Sep 18 17:15:27.052655 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:27.052616    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nwncv" podStartSLOduration=1.41853666 podStartE2EDuration="4.052605047s" totalImagesPullingTime="2.634068387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:15:27.052068397 +0000 UTC m=+39.745995948" watchObservedRunningTime="2026-09-18 17:15:27.052605047 +0000 UTC m=+39.746532574"
Sep 18 17:15:30.968664 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:30.968622    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/80573a4b-9369-42f3-94b4-c0673f122e62-metrics-tls\") pod \"dns-default-s566m\" (UID: \"80573a4b-9369-42f3-94b4-c0673f122e62\") " pod="openshift-dns/dns-default-s566m"
Sep 18 17:15:30.969151 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:30.968768    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 17:15:30.969151 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:30.968828    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80573a4b-9369-42f3-94b4-c0673f122e62-metrics-tls podName:80573a4b-9369-42f3-94b4-c0673f122e62 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:38.968814132 +0000 UTC m=+51.662741643 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/80573a4b-9369-42f3-94b4-c0673f122e62-metrics-tls") pod "dns-default-s566m" (UID: "80573a4b-9369-42f3-94b4-c0673f122e62") : secret "dns-default-metrics-tls" not found
Sep 18 17:15:34.487605 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.487569    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-76465d565b-r2rdc"]
Sep 18 17:15:34.519047 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.519022    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-76465d565b-r2rdc"]
Sep 18 17:15:34.519188 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.519132    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-76465d565b-r2rdc"
Sep 18 17:15:34.525378 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.525284    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 18 17:15:34.525378 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.525333    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-tmvvs\""
Sep 18 17:15:34.525378 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.525363    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 18 17:15:34.525562 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.525502    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 18 17:15:34.525562 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.525547    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 18 17:15:34.525670 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.525609    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 18 17:15:34.525730 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.525716    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 18 17:15:34.600563 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.600540    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-v4nfn"]
Sep 18 17:15:34.623482 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.623458    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-v4nfn"]
Sep 18 17:15:34.623603 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.623550    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-v4nfn"
Sep 18 17:15:34.626824 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.626797    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 17:15:34.626987 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.626797    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 17:15:34.626987 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.626808    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 18 17:15:34.626987 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.626890    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 18 17:15:34.630816 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.630790    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-zsxkm\""
Sep 18 17:15:34.636216 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.636193    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 18 17:15:34.692910 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.692877    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/63db05d0-62bd-4fe6-999f-0d19ef2ff266-stats-auth\") pod \"router-default-76465d565b-r2rdc\" (UID: \"63db05d0-62bd-4fe6-999f-0d19ef2ff266\") " pod="openshift-ingress/router-default-76465d565b-r2rdc"
Sep 18 17:15:34.693112 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.692918    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63db05d0-62bd-4fe6-999f-0d19ef2ff266-metrics-certs\") pod \"router-default-76465d565b-r2rdc\" (UID: \"63db05d0-62bd-4fe6-999f-0d19ef2ff266\") " pod="openshift-ingress/router-default-76465d565b-r2rdc"
Sep 18 17:15:34.693112 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.693005    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/63db05d0-62bd-4fe6-999f-0d19ef2ff266-default-certificate\") pod \"router-default-76465d565b-r2rdc\" (UID: \"63db05d0-62bd-4fe6-999f-0d19ef2ff266\") " pod="openshift-ingress/router-default-76465d565b-r2rdc"
Sep 18 17:15:34.693112 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.693028    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63db05d0-62bd-4fe6-999f-0d19ef2ff266-service-ca-bundle\") pod \"router-default-76465d565b-r2rdc\" (UID: \"63db05d0-62bd-4fe6-999f-0d19ef2ff266\") " pod="openshift-ingress/router-default-76465d565b-r2rdc"
Sep 18 17:15:34.693112 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.693056    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m78q\" (UniqueName: \"kubernetes.io/projected/63db05d0-62bd-4fe6-999f-0d19ef2ff266-kube-api-access-4m78q\") pod \"router-default-76465d565b-r2rdc\" (UID: \"63db05d0-62bd-4fe6-999f-0d19ef2ff266\") " pod="openshift-ingress/router-default-76465d565b-r2rdc"
Sep 18 17:15:34.793688 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.793589    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdd63e16-6865-47a1-86ca-c1870d2eae7d-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-v4nfn\" (UID: \"cdd63e16-6865-47a1-86ca-c1870d2eae7d\") " pod="openshift-insights/insights-operator-6f9c644dd9-v4nfn"
Sep 18 17:15:34.793688 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.793635    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdd63e16-6865-47a1-86ca-c1870d2eae7d-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-v4nfn\" (UID: \"cdd63e16-6865-47a1-86ca-c1870d2eae7d\") " pod="openshift-insights/insights-operator-6f9c644dd9-v4nfn"
Sep 18 17:15:34.793688 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.793676    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cdd63e16-6865-47a1-86ca-c1870d2eae7d-serving-cert\") pod \"insights-operator-6f9c644dd9-v4nfn\" (UID: \"cdd63e16-6865-47a1-86ca-c1870d2eae7d\") " pod="openshift-insights/insights-operator-6f9c644dd9-v4nfn"
Sep 18 17:15:34.793949 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.793701    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/63db05d0-62bd-4fe6-999f-0d19ef2ff266-stats-auth\") pod \"router-default-76465d565b-r2rdc\" (UID: \"63db05d0-62bd-4fe6-999f-0d19ef2ff266\") " pod="openshift-ingress/router-default-76465d565b-r2rdc"
Sep 18 17:15:34.793949 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.793719    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk957\" (UniqueName: \"kubernetes.io/projected/cdd63e16-6865-47a1-86ca-c1870d2eae7d-kube-api-access-sk957\") pod \"insights-operator-6f9c644dd9-v4nfn\" (UID: \"cdd63e16-6865-47a1-86ca-c1870d2eae7d\") " pod="openshift-insights/insights-operator-6f9c644dd9-v4nfn"
Sep 18 17:15:34.793949 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.793754    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63db05d0-62bd-4fe6-999f-0d19ef2ff266-metrics-certs\") pod \"router-default-76465d565b-r2rdc\" (UID: \"63db05d0-62bd-4fe6-999f-0d19ef2ff266\") " pod="openshift-ingress/router-default-76465d565b-r2rdc"
Sep 18 17:15:34.793949 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.793806    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cdd63e16-6865-47a1-86ca-c1870d2eae7d-tmp\") pod \"insights-operator-6f9c644dd9-v4nfn\" (UID: \"cdd63e16-6865-47a1-86ca-c1870d2eae7d\") " pod="openshift-insights/insights-operator-6f9c644dd9-v4nfn"
Sep 18 17:15:34.793949 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.793828    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cdd63e16-6865-47a1-86ca-c1870d2eae7d-snapshots\") pod \"insights-operator-6f9c644dd9-v4nfn\" (UID: \"cdd63e16-6865-47a1-86ca-c1870d2eae7d\") " pod="openshift-insights/insights-operator-6f9c644dd9-v4nfn"
Sep 18 17:15:34.793949 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:34.793841    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 17:15:34.793949 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:34.793930    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63db05d0-62bd-4fe6-999f-0d19ef2ff266-metrics-certs podName:63db05d0-62bd-4fe6-999f-0d19ef2ff266 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:35.293914215 +0000 UTC m=+47.987841731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63db05d0-62bd-4fe6-999f-0d19ef2ff266-metrics-certs") pod "router-default-76465d565b-r2rdc" (UID: "63db05d0-62bd-4fe6-999f-0d19ef2ff266") : secret "router-metrics-certs-default" not found
Sep 18 17:15:34.794249 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.793957    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/63db05d0-62bd-4fe6-999f-0d19ef2ff266-default-certificate\") pod \"router-default-76465d565b-r2rdc\" (UID: \"63db05d0-62bd-4fe6-999f-0d19ef2ff266\") " pod="openshift-ingress/router-default-76465d565b-r2rdc"
Sep 18 17:15:34.794249 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.793979    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63db05d0-62bd-4fe6-999f-0d19ef2ff266-service-ca-bundle\") pod \"router-default-76465d565b-r2rdc\" (UID: \"63db05d0-62bd-4fe6-999f-0d19ef2ff266\") " pod="openshift-ingress/router-default-76465d565b-r2rdc"
Sep 18 17:15:34.794249 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.794019    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m78q\" (UniqueName: \"kubernetes.io/projected/63db05d0-62bd-4fe6-999f-0d19ef2ff266-kube-api-access-4m78q\") pod \"router-default-76465d565b-r2rdc\" (UID: \"63db05d0-62bd-4fe6-999f-0d19ef2ff266\") " pod="openshift-ingress/router-default-76465d565b-r2rdc"
Sep 18 17:15:34.794249 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:34.794184    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63db05d0-62bd-4fe6-999f-0d19ef2ff266-service-ca-bundle podName:63db05d0-62bd-4fe6-999f-0d19ef2ff266 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:35.29416436 +0000 UTC m=+47.988091882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/63db05d0-62bd-4fe6-999f-0d19ef2ff266-service-ca-bundle") pod "router-default-76465d565b-r2rdc" (UID: "63db05d0-62bd-4fe6-999f-0d19ef2ff266") : configmap references non-existent config key: service-ca.crt
Sep 18 17:15:34.797360 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.797340    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/63db05d0-62bd-4fe6-999f-0d19ef2ff266-stats-auth\") pod \"router-default-76465d565b-r2rdc\" (UID: \"63db05d0-62bd-4fe6-999f-0d19ef2ff266\") " pod="openshift-ingress/router-default-76465d565b-r2rdc"
Sep 18 17:15:34.797413 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.797386    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/63db05d0-62bd-4fe6-999f-0d19ef2ff266-default-certificate\") pod \"router-default-76465d565b-r2rdc\" (UID: \"63db05d0-62bd-4fe6-999f-0d19ef2ff266\") " pod="openshift-ingress/router-default-76465d565b-r2rdc"
Sep 18 17:15:34.803285 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.803264    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m78q\" (UniqueName: \"kubernetes.io/projected/63db05d0-62bd-4fe6-999f-0d19ef2ff266-kube-api-access-4m78q\") pod \"router-default-76465d565b-r2rdc\" (UID: \"63db05d0-62bd-4fe6-999f-0d19ef2ff266\") " pod="openshift-ingress/router-default-76465d565b-r2rdc"
Sep 18 17:15:34.895158 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.895127    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cdd63e16-6865-47a1-86ca-c1870d2eae7d-serving-cert\") pod \"insights-operator-6f9c644dd9-v4nfn\" (UID: \"cdd63e16-6865-47a1-86ca-c1870d2eae7d\") " pod="openshift-insights/insights-operator-6f9c644dd9-v4nfn"
Sep 18 17:15:34.895158 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.895161    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk957\" (UniqueName: \"kubernetes.io/projected/cdd63e16-6865-47a1-86ca-c1870d2eae7d-kube-api-access-sk957\") pod \"insights-operator-6f9c644dd9-v4nfn\" (UID: \"cdd63e16-6865-47a1-86ca-c1870d2eae7d\") " pod="openshift-insights/insights-operator-6f9c644dd9-v4nfn"
Sep 18 17:15:34.895347 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.895214    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cdd63e16-6865-47a1-86ca-c1870d2eae7d-tmp\") pod \"insights-operator-6f9c644dd9-v4nfn\" (UID: \"cdd63e16-6865-47a1-86ca-c1870d2eae7d\") " pod="openshift-insights/insights-operator-6f9c644dd9-v4nfn"
Sep 18 17:15:34.895347 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.895231    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cdd63e16-6865-47a1-86ca-c1870d2eae7d-snapshots\") pod \"insights-operator-6f9c644dd9-v4nfn\" (UID: \"cdd63e16-6865-47a1-86ca-c1870d2eae7d\") " pod="openshift-insights/insights-operator-6f9c644dd9-v4nfn"
Sep 18 17:15:34.895347 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.895282    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdd63e16-6865-47a1-86ca-c1870d2eae7d-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-v4nfn\" (UID: \"cdd63e16-6865-47a1-86ca-c1870d2eae7d\") " pod="openshift-insights/insights-operator-6f9c644dd9-v4nfn"
Sep 18 17:15:34.895480 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.895413    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdd63e16-6865-47a1-86ca-c1870d2eae7d-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-v4nfn\" (UID: \"cdd63e16-6865-47a1-86ca-c1870d2eae7d\") " pod="openshift-insights/insights-operator-6f9c644dd9-v4nfn"
Sep 18 17:15:34.895677 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.895592    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cdd63e16-6865-47a1-86ca-c1870d2eae7d-tmp\") pod \"insights-operator-6f9c644dd9-v4nfn\" (UID: \"cdd63e16-6865-47a1-86ca-c1870d2eae7d\") " pod="openshift-insights/insights-operator-6f9c644dd9-v4nfn"
Sep 18 17:15:34.895937 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.895913    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cdd63e16-6865-47a1-86ca-c1870d2eae7d-snapshots\") pod \"insights-operator-6f9c644dd9-v4nfn\" (UID: \"cdd63e16-6865-47a1-86ca-c1870d2eae7d\") " pod="openshift-insights/insights-operator-6f9c644dd9-v4nfn"
Sep 18 17:15:34.896068 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.896050    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdd63e16-6865-47a1-86ca-c1870d2eae7d-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-v4nfn\" (UID: \"cdd63e16-6865-47a1-86ca-c1870d2eae7d\") " pod="openshift-insights/insights-operator-6f9c644dd9-v4nfn"
Sep 18 17:15:34.896121 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.896084    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdd63e16-6865-47a1-86ca-c1870d2eae7d-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-v4nfn\" (UID: \"cdd63e16-6865-47a1-86ca-c1870d2eae7d\") " pod="openshift-insights/insights-operator-6f9c644dd9-v4nfn"
Sep 18 17:15:34.897405 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.897385    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cdd63e16-6865-47a1-86ca-c1870d2eae7d-serving-cert\") pod \"insights-operator-6f9c644dd9-v4nfn\" (UID: \"cdd63e16-6865-47a1-86ca-c1870d2eae7d\") " pod="openshift-insights/insights-operator-6f9c644dd9-v4nfn"
Sep 18 17:15:34.909670 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.909642    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk957\" (UniqueName: \"kubernetes.io/projected/cdd63e16-6865-47a1-86ca-c1870d2eae7d-kube-api-access-sk957\") pod \"insights-operator-6f9c644dd9-v4nfn\" (UID: \"cdd63e16-6865-47a1-86ca-c1870d2eae7d\") " pod="openshift-insights/insights-operator-6f9c644dd9-v4nfn"
Sep 18 17:15:34.934368 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:34.934341    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-v4nfn"
Sep 18 17:15:35.234712 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:35.234686    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-v4nfn"]
Sep 18 17:15:35.237464 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:15:35.237438    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd63e16_6865_47a1_86ca_c1870d2eae7d.slice/crio-fe79dc0d3506fa5631f89a5cd2d131316b7cd54f11ec1245f66186e011e38a7a WatchSource:0}: Error finding container fe79dc0d3506fa5631f89a5cd2d131316b7cd54f11ec1245f66186e011e38a7a: Status 404 returned error can't find the container with id fe79dc0d3506fa5631f89a5cd2d131316b7cd54f11ec1245f66186e011e38a7a
Sep 18 17:15:35.298408 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:35.298380    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63db05d0-62bd-4fe6-999f-0d19ef2ff266-metrics-certs\") pod \"router-default-76465d565b-r2rdc\" (UID: \"63db05d0-62bd-4fe6-999f-0d19ef2ff266\") " pod="openshift-ingress/router-default-76465d565b-r2rdc"
Sep 18 17:15:35.298556 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:35.298455    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63db05d0-62bd-4fe6-999f-0d19ef2ff266-service-ca-bundle\") pod \"router-default-76465d565b-r2rdc\" (UID: \"63db05d0-62bd-4fe6-999f-0d19ef2ff266\") " pod="openshift-ingress/router-default-76465d565b-r2rdc"
Sep 18 17:15:35.298556 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:35.298521    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 17:15:35.298669 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:35.298585    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63db05d0-62bd-4fe6-999f-0d19ef2ff266-metrics-certs podName:63db05d0-62bd-4fe6-999f-0d19ef2ff266 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:36.29856967 +0000 UTC m=+48.992497176 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63db05d0-62bd-4fe6-999f-0d19ef2ff266-metrics-certs") pod "router-default-76465d565b-r2rdc" (UID: "63db05d0-62bd-4fe6-999f-0d19ef2ff266") : secret "router-metrics-certs-default" not found
Sep 18 17:15:35.298669 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:35.298598    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63db05d0-62bd-4fe6-999f-0d19ef2ff266-service-ca-bundle podName:63db05d0-62bd-4fe6-999f-0d19ef2ff266 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:36.298592176 +0000 UTC m=+48.992519682 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/63db05d0-62bd-4fe6-999f-0d19ef2ff266-service-ca-bundle") pod "router-default-76465d565b-r2rdc" (UID: "63db05d0-62bd-4fe6-999f-0d19ef2ff266") : configmap references non-existent config key: service-ca.crt
Sep 18 17:15:36.037433 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:36.037398    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-v4nfn" event={"ID":"cdd63e16-6865-47a1-86ca-c1870d2eae7d","Type":"ContainerStarted","Data":"fe79dc0d3506fa5631f89a5cd2d131316b7cd54f11ec1245f66186e011e38a7a"}
Sep 18 17:15:36.305180 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:36.305097    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63db05d0-62bd-4fe6-999f-0d19ef2ff266-service-ca-bundle\") pod \"router-default-76465d565b-r2rdc\" (UID: \"63db05d0-62bd-4fe6-999f-0d19ef2ff266\") " pod="openshift-ingress/router-default-76465d565b-r2rdc"
Sep 18 17:15:36.305322 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:36.305200    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63db05d0-62bd-4fe6-999f-0d19ef2ff266-metrics-certs\") pod \"router-default-76465d565b-r2rdc\" (UID: \"63db05d0-62bd-4fe6-999f-0d19ef2ff266\") " pod="openshift-ingress/router-default-76465d565b-r2rdc"
Sep 18 17:15:36.305322 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:36.305270    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63db05d0-62bd-4fe6-999f-0d19ef2ff266-service-ca-bundle podName:63db05d0-62bd-4fe6-999f-0d19ef2ff266 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:38.305241762 +0000 UTC m=+50.999169269 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/63db05d0-62bd-4fe6-999f-0d19ef2ff266-service-ca-bundle") pod "router-default-76465d565b-r2rdc" (UID: "63db05d0-62bd-4fe6-999f-0d19ef2ff266") : configmap references non-existent config key: service-ca.crt
Sep 18 17:15:36.305322 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:36.305304    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 17:15:36.305424 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:36.305345    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63db05d0-62bd-4fe6-999f-0d19ef2ff266-metrics-certs podName:63db05d0-62bd-4fe6-999f-0d19ef2ff266 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:38.305332249 +0000 UTC m=+50.999259755 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63db05d0-62bd-4fe6-999f-0d19ef2ff266-metrics-certs") pod "router-default-76465d565b-r2rdc" (UID: "63db05d0-62bd-4fe6-999f-0d19ef2ff266") : secret "router-metrics-certs-default" not found
Sep 18 17:15:38.044519 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:38.044478    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-v4nfn" event={"ID":"cdd63e16-6865-47a1-86ca-c1870d2eae7d","Type":"ContainerStarted","Data":"fd1716625ddfcb7359a2b917ea2ec546dbb3186a2b5de7e5cf9db99ad08995a1"}
Sep 18 17:15:38.319222 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:38.319139    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63db05d0-62bd-4fe6-999f-0d19ef2ff266-metrics-certs\") pod \"router-default-76465d565b-r2rdc\" (UID: \"63db05d0-62bd-4fe6-999f-0d19ef2ff266\") " pod="openshift-ingress/router-default-76465d565b-r2rdc"
Sep 18 17:15:38.319222 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:38.319211    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63db05d0-62bd-4fe6-999f-0d19ef2ff266-service-ca-bundle\") pod \"router-default-76465d565b-r2rdc\" (UID: \"63db05d0-62bd-4fe6-999f-0d19ef2ff266\") " pod="openshift-ingress/router-default-76465d565b-r2rdc"
Sep 18 17:15:38.319418 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:38.319317    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 17:15:38.319418 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:38.319371    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63db05d0-62bd-4fe6-999f-0d19ef2ff266-service-ca-bundle podName:63db05d0-62bd-4fe6-999f-0d19ef2ff266 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:42.319356467 +0000 UTC m=+55.013283974 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/63db05d0-62bd-4fe6-999f-0d19ef2ff266-service-ca-bundle") pod "router-default-76465d565b-r2rdc" (UID: "63db05d0-62bd-4fe6-999f-0d19ef2ff266") : configmap references non-existent config key: service-ca.crt
Sep 18 17:15:38.319418 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:38.319404    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63db05d0-62bd-4fe6-999f-0d19ef2ff266-metrics-certs podName:63db05d0-62bd-4fe6-999f-0d19ef2ff266 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:42.31938502 +0000 UTC m=+55.013312534 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63db05d0-62bd-4fe6-999f-0d19ef2ff266-metrics-certs") pod "router-default-76465d565b-r2rdc" (UID: "63db05d0-62bd-4fe6-999f-0d19ef2ff266") : secret "router-metrics-certs-default" not found
Sep 18 17:15:39.023879 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.023837    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/80573a4b-9369-42f3-94b4-c0673f122e62-metrics-tls\") pod \"dns-default-s566m\" (UID: \"80573a4b-9369-42f3-94b4-c0673f122e62\") " pod="openshift-dns/dns-default-s566m"
Sep 18 17:15:39.024067 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:39.023992    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 17:15:39.024067 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:39.024060    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80573a4b-9369-42f3-94b4-c0673f122e62-metrics-tls podName:80573a4b-9369-42f3-94b4-c0673f122e62 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:55.024045408 +0000 UTC m=+67.717972930 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/80573a4b-9369-42f3-94b4-c0673f122e62-metrics-tls") pod "dns-default-s566m" (UID: "80573a4b-9369-42f3-94b4-c0673f122e62") : secret "dns-default-metrics-tls" not found
Sep 18 17:15:39.024825 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.024779    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-v4nfn" podStartSLOduration=2.762897343 podStartE2EDuration="5.024765041s" totalImagesPullingTime="2.261867698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:15:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:15:38.068016307 +0000 UTC m=+50.761943861" watchObservedRunningTime="2026-09-18 17:15:39.024765041 +0000 UTC m=+51.718692571"
Sep 18 17:15:39.024932 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.024919    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5lm2g"]
Sep 18 17:15:39.037916 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.037897    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5lm2g"
Sep 18 17:15:39.042185 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.042167    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 17:15:39.042516 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.042497    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2sl55\""
Sep 18 17:15:39.043006 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.042988    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 17:15:39.049333 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.049311    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5lm2g"]
Sep 18 17:15:39.125217 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.125188    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/08f50d0f-2329-428f-ab3e-64159a5bfe35-data-volume\") pod \"insights-runtime-extractor-5lm2g\" (UID: \"08f50d0f-2329-428f-ab3e-64159a5bfe35\") " pod="openshift-insights/insights-runtime-extractor-5lm2g"
Sep 18 17:15:39.125405 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.125341    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08f50d0f-2329-428f-ab3e-64159a5bfe35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5lm2g\" (UID: \"08f50d0f-2329-428f-ab3e-64159a5bfe35\") " pod="openshift-insights/insights-runtime-extractor-5lm2g"
Sep 18 17:15:39.125481 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.125450    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc54l\" (UniqueName: \"kubernetes.io/projected/08f50d0f-2329-428f-ab3e-64159a5bfe35-kube-api-access-tc54l\") pod \"insights-runtime-extractor-5lm2g\" (UID: \"08f50d0f-2329-428f-ab3e-64159a5bfe35\") " pod="openshift-insights/insights-runtime-extractor-5lm2g"
Sep 18 17:15:39.125621 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.125603    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/08f50d0f-2329-428f-ab3e-64159a5bfe35-crio-socket\") pod \"insights-runtime-extractor-5lm2g\" (UID: \"08f50d0f-2329-428f-ab3e-64159a5bfe35\") " pod="openshift-insights/insights-runtime-extractor-5lm2g"
Sep 18 17:15:39.125703 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.125635    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08f50d0f-2329-428f-ab3e-64159a5bfe35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5lm2g\" (UID: \"08f50d0f-2329-428f-ab3e-64159a5bfe35\") " pod="openshift-insights/insights-runtime-extractor-5lm2g"
Sep 18 17:15:39.226942 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.226910    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08f50d0f-2329-428f-ab3e-64159a5bfe35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5lm2g\" (UID: \"08f50d0f-2329-428f-ab3e-64159a5bfe35\") " pod="openshift-insights/insights-runtime-extractor-5lm2g"
Sep 18 17:15:39.227141 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.226958    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/08f50d0f-2329-428f-ab3e-64159a5bfe35-data-volume\") pod \"insights-runtime-extractor-5lm2g\" (UID: \"08f50d0f-2329-428f-ab3e-64159a5bfe35\") " pod="openshift-insights/insights-runtime-extractor-5lm2g"
Sep 18 17:15:39.227141 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.227012    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08f50d0f-2329-428f-ab3e-64159a5bfe35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5lm2g\" (UID: \"08f50d0f-2329-428f-ab3e-64159a5bfe35\") " pod="openshift-insights/insights-runtime-extractor-5lm2g"
Sep 18 17:15:39.227141 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:39.227064    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 17:15:39.227141 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.227103    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc54l\" (UniqueName: \"kubernetes.io/projected/08f50d0f-2329-428f-ab3e-64159a5bfe35-kube-api-access-tc54l\") pod \"insights-runtime-extractor-5lm2g\" (UID: \"08f50d0f-2329-428f-ab3e-64159a5bfe35\") " pod="openshift-insights/insights-runtime-extractor-5lm2g"
Sep 18 17:15:39.227141 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:39.227125    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08f50d0f-2329-428f-ab3e-64159a5bfe35-insights-runtime-extractor-tls podName:08f50d0f-2329-428f-ab3e-64159a5bfe35 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:39.727107207 +0000 UTC m=+52.421034731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/08f50d0f-2329-428f-ab3e-64159a5bfe35-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5lm2g" (UID: "08f50d0f-2329-428f-ab3e-64159a5bfe35") : secret "insights-runtime-extractor-tls" not found
Sep 18 17:15:39.227410 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.227160    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/08f50d0f-2329-428f-ab3e-64159a5bfe35-crio-socket\") pod \"insights-runtime-extractor-5lm2g\" (UID: \"08f50d0f-2329-428f-ab3e-64159a5bfe35\") " pod="openshift-insights/insights-runtime-extractor-5lm2g"
Sep 18 17:15:39.227410 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:39.227182    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 17:15:39.227410 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:39.227248    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/08f50d0f-2329-428f-ab3e-64159a5bfe35-kube-rbac-proxy-cm podName:08f50d0f-2329-428f-ab3e-64159a5bfe35 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:39.727230862 +0000 UTC m=+52.421158388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/08f50d0f-2329-428f-ab3e-64159a5bfe35-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5lm2g" (UID: "08f50d0f-2329-428f-ab3e-64159a5bfe35") : configmap "kube-rbac-proxy" not found
Sep 18 17:15:39.227410 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.227337    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/08f50d0f-2329-428f-ab3e-64159a5bfe35-crio-socket\") pod \"insights-runtime-extractor-5lm2g\" (UID: \"08f50d0f-2329-428f-ab3e-64159a5bfe35\") " pod="openshift-insights/insights-runtime-extractor-5lm2g"
Sep 18 17:15:39.227410 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.227374    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/08f50d0f-2329-428f-ab3e-64159a5bfe35-data-volume\") pod \"insights-runtime-extractor-5lm2g\" (UID: \"08f50d0f-2329-428f-ab3e-64159a5bfe35\") " pod="openshift-insights/insights-runtime-extractor-5lm2g"
Sep 18 17:15:39.238650 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.238631    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc54l\" (UniqueName: \"kubernetes.io/projected/08f50d0f-2329-428f-ab3e-64159a5bfe35-kube-api-access-tc54l\") pod \"insights-runtime-extractor-5lm2g\" (UID: \"08f50d0f-2329-428f-ab3e-64159a5bfe35\") " pod="openshift-insights/insights-runtime-extractor-5lm2g"
Sep 18 17:15:39.440945 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.440853    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-5kvpm"]
Sep 18 17:15:39.442686 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.442671    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5kvpm"
Sep 18 17:15:39.445756 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.445731    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 18 17:15:39.445922 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.445757    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 18 17:15:39.445922 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.445732    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 18 17:15:39.445922 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.445886    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 18 17:15:39.446189 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.446173    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-q5x54\""
Sep 18 17:15:39.454019 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.453996    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-5kvpm"]
Sep 18 17:15:39.530025 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.529990    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/00c0c1f1-e495-4fc3-8dd6-ec7a4d316728-config\") pod \"service-ca-operator-57cdc4b768-5kvpm\" (UID: \"00c0c1f1-e495-4fc3-8dd6-ec7a4d316728\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5kvpm"
Sep 18 17:15:39.530182 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.530050    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/00c0c1f1-e495-4fc3-8dd6-ec7a4d316728-serving-cert\") pod \"service-ca-operator-57cdc4b768-5kvpm\" (UID: \"00c0c1f1-e495-4fc3-8dd6-ec7a4d316728\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5kvpm"
Sep 18 17:15:39.530182 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.530129    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l87bz\" (UniqueName: \"kubernetes.io/projected/00c0c1f1-e495-4fc3-8dd6-ec7a4d316728-kube-api-access-l87bz\") pod \"service-ca-operator-57cdc4b768-5kvpm\" (UID: \"00c0c1f1-e495-4fc3-8dd6-ec7a4d316728\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5kvpm"
Sep 18 17:15:39.631204 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.631164    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/00c0c1f1-e495-4fc3-8dd6-ec7a4d316728-serving-cert\") pod \"service-ca-operator-57cdc4b768-5kvpm\" (UID: \"00c0c1f1-e495-4fc3-8dd6-ec7a4d316728\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5kvpm"
Sep 18 17:15:39.631370 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.631220    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l87bz\" (UniqueName: \"kubernetes.io/projected/00c0c1f1-e495-4fc3-8dd6-ec7a4d316728-kube-api-access-l87bz\") pod \"service-ca-operator-57cdc4b768-5kvpm\" (UID: \"00c0c1f1-e495-4fc3-8dd6-ec7a4d316728\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5kvpm"
Sep 18 17:15:39.631411 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.631391    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/00c0c1f1-e495-4fc3-8dd6-ec7a4d316728-config\") pod \"service-ca-operator-57cdc4b768-5kvpm\" (UID: \"00c0c1f1-e495-4fc3-8dd6-ec7a4d316728\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5kvpm"
Sep 18 17:15:39.632324 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.632307    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/00c0c1f1-e495-4fc3-8dd6-ec7a4d316728-config\") pod \"service-ca-operator-57cdc4b768-5kvpm\" (UID: \"00c0c1f1-e495-4fc3-8dd6-ec7a4d316728\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5kvpm"
Sep 18 17:15:39.633257 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.633239    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/00c0c1f1-e495-4fc3-8dd6-ec7a4d316728-serving-cert\") pod \"service-ca-operator-57cdc4b768-5kvpm\" (UID: \"00c0c1f1-e495-4fc3-8dd6-ec7a4d316728\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5kvpm"
Sep 18 17:15:39.643422 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.643390    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l87bz\" (UniqueName: \"kubernetes.io/projected/00c0c1f1-e495-4fc3-8dd6-ec7a4d316728-kube-api-access-l87bz\") pod \"service-ca-operator-57cdc4b768-5kvpm\" (UID: \"00c0c1f1-e495-4fc3-8dd6-ec7a4d316728\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5kvpm"
Sep 18 17:15:39.732126 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.732096    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08f50d0f-2329-428f-ab3e-64159a5bfe35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5lm2g\" (UID: \"08f50d0f-2329-428f-ab3e-64159a5bfe35\") " pod="openshift-insights/insights-runtime-extractor-5lm2g"
Sep 18 17:15:39.732282 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.732161    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08f50d0f-2329-428f-ab3e-64159a5bfe35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5lm2g\" (UID: \"08f50d0f-2329-428f-ab3e-64159a5bfe35\") " pod="openshift-insights/insights-runtime-extractor-5lm2g"
Sep 18 17:15:39.732282 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:39.732232    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 17:15:39.732282 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:39.732247    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 17:15:39.732282 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:39.732283    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/08f50d0f-2329-428f-ab3e-64159a5bfe35-kube-rbac-proxy-cm podName:08f50d0f-2329-428f-ab3e-64159a5bfe35 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:40.73226962 +0000 UTC m=+53.426197139 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/08f50d0f-2329-428f-ab3e-64159a5bfe35-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5lm2g" (UID: "08f50d0f-2329-428f-ab3e-64159a5bfe35") : configmap "kube-rbac-proxy" not found
Sep 18 17:15:39.732417 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:39.732309    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08f50d0f-2329-428f-ab3e-64159a5bfe35-insights-runtime-extractor-tls podName:08f50d0f-2329-428f-ab3e-64159a5bfe35 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:40.732295516 +0000 UTC m=+53.426223035 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/08f50d0f-2329-428f-ab3e-64159a5bfe35-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5lm2g" (UID: "08f50d0f-2329-428f-ab3e-64159a5bfe35") : secret "insights-runtime-extractor-tls" not found
Sep 18 17:15:39.751016 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:39.750985    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5kvpm"
Sep 18 17:15:40.025608 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:40.025517    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-5kvpm"]
Sep 18 17:15:40.029131 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:15:40.029104    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c0c1f1_e495_4fc3_8dd6_ec7a4d316728.slice/crio-85b4870fd6dd7bb8b9e4678ea4de774f0fbf730500b057cfa3158d8232350bba WatchSource:0}: Error finding container 85b4870fd6dd7bb8b9e4678ea4de774f0fbf730500b057cfa3158d8232350bba: Status 404 returned error can't find the container with id 85b4870fd6dd7bb8b9e4678ea4de774f0fbf730500b057cfa3158d8232350bba
Sep 18 17:15:40.049818 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:40.049784    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5kvpm" event={"ID":"00c0c1f1-e495-4fc3-8dd6-ec7a4d316728","Type":"ContainerStarted","Data":"85b4870fd6dd7bb8b9e4678ea4de774f0fbf730500b057cfa3158d8232350bba"}
Sep 18 17:15:40.740260 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:40.740215    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08f50d0f-2329-428f-ab3e-64159a5bfe35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5lm2g\" (UID: \"08f50d0f-2329-428f-ab3e-64159a5bfe35\") " pod="openshift-insights/insights-runtime-extractor-5lm2g"
Sep 18 17:15:40.740453 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:40.740339    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08f50d0f-2329-428f-ab3e-64159a5bfe35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5lm2g\" (UID: \"08f50d0f-2329-428f-ab3e-64159a5bfe35\") " pod="openshift-insights/insights-runtime-extractor-5lm2g"
Sep 18 17:15:40.740453 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:40.740348    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 17:15:40.740453 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:40.740413    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/08f50d0f-2329-428f-ab3e-64159a5bfe35-kube-rbac-proxy-cm podName:08f50d0f-2329-428f-ab3e-64159a5bfe35 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:42.740394505 +0000 UTC m=+55.434322028 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/08f50d0f-2329-428f-ab3e-64159a5bfe35-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5lm2g" (UID: "08f50d0f-2329-428f-ab3e-64159a5bfe35") : configmap "kube-rbac-proxy" not found
Sep 18 17:15:40.740453 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:40.740443    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 17:15:40.740668 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:40.740490    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08f50d0f-2329-428f-ab3e-64159a5bfe35-insights-runtime-extractor-tls podName:08f50d0f-2329-428f-ab3e-64159a5bfe35 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:42.740479353 +0000 UTC m=+55.434406874 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/08f50d0f-2329-428f-ab3e-64159a5bfe35-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5lm2g" (UID: "08f50d0f-2329-428f-ab3e-64159a5bfe35") : secret "insights-runtime-extractor-tls" not found
Sep 18 17:15:41.108553 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:41.108479    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zjhnk_fa3b7283-8f5d-4d10-a7ab-6bd10036542f/dns-node-resolver/0.log"
Sep 18 17:15:41.316157 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:41.316121    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-fls6r"]
Sep 18 17:15:41.318163 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:41.318145    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-fls6r"
Sep 18 17:15:41.321029 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:41.321007    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 18 17:15:41.321136 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:41.321077    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-qnw72\""
Sep 18 17:15:41.321200 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:41.321007    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 18 17:15:41.327588 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:41.327567    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-fls6r"]
Sep 18 17:15:41.346512 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:41.346488    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cp25\" (UniqueName: \"kubernetes.io/projected/882e18bc-0e22-45aa-88ac-e934bffd8b94-kube-api-access-2cp25\") pod \"migrator-76794cbd6c-fls6r\" (UID: \"882e18bc-0e22-45aa-88ac-e934bffd8b94\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-fls6r"
Sep 18 17:15:41.447181 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:41.447086    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cp25\" (UniqueName: \"kubernetes.io/projected/882e18bc-0e22-45aa-88ac-e934bffd8b94-kube-api-access-2cp25\") pod \"migrator-76794cbd6c-fls6r\" (UID: \"882e18bc-0e22-45aa-88ac-e934bffd8b94\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-fls6r"
Sep 18 17:15:41.458084 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:41.458053    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cp25\" (UniqueName: \"kubernetes.io/projected/882e18bc-0e22-45aa-88ac-e934bffd8b94-kube-api-access-2cp25\") pod \"migrator-76794cbd6c-fls6r\" (UID: \"882e18bc-0e22-45aa-88ac-e934bffd8b94\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-fls6r"
Sep 18 17:15:41.628470 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:41.628437    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-fls6r"
Sep 18 17:15:41.901915 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:41.901884    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-fls6r"]
Sep 18 17:15:41.905473 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:15:41.905446    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod882e18bc_0e22_45aa_88ac_e934bffd8b94.slice/crio-e2036e8e86348b13f3dcf9ec049473703b58d51d0f6aa685cd435ac2c4659aa4 WatchSource:0}: Error finding container e2036e8e86348b13f3dcf9ec049473703b58d51d0f6aa685cd435ac2c4659aa4: Status 404 returned error can't find the container with id e2036e8e86348b13f3dcf9ec049473703b58d51d0f6aa685cd435ac2c4659aa4
Sep 18 17:15:42.056764 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:42.056729    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5kvpm" event={"ID":"00c0c1f1-e495-4fc3-8dd6-ec7a4d316728","Type":"ContainerStarted","Data":"7e825c901e36d330eff38c145316774443c58017e4371adc5464556fb379c524"}
Sep 18 17:15:42.057941 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:42.057910    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-fls6r" event={"ID":"882e18bc-0e22-45aa-88ac-e934bffd8b94","Type":"ContainerStarted","Data":"e2036e8e86348b13f3dcf9ec049473703b58d51d0f6aa685cd435ac2c4659aa4"}
Sep 18 17:15:42.082709 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:42.082664    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5kvpm" podStartSLOduration=1.593105116 podStartE2EDuration="3.082646157s" totalImagesPullingTime="1.489541041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:15:42.082332779 +0000 UTC m=+54.776260306" watchObservedRunningTime="2026-09-18 17:15:42.082646157 +0000 UTC m=+54.776573684"
Sep 18 17:15:42.110087 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:42.110060    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z6nxd_2af39eb4-05e3-4eca-87d1-87558b5ae1c1/node-ca/0.log"
Sep 18 17:15:42.354139 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:42.354106    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63db05d0-62bd-4fe6-999f-0d19ef2ff266-metrics-certs\") pod \"router-default-76465d565b-r2rdc\" (UID: \"63db05d0-62bd-4fe6-999f-0d19ef2ff266\") " pod="openshift-ingress/router-default-76465d565b-r2rdc"
Sep 18 17:15:42.354329 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:42.354179    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63db05d0-62bd-4fe6-999f-0d19ef2ff266-service-ca-bundle\") pod \"router-default-76465d565b-r2rdc\" (UID: \"63db05d0-62bd-4fe6-999f-0d19ef2ff266\") " pod="openshift-ingress/router-default-76465d565b-r2rdc"
Sep 18 17:15:42.354329 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:42.354270    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 17:15:42.354329 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:42.354308    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63db05d0-62bd-4fe6-999f-0d19ef2ff266-service-ca-bundle podName:63db05d0-62bd-4fe6-999f-0d19ef2ff266 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:50.354290645 +0000 UTC m=+63.048218173 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/63db05d0-62bd-4fe6-999f-0d19ef2ff266-service-ca-bundle") pod "router-default-76465d565b-r2rdc" (UID: "63db05d0-62bd-4fe6-999f-0d19ef2ff266") : configmap references non-existent config key: service-ca.crt
Sep 18 17:15:42.354329 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:42.354324    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63db05d0-62bd-4fe6-999f-0d19ef2ff266-metrics-certs podName:63db05d0-62bd-4fe6-999f-0d19ef2ff266 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:50.354317771 +0000 UTC m=+63.048245276 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63db05d0-62bd-4fe6-999f-0d19ef2ff266-metrics-certs") pod "router-default-76465d565b-r2rdc" (UID: "63db05d0-62bd-4fe6-999f-0d19ef2ff266") : secret "router-metrics-certs-default" not found
Sep 18 17:15:42.755807 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:42.755774    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08f50d0f-2329-428f-ab3e-64159a5bfe35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5lm2g\" (UID: \"08f50d0f-2329-428f-ab3e-64159a5bfe35\") " pod="openshift-insights/insights-runtime-extractor-5lm2g"
Sep 18 17:15:42.755999 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:42.755851    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08f50d0f-2329-428f-ab3e-64159a5bfe35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5lm2g\" (UID: \"08f50d0f-2329-428f-ab3e-64159a5bfe35\") " pod="openshift-insights/insights-runtime-extractor-5lm2g"
Sep 18 17:15:42.755999 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:42.755969    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 17:15:42.756082 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:42.756029    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08f50d0f-2329-428f-ab3e-64159a5bfe35-insights-runtime-extractor-tls podName:08f50d0f-2329-428f-ab3e-64159a5bfe35 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:46.756014861 +0000 UTC m=+59.449942367 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/08f50d0f-2329-428f-ab3e-64159a5bfe35-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5lm2g" (UID: "08f50d0f-2329-428f-ab3e-64159a5bfe35") : secret "insights-runtime-extractor-tls" not found
Sep 18 17:15:42.756082 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:42.755969    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 17:15:42.756167 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:42.756099    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/08f50d0f-2329-428f-ab3e-64159a5bfe35-kube-rbac-proxy-cm podName:08f50d0f-2329-428f-ab3e-64159a5bfe35 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:46.756085403 +0000 UTC m=+59.450012912 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/08f50d0f-2329-428f-ab3e-64159a5bfe35-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5lm2g" (UID: "08f50d0f-2329-428f-ab3e-64159a5bfe35") : configmap "kube-rbac-proxy" not found
Sep 18 17:15:42.911331 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:42.911276    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nwncv_1549ca4a-cf93-4295-8f95-c869e2b6a1c6/serve-healthcheck-canary/0.log"
Sep 18 17:15:45.067600 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:45.067561    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-fls6r" event={"ID":"882e18bc-0e22-45aa-88ac-e934bffd8b94","Type":"ContainerStarted","Data":"75e8a18c0b579ed4abe40e0f3f690bf5dd00a16041e03c4afe3eca003f643f3d"}
Sep 18 17:15:45.068048 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:45.067614    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-fls6r" event={"ID":"882e18bc-0e22-45aa-88ac-e934bffd8b94","Type":"ContainerStarted","Data":"73ba5bddf7e2ad7d7a039bbad93316f92fc042dd9cb884dcf22b33a62e81b9c3"}
Sep 18 17:15:45.087267 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:45.087221    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-fls6r" podStartSLOduration=1.6475869090000002 podStartE2EDuration="4.087206584s" totalImagesPullingTime="2.439619675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:15:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:15:45.085783465 +0000 UTC m=+57.779710993" watchObservedRunningTime="2026-09-18 17:15:45.087206584 +0000 UTC m=+57.781134160"
Sep 18 17:15:46.791056 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:46.791015    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08f50d0f-2329-428f-ab3e-64159a5bfe35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5lm2g\" (UID: \"08f50d0f-2329-428f-ab3e-64159a5bfe35\") " pod="openshift-insights/insights-runtime-extractor-5lm2g"
Sep 18 17:15:46.791491 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:46.791164    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 17:15:46.791491 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:46.791173    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08f50d0f-2329-428f-ab3e-64159a5bfe35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5lm2g\" (UID: \"08f50d0f-2329-428f-ab3e-64159a5bfe35\") " pod="openshift-insights/insights-runtime-extractor-5lm2g"
Sep 18 17:15:46.791491 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:46.791256    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08f50d0f-2329-428f-ab3e-64159a5bfe35-insights-runtime-extractor-tls podName:08f50d0f-2329-428f-ab3e-64159a5bfe35 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:54.791237046 +0000 UTC m=+67.485164554 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/08f50d0f-2329-428f-ab3e-64159a5bfe35-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5lm2g" (UID: "08f50d0f-2329-428f-ab3e-64159a5bfe35") : secret "insights-runtime-extractor-tls" not found
Sep 18 17:15:46.791491 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:46.791284    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 17:15:46.791491 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:46.791335    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/08f50d0f-2329-428f-ab3e-64159a5bfe35-kube-rbac-proxy-cm podName:08f50d0f-2329-428f-ab3e-64159a5bfe35 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:54.791322959 +0000 UTC m=+67.485250465 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/08f50d0f-2329-428f-ab3e-64159a5bfe35-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5lm2g" (UID: "08f50d0f-2329-428f-ab3e-64159a5bfe35") : configmap "kube-rbac-proxy" not found
Sep 18 17:15:47.984277 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:47.984247    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-54gxw"
Sep 18 17:15:50.420319 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:50.420278    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63db05d0-62bd-4fe6-999f-0d19ef2ff266-metrics-certs\") pod \"router-default-76465d565b-r2rdc\" (UID: \"63db05d0-62bd-4fe6-999f-0d19ef2ff266\") " pod="openshift-ingress/router-default-76465d565b-r2rdc"
Sep 18 17:15:50.420694 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:50.420364    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63db05d0-62bd-4fe6-999f-0d19ef2ff266-service-ca-bundle\") pod \"router-default-76465d565b-r2rdc\" (UID: \"63db05d0-62bd-4fe6-999f-0d19ef2ff266\") " pod="openshift-ingress/router-default-76465d565b-r2rdc"
Sep 18 17:15:50.420694 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:50.420425    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 17:15:50.420694 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:50.420491    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63db05d0-62bd-4fe6-999f-0d19ef2ff266-metrics-certs podName:63db05d0-62bd-4fe6-999f-0d19ef2ff266 nodeName:}" failed. No retries permitted until 2026-09-18 17:16:06.420475958 +0000 UTC m=+79.114403463 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63db05d0-62bd-4fe6-999f-0d19ef2ff266-metrics-certs") pod "router-default-76465d565b-r2rdc" (UID: "63db05d0-62bd-4fe6-999f-0d19ef2ff266") : secret "router-metrics-certs-default" not found
Sep 18 17:15:50.420694 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:50.420507    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63db05d0-62bd-4fe6-999f-0d19ef2ff266-service-ca-bundle podName:63db05d0-62bd-4fe6-999f-0d19ef2ff266 nodeName:}" failed. No retries permitted until 2026-09-18 17:16:06.420501088 +0000 UTC m=+79.114428593 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/63db05d0-62bd-4fe6-999f-0d19ef2ff266-service-ca-bundle") pod "router-default-76465d565b-r2rdc" (UID: "63db05d0-62bd-4fe6-999f-0d19ef2ff266") : configmap references non-existent config key: service-ca.crt
Sep 18 17:15:53.546149 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:53.546082    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91a9651b-affa-44e8-8996-1ea02fa0c2c4-metrics-certs\") pod \"network-metrics-daemon-ltwd4\" (UID: \"91a9651b-affa-44e8-8996-1ea02fa0c2c4\") " pod="openshift-multus/network-metrics-daemon-ltwd4"
Sep 18 17:15:53.546691 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:53.546163    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/491a6ab9-f579-4c60-a0ea-6b6aaad27d52-original-pull-secret\") pod \"global-pull-secret-syncer-7kwfm\" (UID: \"491a6ab9-f579-4c60-a0ea-6b6aaad27d52\") " pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:15:53.548967 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:53.548945    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 17:15:53.549089 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:53.548982    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 17:15:53.556607 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:53.556590    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 17:15:53.556711 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:53.556666    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91a9651b-affa-44e8-8996-1ea02fa0c2c4-metrics-certs podName:91a9651b-affa-44e8-8996-1ea02fa0c2c4 nodeName:}" failed. No retries permitted until 2026-09-18 17:16:57.55664615 +0000 UTC m=+130.250573670 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91a9651b-affa-44e8-8996-1ea02fa0c2c4-metrics-certs") pod "network-metrics-daemon-ltwd4" (UID: "91a9651b-affa-44e8-8996-1ea02fa0c2c4") : secret "metrics-daemon-secret" not found
Sep 18 17:15:53.558694 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:53.558675    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/491a6ab9-f579-4c60-a0ea-6b6aaad27d52-original-pull-secret\") pod \"global-pull-secret-syncer-7kwfm\" (UID: \"491a6ab9-f579-4c60-a0ea-6b6aaad27d52\") " pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:15:53.577275 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:53.577255    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kwfm"
Sep 18 17:15:53.646567 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:53.646535    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v5fr\" (UniqueName: \"kubernetes.io/projected/3641bd80-d924-469d-a138-2300c1747d0f-kube-api-access-8v5fr\") pod \"network-check-target-l9knh\" (UID: \"3641bd80-d924-469d-a138-2300c1747d0f\") " pod="openshift-network-diagnostics/network-check-target-l9knh"
Sep 18 17:15:53.649753 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:53.649732    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 17:15:53.660297 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:53.660278    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 17:15:53.670499 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:53.670470    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v5fr\" (UniqueName: \"kubernetes.io/projected/3641bd80-d924-469d-a138-2300c1747d0f-kube-api-access-8v5fr\") pod \"network-check-target-l9knh\" (UID: \"3641bd80-d924-469d-a138-2300c1747d0f\") " pod="openshift-network-diagnostics/network-check-target-l9knh"
Sep 18 17:15:53.842099 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:53.842024    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7kwfm"]
Sep 18 17:15:53.845187 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:15:53.845156    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod491a6ab9_f579_4c60_a0ea_6b6aaad27d52.slice/crio-942da15812a3e07dd4a32da9f2d59b605a17c7c4b616e16acd218cbd2f570336 WatchSource:0}: Error finding container 942da15812a3e07dd4a32da9f2d59b605a17c7c4b616e16acd218cbd2f570336: Status 404 returned error can't find the container with id 942da15812a3e07dd4a32da9f2d59b605a17c7c4b616e16acd218cbd2f570336
Sep 18 17:15:53.874213 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:53.874191    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-f2lgv\""
Sep 18 17:15:53.882236 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:53.882220    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9knh"
Sep 18 17:15:54.093357 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:54.093273    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7kwfm" event={"ID":"491a6ab9-f579-4c60-a0ea-6b6aaad27d52","Type":"ContainerStarted","Data":"942da15812a3e07dd4a32da9f2d59b605a17c7c4b616e16acd218cbd2f570336"}
Sep 18 17:15:54.148482 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:54.148454    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l9knh"]
Sep 18 17:15:54.152272 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:15:54.152252    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3641bd80_d924_469d_a138_2300c1747d0f.slice/crio-e1354d22974cd9776dc84b2787c1555fda58c80863af63e0d0ebc86090b04416 WatchSource:0}: Error finding container e1354d22974cd9776dc84b2787c1555fda58c80863af63e0d0ebc86090b04416: Status 404 returned error can't find the container with id e1354d22974cd9776dc84b2787c1555fda58c80863af63e0d0ebc86090b04416
Sep 18 17:15:54.854097 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:54.854062    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08f50d0f-2329-428f-ab3e-64159a5bfe35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5lm2g\" (UID: \"08f50d0f-2329-428f-ab3e-64159a5bfe35\") " pod="openshift-insights/insights-runtime-extractor-5lm2g"
Sep 18 17:15:54.854564 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:54.854139    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08f50d0f-2329-428f-ab3e-64159a5bfe35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5lm2g\" (UID: \"08f50d0f-2329-428f-ab3e-64159a5bfe35\") " pod="openshift-insights/insights-runtime-extractor-5lm2g"
Sep 18 17:15:54.854564 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:54.854249    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 17:15:54.854564 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:15:54.854318    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08f50d0f-2329-428f-ab3e-64159a5bfe35-insights-runtime-extractor-tls podName:08f50d0f-2329-428f-ab3e-64159a5bfe35 nodeName:}" failed. No retries permitted until 2026-09-18 17:16:10.854301628 +0000 UTC m=+83.548229139 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/08f50d0f-2329-428f-ab3e-64159a5bfe35-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5lm2g" (UID: "08f50d0f-2329-428f-ab3e-64159a5bfe35") : secret "insights-runtime-extractor-tls" not found
Sep 18 17:15:54.854816 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:54.854786    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08f50d0f-2329-428f-ab3e-64159a5bfe35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5lm2g\" (UID: \"08f50d0f-2329-428f-ab3e-64159a5bfe35\") " pod="openshift-insights/insights-runtime-extractor-5lm2g"
Sep 18 17:15:55.056049 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:55.056017    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/80573a4b-9369-42f3-94b4-c0673f122e62-metrics-tls\") pod \"dns-default-s566m\" (UID: \"80573a4b-9369-42f3-94b4-c0673f122e62\") " pod="openshift-dns/dns-default-s566m"
Sep 18 17:15:55.058692 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:55.058668    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/80573a4b-9369-42f3-94b4-c0673f122e62-metrics-tls\") pod \"dns-default-s566m\" (UID: \"80573a4b-9369-42f3-94b4-c0673f122e62\") " pod="openshift-dns/dns-default-s566m"
Sep 18 17:15:55.097729 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:55.097691    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l9knh" event={"ID":"3641bd80-d924-469d-a138-2300c1747d0f","Type":"ContainerStarted","Data":"e1354d22974cd9776dc84b2787c1555fda58c80863af63e0d0ebc86090b04416"}
Sep 18 17:15:55.264881 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:55.264834    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-99knh\""
Sep 18 17:15:55.271851 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:55.271824    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-s566m"
Sep 18 17:15:55.547080 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:55.546936    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-s566m"]
Sep 18 17:15:55.549529 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:15:55.549504    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80573a4b_9369_42f3_94b4_c0673f122e62.slice/crio-1f9dae7d28cf02c571fccff47eca18564b1a6a716056bde2c07aefb6782f1aa1 WatchSource:0}: Error finding container 1f9dae7d28cf02c571fccff47eca18564b1a6a716056bde2c07aefb6782f1aa1: Status 404 returned error can't find the container with id 1f9dae7d28cf02c571fccff47eca18564b1a6a716056bde2c07aefb6782f1aa1
Sep 18 17:15:56.104790 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:15:56.104748    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s566m" event={"ID":"80573a4b-9369-42f3-94b4-c0673f122e62","Type":"ContainerStarted","Data":"1f9dae7d28cf02c571fccff47eca18564b1a6a716056bde2c07aefb6782f1aa1"}
Sep 18 17:16:00.118850 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:00.118803    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7kwfm" event={"ID":"491a6ab9-f579-4c60-a0ea-6b6aaad27d52","Type":"ContainerStarted","Data":"7afd110a90dd00a48b13dce4b3ca429fa089a351878cd7cbe49a80d1534b5f44"}
Sep 18 17:16:00.120603 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:00.120578    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l9knh" event={"ID":"3641bd80-d924-469d-a138-2300c1747d0f","Type":"ContainerStarted","Data":"2f602e57170a4743cede959e36b43240f459f1885643f82d6cbb4d10c23de70e"}
Sep 18 17:16:00.120734 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:00.120627    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-l9knh"
Sep 18 17:16:00.122314 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:00.122295    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s566m" event={"ID":"80573a4b-9369-42f3-94b4-c0673f122e62","Type":"ContainerStarted","Data":"3c7b5debcc55f4ac2e285c378b30b103b90b8b41ea41e2b72eee132969ec0fd5"}
Sep 18 17:16:00.122382 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:00.122339    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s566m" event={"ID":"80573a4b-9369-42f3-94b4-c0673f122e62","Type":"ContainerStarted","Data":"c750633f245cf579c519007ad0e50749d3e4af704a4f41abad2a9128a3a2c953"}
Sep 18 17:16:00.122382 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:00.122360    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-s566m"
Sep 18 17:16:00.134695 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:00.134654    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7kwfm" podStartSLOduration=66.301629619 podStartE2EDuration="1m12.134641151s" totalImagesPullingTime="5.833011532s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:14:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:16:00.133776401 +0000 UTC m=+72.827703949" watchObservedRunningTime="2026-09-18 17:16:00.134641151 +0000 UTC m=+72.828568678"
Sep 18 17:16:00.150981 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:00.150935    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-l9knh" podStartSLOduration=66.615549782 podStartE2EDuration="1m12.150922748s" totalImagesPullingTime="5.535372966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:14:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:16:00.150034261 +0000 UTC m=+72.843961800" watchObservedRunningTime="2026-09-18 17:16:00.150922748 +0000 UTC m=+72.844850284"
Sep 18 17:16:00.169007 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:00.168958    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-s566m" podStartSLOduration=33.035332357 podStartE2EDuration="37.168943388s" totalImagesPullingTime="4.133611031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:16:00.168224214 +0000 UTC m=+72.862151743" watchObservedRunningTime="2026-09-18 17:16:00.168943388 +0000 UTC m=+72.862870917"
Sep 18 17:16:04.226086 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.226052    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4d9599b-2kxt5"]
Sep 18 17:16:04.227887 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.227854    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4d9599b-2kxt5"
Sep 18 17:16:04.230788 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.230766    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-sw2mj\""
Sep 18 17:16:04.232131 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.232115    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 18 17:16:04.232218 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.232118    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 18 17:16:04.232218 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.232120    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 18 17:16:04.232507 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.232491    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 18 17:16:04.246487 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.246464    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4d9599b-2kxt5"]
Sep 18 17:16:04.306330 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.306302    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-n6wb8"]
Sep 18 17:16:04.308313 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.308296    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-n6wb8"
Sep 18 17:16:04.311928 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.311907    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-hngfm\""
Sep 18 17:16:04.312361 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.312340    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 18 17:16:04.312361 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.312354    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 18 17:16:04.319785 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.319760    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-86b9797894-fbhdx"]
Sep 18 17:16:04.322048 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.322027    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86b9797894-fbhdx"
Sep 18 17:16:04.325500 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.325449    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 17:16:04.325500 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.325466    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 17:16:04.325812 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.325791    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mr4mj\""
Sep 18 17:16:04.326050 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.326034    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 17:16:04.326999 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.326979    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-n6wb8"]
Sep 18 17:16:04.329140 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.329020    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cf5a984f-55ce-4fee-b70f-532033699e50-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7b4d9599b-2kxt5\" (UID: \"cf5a984f-55ce-4fee-b70f-532033699e50\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4d9599b-2kxt5"
Sep 18 17:16:04.329140 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.329093    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnc6d\" (UniqueName: \"kubernetes.io/projected/cf5a984f-55ce-4fee-b70f-532033699e50-kube-api-access-fnc6d\") pod \"managed-serviceaccount-addon-agent-7b4d9599b-2kxt5\" (UID: \"cf5a984f-55ce-4fee-b70f-532033699e50\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4d9599b-2kxt5"
Sep 18 17:16:04.330618 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.330595    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 17:16:04.340373 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.340345    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86b9797894-fbhdx"]
Sep 18 17:16:04.429763 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.429731    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9e770835-5d2a-4225-8465-88b59b66badb-installation-pull-secrets\") pod \"image-registry-86b9797894-fbhdx\" (UID: \"9e770835-5d2a-4225-8465-88b59b66badb\") " pod="openshift-image-registry/image-registry-86b9797894-fbhdx"
Sep 18 17:16:04.429971 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.429791    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnc6d\" (UniqueName: \"kubernetes.io/projected/cf5a984f-55ce-4fee-b70f-532033699e50-kube-api-access-fnc6d\") pod \"managed-serviceaccount-addon-agent-7b4d9599b-2kxt5\" (UID: \"cf5a984f-55ce-4fee-b70f-532033699e50\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4d9599b-2kxt5"
Sep 18 17:16:04.429971 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.429810    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9e770835-5d2a-4225-8465-88b59b66badb-ca-trust-extracted\") pod \"image-registry-86b9797894-fbhdx\" (UID: \"9e770835-5d2a-4225-8465-88b59b66badb\") " pod="openshift-image-registry/image-registry-86b9797894-fbhdx"
Sep 18 17:16:04.429971 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.429826    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p78ts\" (UniqueName: \"kubernetes.io/projected/9e770835-5d2a-4225-8465-88b59b66badb-kube-api-access-p78ts\") pod \"image-registry-86b9797894-fbhdx\" (UID: \"9e770835-5d2a-4225-8465-88b59b66badb\") " pod="openshift-image-registry/image-registry-86b9797894-fbhdx"
Sep 18 17:16:04.430077 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.429974    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e770835-5d2a-4225-8465-88b59b66badb-bound-sa-token\") pod \"image-registry-86b9797894-fbhdx\" (UID: \"9e770835-5d2a-4225-8465-88b59b66badb\") " pod="openshift-image-registry/image-registry-86b9797894-fbhdx"
Sep 18 17:16:04.430077 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.430013    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9e770835-5d2a-4225-8465-88b59b66badb-registry-tls\") pod \"image-registry-86b9797894-fbhdx\" (UID: \"9e770835-5d2a-4225-8465-88b59b66badb\") " pod="openshift-image-registry/image-registry-86b9797894-fbhdx"
Sep 18 17:16:04.430077 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.430066    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9e770835-5d2a-4225-8465-88b59b66badb-registry-certificates\") pod \"image-registry-86b9797894-fbhdx\" (UID: \"9e770835-5d2a-4225-8465-88b59b66badb\") " pod="openshift-image-registry/image-registry-86b9797894-fbhdx"
Sep 18 17:16:04.430170 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.430090    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e770835-5d2a-4225-8465-88b59b66badb-trusted-ca\") pod \"image-registry-86b9797894-fbhdx\" (UID: \"9e770835-5d2a-4225-8465-88b59b66badb\") " pod="openshift-image-registry/image-registry-86b9797894-fbhdx"
Sep 18 17:16:04.430170 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.430114    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b214c735-b2fd-42a5-9f4e-ee9258c4bfae-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-n6wb8\" (UID: \"b214c735-b2fd-42a5-9f4e-ee9258c4bfae\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-n6wb8"
Sep 18 17:16:04.430170 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.430138    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9e770835-5d2a-4225-8465-88b59b66badb-image-registry-private-configuration\") pod \"image-registry-86b9797894-fbhdx\" (UID: \"9e770835-5d2a-4225-8465-88b59b66badb\") " pod="openshift-image-registry/image-registry-86b9797894-fbhdx"
Sep 18 17:16:04.430255 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.430216    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b214c735-b2fd-42a5-9f4e-ee9258c4bfae-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-n6wb8\" (UID: \"b214c735-b2fd-42a5-9f4e-ee9258c4bfae\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-n6wb8"
Sep 18 17:16:04.430284 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.430250    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cf5a984f-55ce-4fee-b70f-532033699e50-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7b4d9599b-2kxt5\" (UID: \"cf5a984f-55ce-4fee-b70f-532033699e50\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4d9599b-2kxt5"
Sep 18 17:16:04.432652 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.432631    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cf5a984f-55ce-4fee-b70f-532033699e50-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7b4d9599b-2kxt5\" (UID: \"cf5a984f-55ce-4fee-b70f-532033699e50\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4d9599b-2kxt5"
Sep 18 17:16:04.448625 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.448599    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnc6d\" (UniqueName: \"kubernetes.io/projected/cf5a984f-55ce-4fee-b70f-532033699e50-kube-api-access-fnc6d\") pod \"managed-serviceaccount-addon-agent-7b4d9599b-2kxt5\" (UID: \"cf5a984f-55ce-4fee-b70f-532033699e50\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4d9599b-2kxt5"
Sep 18 17:16:04.531340 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.531311    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e770835-5d2a-4225-8465-88b59b66badb-bound-sa-token\") pod \"image-registry-86b9797894-fbhdx\" (UID: \"9e770835-5d2a-4225-8465-88b59b66badb\") " pod="openshift-image-registry/image-registry-86b9797894-fbhdx"
Sep 18 17:16:04.531479 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.531348    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9e770835-5d2a-4225-8465-88b59b66badb-registry-tls\") pod \"image-registry-86b9797894-fbhdx\" (UID: \"9e770835-5d2a-4225-8465-88b59b66badb\") " pod="openshift-image-registry/image-registry-86b9797894-fbhdx"
Sep 18 17:16:04.531479 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.531389    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9e770835-5d2a-4225-8465-88b59b66badb-registry-certificates\") pod \"image-registry-86b9797894-fbhdx\" (UID: \"9e770835-5d2a-4225-8465-88b59b66badb\") " pod="openshift-image-registry/image-registry-86b9797894-fbhdx"
Sep 18 17:16:04.531479 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.531415    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e770835-5d2a-4225-8465-88b59b66badb-trusted-ca\") pod \"image-registry-86b9797894-fbhdx\" (UID: \"9e770835-5d2a-4225-8465-88b59b66badb\") " pod="openshift-image-registry/image-registry-86b9797894-fbhdx"
Sep 18 17:16:04.531479 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.531441    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b214c735-b2fd-42a5-9f4e-ee9258c4bfae-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-n6wb8\" (UID: \"b214c735-b2fd-42a5-9f4e-ee9258c4bfae\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-n6wb8"
Sep 18 17:16:04.531479 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.531473    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9e770835-5d2a-4225-8465-88b59b66badb-image-registry-private-configuration\") pod \"image-registry-86b9797894-fbhdx\" (UID: \"9e770835-5d2a-4225-8465-88b59b66badb\") " pod="openshift-image-registry/image-registry-86b9797894-fbhdx"
Sep 18 17:16:04.531728 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.531536    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b214c735-b2fd-42a5-9f4e-ee9258c4bfae-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-n6wb8\" (UID: \"b214c735-b2fd-42a5-9f4e-ee9258c4bfae\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-n6wb8"
Sep 18 17:16:04.531728 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.531568    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9e770835-5d2a-4225-8465-88b59b66badb-installation-pull-secrets\") pod \"image-registry-86b9797894-fbhdx\" (UID: \"9e770835-5d2a-4225-8465-88b59b66badb\") " pod="openshift-image-registry/image-registry-86b9797894-fbhdx"
Sep 18 17:16:04.531728 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.531616    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9e770835-5d2a-4225-8465-88b59b66badb-ca-trust-extracted\") pod \"image-registry-86b9797894-fbhdx\" (UID: \"9e770835-5d2a-4225-8465-88b59b66badb\") " pod="openshift-image-registry/image-registry-86b9797894-fbhdx"
Sep 18 17:16:04.531728 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.531639    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p78ts\" (UniqueName: \"kubernetes.io/projected/9e770835-5d2a-4225-8465-88b59b66badb-kube-api-access-p78ts\") pod \"image-registry-86b9797894-fbhdx\" (UID: \"9e770835-5d2a-4225-8465-88b59b66badb\") " pod="openshift-image-registry/image-registry-86b9797894-fbhdx"
Sep 18 17:16:04.532250 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.532203    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9e770835-5d2a-4225-8465-88b59b66badb-ca-trust-extracted\") pod \"image-registry-86b9797894-fbhdx\" (UID: \"9e770835-5d2a-4225-8465-88b59b66badb\") " pod="openshift-image-registry/image-registry-86b9797894-fbhdx"
Sep 18 17:16:04.532362 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.532289    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b214c735-b2fd-42a5-9f4e-ee9258c4bfae-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-n6wb8\" (UID: \"b214c735-b2fd-42a5-9f4e-ee9258c4bfae\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-n6wb8"
Sep 18 17:16:04.532362 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.532332    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9e770835-5d2a-4225-8465-88b59b66badb-registry-certificates\") pod \"image-registry-86b9797894-fbhdx\" (UID: \"9e770835-5d2a-4225-8465-88b59b66badb\") " pod="openshift-image-registry/image-registry-86b9797894-fbhdx"
Sep 18 17:16:04.532476 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.532435    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e770835-5d2a-4225-8465-88b59b66badb-trusted-ca\") pod \"image-registry-86b9797894-fbhdx\" (UID: \"9e770835-5d2a-4225-8465-88b59b66badb\") " pod="openshift-image-registry/image-registry-86b9797894-fbhdx"
Sep 18 17:16:04.534043 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.534023    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9e770835-5d2a-4225-8465-88b59b66badb-registry-tls\") pod \"image-registry-86b9797894-fbhdx\" (UID: \"9e770835-5d2a-4225-8465-88b59b66badb\") " pod="openshift-image-registry/image-registry-86b9797894-fbhdx"
Sep 18 17:16:04.534158 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.534106    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9e770835-5d2a-4225-8465-88b59b66badb-image-registry-private-configuration\") pod \"image-registry-86b9797894-fbhdx\" (UID: \"9e770835-5d2a-4225-8465-88b59b66badb\") " pod="openshift-image-registry/image-registry-86b9797894-fbhdx"
Sep 18 17:16:04.534224 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.534208    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b214c735-b2fd-42a5-9f4e-ee9258c4bfae-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-n6wb8\" (UID: \"b214c735-b2fd-42a5-9f4e-ee9258c4bfae\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-n6wb8"
Sep 18 17:16:04.534280 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.534253    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9e770835-5d2a-4225-8465-88b59b66badb-installation-pull-secrets\") pod \"image-registry-86b9797894-fbhdx\" (UID: \"9e770835-5d2a-4225-8465-88b59b66badb\") " pod="openshift-image-registry/image-registry-86b9797894-fbhdx"
Sep 18 17:16:04.541357 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.541334    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e770835-5d2a-4225-8465-88b59b66badb-bound-sa-token\") pod \"image-registry-86b9797894-fbhdx\" (UID: \"9e770835-5d2a-4225-8465-88b59b66badb\") " pod="openshift-image-registry/image-registry-86b9797894-fbhdx"
Sep 18 17:16:04.541788 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.541771    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p78ts\" (UniqueName: \"kubernetes.io/projected/9e770835-5d2a-4225-8465-88b59b66badb-kube-api-access-p78ts\") pod \"image-registry-86b9797894-fbhdx\" (UID: \"9e770835-5d2a-4225-8465-88b59b66badb\") " pod="openshift-image-registry/image-registry-86b9797894-fbhdx"
Sep 18 17:16:04.551532 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.551511    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4d9599b-2kxt5"
Sep 18 17:16:04.616709 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.616605    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-n6wb8"
Sep 18 17:16:04.632122 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.632020    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86b9797894-fbhdx"
Sep 18 17:16:04.818422 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.818349    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4d9599b-2kxt5"]
Sep 18 17:16:04.821757 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:16:04.821731    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf5a984f_55ce_4fee_b70f_532033699e50.slice/crio-632761022c92bc4b6786b3d616ee4658df0adf400960d003e17ec1692965f032 WatchSource:0}: Error finding container 632761022c92bc4b6786b3d616ee4658df0adf400960d003e17ec1692965f032: Status 404 returned error can't find the container with id 632761022c92bc4b6786b3d616ee4658df0adf400960d003e17ec1692965f032
Sep 18 17:16:04.892297 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.892266    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-n6wb8"]
Sep 18 17:16:04.895332 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:16:04.895301    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb214c735_b2fd_42a5_9f4e_ee9258c4bfae.slice/crio-234460d00ec65f93e9fcf6799de0a8cbc4a1a64539de7a244ad41d9feb555c06 WatchSource:0}: Error finding container 234460d00ec65f93e9fcf6799de0a8cbc4a1a64539de7a244ad41d9feb555c06: Status 404 returned error can't find the container with id 234460d00ec65f93e9fcf6799de0a8cbc4a1a64539de7a244ad41d9feb555c06
Sep 18 17:16:04.916348 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:04.916322    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86b9797894-fbhdx"]
Sep 18 17:16:04.919343 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:16:04.919316    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e770835_5d2a_4225_8465_88b59b66badb.slice/crio-c7b38d12122304c98c4c8b27ad490179d93034e9938d795c0253ef7a0ec9220f WatchSource:0}: Error finding container c7b38d12122304c98c4c8b27ad490179d93034e9938d795c0253ef7a0ec9220f: Status 404 returned error can't find the container with id c7b38d12122304c98c4c8b27ad490179d93034e9938d795c0253ef7a0ec9220f
Sep 18 17:16:05.139165 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:05.139074    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4d9599b-2kxt5" event={"ID":"cf5a984f-55ce-4fee-b70f-532033699e50","Type":"ContainerStarted","Data":"632761022c92bc4b6786b3d616ee4658df0adf400960d003e17ec1692965f032"}
Sep 18 17:16:05.140400 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:05.140371    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-n6wb8" event={"ID":"b214c735-b2fd-42a5-9f4e-ee9258c4bfae","Type":"ContainerStarted","Data":"234460d00ec65f93e9fcf6799de0a8cbc4a1a64539de7a244ad41d9feb555c06"}
Sep 18 17:16:05.141817 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:05.141793    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86b9797894-fbhdx" event={"ID":"9e770835-5d2a-4225-8465-88b59b66badb","Type":"ContainerStarted","Data":"6561bdfe40d517c6121dc66b29d0595b72c59570b2757b5d6fa9fcaf553bf717"}
Sep 18 17:16:05.141946 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:05.141834    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86b9797894-fbhdx" event={"ID":"9e770835-5d2a-4225-8465-88b59b66badb","Type":"ContainerStarted","Data":"c7b38d12122304c98c4c8b27ad490179d93034e9938d795c0253ef7a0ec9220f"}
Sep 18 17:16:05.141946 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:05.141923    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-86b9797894-fbhdx"
Sep 18 17:16:05.163680 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:05.163527    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-86b9797894-fbhdx" podStartSLOduration=1.16350076 podStartE2EDuration="1.16350076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:16:05.16321977 +0000 UTC m=+77.857147330" watchObservedRunningTime="2026-09-18 17:16:05.16350076 +0000 UTC m=+77.857428285"
Sep 18 17:16:06.448289 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:06.448188    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63db05d0-62bd-4fe6-999f-0d19ef2ff266-metrics-certs\") pod \"router-default-76465d565b-r2rdc\" (UID: \"63db05d0-62bd-4fe6-999f-0d19ef2ff266\") " pod="openshift-ingress/router-default-76465d565b-r2rdc"
Sep 18 17:16:06.448736 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:06.448301    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63db05d0-62bd-4fe6-999f-0d19ef2ff266-service-ca-bundle\") pod \"router-default-76465d565b-r2rdc\" (UID: \"63db05d0-62bd-4fe6-999f-0d19ef2ff266\") " pod="openshift-ingress/router-default-76465d565b-r2rdc"
Sep 18 17:16:06.448971 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:06.448950    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63db05d0-62bd-4fe6-999f-0d19ef2ff266-service-ca-bundle\") pod \"router-default-76465d565b-r2rdc\" (UID: \"63db05d0-62bd-4fe6-999f-0d19ef2ff266\") " pod="openshift-ingress/router-default-76465d565b-r2rdc"
Sep 18 17:16:06.450842 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:06.450810    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63db05d0-62bd-4fe6-999f-0d19ef2ff266-metrics-certs\") pod \"router-default-76465d565b-r2rdc\" (UID: \"63db05d0-62bd-4fe6-999f-0d19ef2ff266\") " pod="openshift-ingress/router-default-76465d565b-r2rdc"
Sep 18 17:16:06.630028 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:06.629998    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-tmvvs\""
Sep 18 17:16:06.638340 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:06.638319    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-76465d565b-r2rdc"
Sep 18 17:16:06.912585 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:06.912559    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-76465d565b-r2rdc"]
Sep 18 17:16:06.915452 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:16:06.915420    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63db05d0_62bd_4fe6_999f_0d19ef2ff266.slice/crio-2d6864cf8f003758d07a0e08d0adbf6c96b4a27d5b8bf66c724ff0d50b9ef2e8 WatchSource:0}: Error finding container 2d6864cf8f003758d07a0e08d0adbf6c96b4a27d5b8bf66c724ff0d50b9ef2e8: Status 404 returned error can't find the container with id 2d6864cf8f003758d07a0e08d0adbf6c96b4a27d5b8bf66c724ff0d50b9ef2e8
Sep 18 17:16:07.153125 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:07.152911    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-76465d565b-r2rdc" event={"ID":"63db05d0-62bd-4fe6-999f-0d19ef2ff266","Type":"ContainerStarted","Data":"66cc81a408c14e5d2e0fcf5bfe2145114435295f6e1fb4ae9a94fdea955e7ab9"}
Sep 18 17:16:07.153125 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:07.152974    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-76465d565b-r2rdc" event={"ID":"63db05d0-62bd-4fe6-999f-0d19ef2ff266","Type":"ContainerStarted","Data":"2d6864cf8f003758d07a0e08d0adbf6c96b4a27d5b8bf66c724ff0d50b9ef2e8"}
Sep 18 17:16:07.154721 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:07.154676    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-n6wb8" event={"ID":"b214c735-b2fd-42a5-9f4e-ee9258c4bfae","Type":"ContainerStarted","Data":"83bb007bfb204abf10f9ea03bd7e1b5980eb3c4b23ee51a02e6c65c134c2ed80"}
Sep 18 17:16:07.227709 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:07.227608    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-76465d565b-r2rdc" podStartSLOduration=33.227585688 podStartE2EDuration="33.227585688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:16:07.227169906 +0000 UTC m=+79.921097455" watchObservedRunningTime="2026-09-18 17:16:07.227585688 +0000 UTC m=+79.921513217"
Sep 18 17:16:07.244191 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:07.244138    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-n6wb8" podStartSLOduration=1.956262658 podStartE2EDuration="3.244116739s" totalImagesPullingTime="1.287854081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:16:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:16:07.24383884 +0000 UTC m=+79.937766395" watchObservedRunningTime="2026-09-18 17:16:07.244116739 +0000 UTC m=+79.938044267"
Sep 18 17:16:07.638783 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:07.638739    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-76465d565b-r2rdc"
Sep 18 17:16:07.641280 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:07.641256    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-76465d565b-r2rdc"
Sep 18 17:16:08.159726 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:08.159691    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4d9599b-2kxt5" event={"ID":"cf5a984f-55ce-4fee-b70f-532033699e50","Type":"ContainerStarted","Data":"6eedc5c7635cb755e922259ce9fb092766f5c0544915828620e8bcde0aa2336b"}
Sep 18 17:16:08.160010 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:08.159983    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-76465d565b-r2rdc"
Sep 18 17:16:08.161019 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:08.161001    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-76465d565b-r2rdc"
Sep 18 17:16:08.178406 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:08.178363    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4d9599b-2kxt5" podStartSLOduration=0.96344414 podStartE2EDuration="4.178349837s" totalImagesPullingTime="3.214905697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:16:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:16:08.17750679 +0000 UTC m=+80.871434318" watchObservedRunningTime="2026-09-18 17:16:08.178349837 +0000 UTC m=+80.872277364"
Sep 18 17:16:10.127576 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:10.127446    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-s566m"
Sep 18 17:16:10.883517 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:10.883480    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08f50d0f-2329-428f-ab3e-64159a5bfe35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5lm2g\" (UID: \"08f50d0f-2329-428f-ab3e-64159a5bfe35\") " pod="openshift-insights/insights-runtime-extractor-5lm2g"
Sep 18 17:16:10.885725 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:10.885703    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08f50d0f-2329-428f-ab3e-64159a5bfe35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5lm2g\" (UID: \"08f50d0f-2329-428f-ab3e-64159a5bfe35\") " pod="openshift-insights/insights-runtime-extractor-5lm2g"
Sep 18 17:16:11.149090 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:11.149010    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2sl55\""
Sep 18 17:16:11.157092 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:11.157069    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5lm2g"
Sep 18 17:16:11.429496 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:11.429470    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5lm2g"]
Sep 18 17:16:11.431765 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:16:11.431743    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08f50d0f_2329_428f_ab3e_64159a5bfe35.slice/crio-7d8b933db4718f55f680e3c3f79b08d6c18a59efa2f67a8ca2986b4e3f05b853 WatchSource:0}: Error finding container 7d8b933db4718f55f680e3c3f79b08d6c18a59efa2f67a8ca2986b4e3f05b853: Status 404 returned error can't find the container with id 7d8b933db4718f55f680e3c3f79b08d6c18a59efa2f67a8ca2986b4e3f05b853
Sep 18 17:16:12.174394 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:12.174354    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5lm2g" event={"ID":"08f50d0f-2329-428f-ab3e-64159a5bfe35","Type":"ContainerStarted","Data":"23cf470393143979558c5e4ef6697997ad7a887912609d489d92b25c8fd9ffe8"}
Sep 18 17:16:12.174752 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:12.174435    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5lm2g" event={"ID":"08f50d0f-2329-428f-ab3e-64159a5bfe35","Type":"ContainerStarted","Data":"f76ffd03f89821354b18642ddadd3622b1d9554a47d265bf1a0d16c87bd08048"}
Sep 18 17:16:12.174752 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:12.174453    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5lm2g" event={"ID":"08f50d0f-2329-428f-ab3e-64159a5bfe35","Type":"ContainerStarted","Data":"7d8b933db4718f55f680e3c3f79b08d6c18a59efa2f67a8ca2986b4e3f05b853"}
Sep 18 17:16:14.182222 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:14.182189    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5lm2g" event={"ID":"08f50d0f-2329-428f-ab3e-64159a5bfe35","Type":"ContainerStarted","Data":"1a501ae0d88e5387649a235a6aa51a1f7ddc179ea3d1e69480f84f674774b638"}
Sep 18 17:16:14.204103 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:14.204063    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5lm2g" podStartSLOduration=34.502122433 podStartE2EDuration="36.204047384s" totalImagesPullingTime="1.701924951s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:15:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:16:14.203296489 +0000 UTC m=+86.897224016" watchObservedRunningTime="2026-09-18 17:16:14.204047384 +0000 UTC m=+86.897974890"
Sep 18 17:16:18.006152 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.006119    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84d7d579f4-7ht29"]
Sep 18 17:16:18.009674 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.009657    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d7d579f4-7ht29"
Sep 18 17:16:18.012359 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.012336    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 17:16:18.013050 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.013019    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 17:16:18.013148 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.013019    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 17:16:18.013148 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.013068    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 17:16:18.013148 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.013054    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 17:16:18.013539 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.013506    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 17:16:18.013539 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.013518    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 17:16:18.013699 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.013564    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kp6ft\""
Sep 18 17:16:18.017412 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.017389    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 18 17:16:18.021180 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.021159    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84d7d579f4-7ht29"]
Sep 18 17:16:18.139151 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.139115    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2429234-6e97-4b9c-8528-bb0463f8da5c-trusted-ca-bundle\") pod \"console-84d7d579f4-7ht29\" (UID: \"c2429234-6e97-4b9c-8528-bb0463f8da5c\") " pod="openshift-console/console-84d7d579f4-7ht29"
Sep 18 17:16:18.139341 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.139183    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c2429234-6e97-4b9c-8528-bb0463f8da5c-oauth-serving-cert\") pod \"console-84d7d579f4-7ht29\" (UID: \"c2429234-6e97-4b9c-8528-bb0463f8da5c\") " pod="openshift-console/console-84d7d579f4-7ht29"
Sep 18 17:16:18.139341 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.139250    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt5ps\" (UniqueName: \"kubernetes.io/projected/c2429234-6e97-4b9c-8528-bb0463f8da5c-kube-api-access-tt5ps\") pod \"console-84d7d579f4-7ht29\" (UID: \"c2429234-6e97-4b9c-8528-bb0463f8da5c\") " pod="openshift-console/console-84d7d579f4-7ht29"
Sep 18 17:16:18.139341 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.139302    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c2429234-6e97-4b9c-8528-bb0463f8da5c-service-ca\") pod \"console-84d7d579f4-7ht29\" (UID: \"c2429234-6e97-4b9c-8528-bb0463f8da5c\") " pod="openshift-console/console-84d7d579f4-7ht29"
Sep 18 17:16:18.139341 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.139334    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c2429234-6e97-4b9c-8528-bb0463f8da5c-console-serving-cert\") pod \"console-84d7d579f4-7ht29\" (UID: \"c2429234-6e97-4b9c-8528-bb0463f8da5c\") " pod="openshift-console/console-84d7d579f4-7ht29"
Sep 18 17:16:18.139520 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.139370    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c2429234-6e97-4b9c-8528-bb0463f8da5c-console-oauth-config\") pod \"console-84d7d579f4-7ht29\" (UID: \"c2429234-6e97-4b9c-8528-bb0463f8da5c\") " pod="openshift-console/console-84d7d579f4-7ht29"
Sep 18 17:16:18.139520 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.139395    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c2429234-6e97-4b9c-8528-bb0463f8da5c-console-config\") pod \"console-84d7d579f4-7ht29\" (UID: \"c2429234-6e97-4b9c-8528-bb0463f8da5c\") " pod="openshift-console/console-84d7d579f4-7ht29"
Sep 18 17:16:18.240724 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.240683    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c2429234-6e97-4b9c-8528-bb0463f8da5c-oauth-serving-cert\") pod \"console-84d7d579f4-7ht29\" (UID: \"c2429234-6e97-4b9c-8528-bb0463f8da5c\") " pod="openshift-console/console-84d7d579f4-7ht29"
Sep 18 17:16:18.240944 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.240732    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt5ps\" (UniqueName: \"kubernetes.io/projected/c2429234-6e97-4b9c-8528-bb0463f8da5c-kube-api-access-tt5ps\") pod \"console-84d7d579f4-7ht29\" (UID: \"c2429234-6e97-4b9c-8528-bb0463f8da5c\") " pod="openshift-console/console-84d7d579f4-7ht29"
Sep 18 17:16:18.240944 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.240769    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c2429234-6e97-4b9c-8528-bb0463f8da5c-service-ca\") pod \"console-84d7d579f4-7ht29\" (UID: \"c2429234-6e97-4b9c-8528-bb0463f8da5c\") " pod="openshift-console/console-84d7d579f4-7ht29"
Sep 18 17:16:18.240944 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.240796    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c2429234-6e97-4b9c-8528-bb0463f8da5c-console-serving-cert\") pod \"console-84d7d579f4-7ht29\" (UID: \"c2429234-6e97-4b9c-8528-bb0463f8da5c\") " pod="openshift-console/console-84d7d579f4-7ht29"
Sep 18 17:16:18.240944 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.240827    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c2429234-6e97-4b9c-8528-bb0463f8da5c-console-oauth-config\") pod \"console-84d7d579f4-7ht29\" (UID: \"c2429234-6e97-4b9c-8528-bb0463f8da5c\") " pod="openshift-console/console-84d7d579f4-7ht29"
Sep 18 17:16:18.240944 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.240848    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c2429234-6e97-4b9c-8528-bb0463f8da5c-console-config\") pod \"console-84d7d579f4-7ht29\" (UID: \"c2429234-6e97-4b9c-8528-bb0463f8da5c\") " pod="openshift-console/console-84d7d579f4-7ht29"
Sep 18 17:16:18.241196 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.240998    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2429234-6e97-4b9c-8528-bb0463f8da5c-trusted-ca-bundle\") pod \"console-84d7d579f4-7ht29\" (UID: \"c2429234-6e97-4b9c-8528-bb0463f8da5c\") " pod="openshift-console/console-84d7d579f4-7ht29"
Sep 18 17:16:18.241561 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.241537    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c2429234-6e97-4b9c-8528-bb0463f8da5c-service-ca\") pod \"console-84d7d579f4-7ht29\" (UID: \"c2429234-6e97-4b9c-8528-bb0463f8da5c\") " pod="openshift-console/console-84d7d579f4-7ht29"
Sep 18 17:16:18.241653 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.241576    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c2429234-6e97-4b9c-8528-bb0463f8da5c-oauth-serving-cert\") pod \"console-84d7d579f4-7ht29\" (UID: \"c2429234-6e97-4b9c-8528-bb0463f8da5c\") " pod="openshift-console/console-84d7d579f4-7ht29"
Sep 18 17:16:18.241782 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.241762    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2429234-6e97-4b9c-8528-bb0463f8da5c-trusted-ca-bundle\") pod \"console-84d7d579f4-7ht29\" (UID: \"c2429234-6e97-4b9c-8528-bb0463f8da5c\") " pod="openshift-console/console-84d7d579f4-7ht29"
Sep 18 17:16:18.241819 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.241786    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c2429234-6e97-4b9c-8528-bb0463f8da5c-console-config\") pod \"console-84d7d579f4-7ht29\" (UID: \"c2429234-6e97-4b9c-8528-bb0463f8da5c\") " pod="openshift-console/console-84d7d579f4-7ht29"
Sep 18 17:16:18.243369 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.243341    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c2429234-6e97-4b9c-8528-bb0463f8da5c-console-oauth-config\") pod \"console-84d7d579f4-7ht29\" (UID: \"c2429234-6e97-4b9c-8528-bb0463f8da5c\") " pod="openshift-console/console-84d7d579f4-7ht29"
Sep 18 17:16:18.243444 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.243368    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c2429234-6e97-4b9c-8528-bb0463f8da5c-console-serving-cert\") pod \"console-84d7d579f4-7ht29\" (UID: \"c2429234-6e97-4b9c-8528-bb0463f8da5c\") " pod="openshift-console/console-84d7d579f4-7ht29"
Sep 18 17:16:18.249859 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.249839    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt5ps\" (UniqueName: \"kubernetes.io/projected/c2429234-6e97-4b9c-8528-bb0463f8da5c-kube-api-access-tt5ps\") pod \"console-84d7d579f4-7ht29\" (UID: \"c2429234-6e97-4b9c-8528-bb0463f8da5c\") " pod="openshift-console/console-84d7d579f4-7ht29"
Sep 18 17:16:18.319476 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.319404    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d7d579f4-7ht29"
Sep 18 17:16:18.591184 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:18.591103    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84d7d579f4-7ht29"]
Sep 18 17:16:18.593772 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:16:18.593743    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2429234_6e97_4b9c_8528_bb0463f8da5c.slice/crio-b86e7c942dedb3e87cae556db7822c91082783f43f786989eb721baf612a3554 WatchSource:0}: Error finding container b86e7c942dedb3e87cae556db7822c91082783f43f786989eb721baf612a3554: Status 404 returned error can't find the container with id b86e7c942dedb3e87cae556db7822c91082783f43f786989eb721baf612a3554
Sep 18 17:16:19.202279 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:19.202237    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d7d579f4-7ht29" event={"ID":"c2429234-6e97-4b9c-8528-bb0463f8da5c","Type":"ContainerStarted","Data":"b86e7c942dedb3e87cae556db7822c91082783f43f786989eb721baf612a3554"}
Sep 18 17:16:23.220690 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:23.220639    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d7d579f4-7ht29" event={"ID":"c2429234-6e97-4b9c-8528-bb0463f8da5c","Type":"ContainerStarted","Data":"33ec74ceaa70799c5bd81070da923d83754f8e05727f66e920e3a1677cc77d97"}
Sep 18 17:16:23.239359 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:23.239285    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84d7d579f4-7ht29" podStartSLOduration=2.437092751 podStartE2EDuration="6.23926894s" totalImagesPullingTime="3.802176189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:16:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:16:23.23890836 +0000 UTC m=+95.932835885" watchObservedRunningTime="2026-09-18 17:16:23.23926894 +0000 UTC m=+95.933196469"
Sep 18 17:16:26.151789 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:26.151758    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-86b9797894-fbhdx"
Sep 18 17:16:28.320175 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:28.320120    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84d7d579f4-7ht29"
Sep 18 17:16:28.320613 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:28.320199    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84d7d579f4-7ht29"
Sep 18 17:16:28.325189 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:28.325165    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84d7d579f4-7ht29"
Sep 18 17:16:29.245644 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:29.245618    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84d7d579f4-7ht29"
Sep 18 17:16:30.492273 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.492238    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9tw2m"]
Sep 18 17:16:30.495037 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.495015    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9tw2m"
Sep 18 17:16:30.498538 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.498518    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 17:16:30.498667 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.498519    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 17:16:30.499668 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.499652    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 17:16:30.499761 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.499676    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 17:16:30.499761 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.499723    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 17:16:30.499761 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.499730    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jmw5s\""
Sep 18 17:16:30.504459 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.504444    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 17:16:30.643460 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.643421    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0d81ebf0-5191-4fe8-820b-f16d136e9044-node-exporter-textfile\") pod \"node-exporter-9tw2m\" (UID: \"0d81ebf0-5191-4fe8-820b-f16d136e9044\") " pod="openshift-monitoring/node-exporter-9tw2m"
Sep 18 17:16:30.643616 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.643468    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0d81ebf0-5191-4fe8-820b-f16d136e9044-node-exporter-tls\") pod \"node-exporter-9tw2m\" (UID: \"0d81ebf0-5191-4fe8-820b-f16d136e9044\") " pod="openshift-monitoring/node-exporter-9tw2m"
Sep 18 17:16:30.643616 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.643495    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0d81ebf0-5191-4fe8-820b-f16d136e9044-node-exporter-wtmp\") pod \"node-exporter-9tw2m\" (UID: \"0d81ebf0-5191-4fe8-820b-f16d136e9044\") " pod="openshift-monitoring/node-exporter-9tw2m"
Sep 18 17:16:30.643616 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.643584    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfcjf\" (UniqueName: \"kubernetes.io/projected/0d81ebf0-5191-4fe8-820b-f16d136e9044-kube-api-access-hfcjf\") pod \"node-exporter-9tw2m\" (UID: \"0d81ebf0-5191-4fe8-820b-f16d136e9044\") " pod="openshift-monitoring/node-exporter-9tw2m"
Sep 18 17:16:30.643752 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.643628    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0d81ebf0-5191-4fe8-820b-f16d136e9044-root\") pod \"node-exporter-9tw2m\" (UID: \"0d81ebf0-5191-4fe8-820b-f16d136e9044\") " pod="openshift-monitoring/node-exporter-9tw2m"
Sep 18 17:16:30.643752 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.643659    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d81ebf0-5191-4fe8-820b-f16d136e9044-metrics-client-ca\") pod \"node-exporter-9tw2m\" (UID: \"0d81ebf0-5191-4fe8-820b-f16d136e9044\") " pod="openshift-monitoring/node-exporter-9tw2m"
Sep 18 17:16:30.643752 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.643676    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d81ebf0-5191-4fe8-820b-f16d136e9044-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9tw2m\" (UID: \"0d81ebf0-5191-4fe8-820b-f16d136e9044\") " pod="openshift-monitoring/node-exporter-9tw2m"
Sep 18 17:16:30.643752 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.643696    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d81ebf0-5191-4fe8-820b-f16d136e9044-sys\") pod \"node-exporter-9tw2m\" (UID: \"0d81ebf0-5191-4fe8-820b-f16d136e9044\") " pod="openshift-monitoring/node-exporter-9tw2m"
Sep 18 17:16:30.643752 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.643713    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0d81ebf0-5191-4fe8-820b-f16d136e9044-node-exporter-accelerators-collector-config\") pod \"node-exporter-9tw2m\" (UID: \"0d81ebf0-5191-4fe8-820b-f16d136e9044\") " pod="openshift-monitoring/node-exporter-9tw2m"
Sep 18 17:16:30.744462 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.744377    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0d81ebf0-5191-4fe8-820b-f16d136e9044-node-exporter-tls\") pod \"node-exporter-9tw2m\" (UID: \"0d81ebf0-5191-4fe8-820b-f16d136e9044\") " pod="openshift-monitoring/node-exporter-9tw2m"
Sep 18 17:16:30.744462 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.744414    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0d81ebf0-5191-4fe8-820b-f16d136e9044-node-exporter-wtmp\") pod \"node-exporter-9tw2m\" (UID: \"0d81ebf0-5191-4fe8-820b-f16d136e9044\") " pod="openshift-monitoring/node-exporter-9tw2m"
Sep 18 17:16:30.744462 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.744438    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfcjf\" (UniqueName: \"kubernetes.io/projected/0d81ebf0-5191-4fe8-820b-f16d136e9044-kube-api-access-hfcjf\") pod \"node-exporter-9tw2m\" (UID: \"0d81ebf0-5191-4fe8-820b-f16d136e9044\") " pod="openshift-monitoring/node-exporter-9tw2m"
Sep 18 17:16:30.744462 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.744461    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0d81ebf0-5191-4fe8-820b-f16d136e9044-root\") pod \"node-exporter-9tw2m\" (UID: \"0d81ebf0-5191-4fe8-820b-f16d136e9044\") " pod="openshift-monitoring/node-exporter-9tw2m"
Sep 18 17:16:30.744754 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.744488    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d81ebf0-5191-4fe8-820b-f16d136e9044-metrics-client-ca\") pod \"node-exporter-9tw2m\" (UID: \"0d81ebf0-5191-4fe8-820b-f16d136e9044\") " pod="openshift-monitoring/node-exporter-9tw2m"
Sep 18 17:16:30.744754 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.744505    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d81ebf0-5191-4fe8-820b-f16d136e9044-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9tw2m\" (UID: \"0d81ebf0-5191-4fe8-820b-f16d136e9044\") " pod="openshift-monitoring/node-exporter-9tw2m"
Sep 18 17:16:30.744754 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.744523    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d81ebf0-5191-4fe8-820b-f16d136e9044-sys\") pod \"node-exporter-9tw2m\" (UID: \"0d81ebf0-5191-4fe8-820b-f16d136e9044\") " pod="openshift-monitoring/node-exporter-9tw2m"
Sep 18 17:16:30.744754 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.744545    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0d81ebf0-5191-4fe8-820b-f16d136e9044-node-exporter-accelerators-collector-config\") pod \"node-exporter-9tw2m\" (UID: \"0d81ebf0-5191-4fe8-820b-f16d136e9044\") " pod="openshift-monitoring/node-exporter-9tw2m"
Sep 18 17:16:30.744754 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.744576    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0d81ebf0-5191-4fe8-820b-f16d136e9044-root\") pod \"node-exporter-9tw2m\" (UID: \"0d81ebf0-5191-4fe8-820b-f16d136e9044\") " pod="openshift-monitoring/node-exporter-9tw2m"
Sep 18 17:16:30.744754 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.744576    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0d81ebf0-5191-4fe8-820b-f16d136e9044-node-exporter-wtmp\") pod \"node-exporter-9tw2m\" (UID: \"0d81ebf0-5191-4fe8-820b-f16d136e9044\") " pod="openshift-monitoring/node-exporter-9tw2m"
Sep 18 17:16:30.744754 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.744635    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d81ebf0-5191-4fe8-820b-f16d136e9044-sys\") pod \"node-exporter-9tw2m\" (UID: \"0d81ebf0-5191-4fe8-820b-f16d136e9044\") " pod="openshift-monitoring/node-exporter-9tw2m"
Sep 18 17:16:30.744754 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.744683    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0d81ebf0-5191-4fe8-820b-f16d136e9044-node-exporter-textfile\") pod \"node-exporter-9tw2m\" (UID: \"0d81ebf0-5191-4fe8-820b-f16d136e9044\") " pod="openshift-monitoring/node-exporter-9tw2m"
Sep 18 17:16:30.745134 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.745007    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0d81ebf0-5191-4fe8-820b-f16d136e9044-node-exporter-textfile\") pod \"node-exporter-9tw2m\" (UID: \"0d81ebf0-5191-4fe8-820b-f16d136e9044\") " pod="openshift-monitoring/node-exporter-9tw2m"
Sep 18 17:16:30.745225 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.745190    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d81ebf0-5191-4fe8-820b-f16d136e9044-metrics-client-ca\") pod \"node-exporter-9tw2m\" (UID: \"0d81ebf0-5191-4fe8-820b-f16d136e9044\") " pod="openshift-monitoring/node-exporter-9tw2m"
Sep 18 17:16:30.745368 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.745347    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0d81ebf0-5191-4fe8-820b-f16d136e9044-node-exporter-accelerators-collector-config\") pod \"node-exporter-9tw2m\" (UID: \"0d81ebf0-5191-4fe8-820b-f16d136e9044\") " pod="openshift-monitoring/node-exporter-9tw2m"
Sep 18 17:16:30.746831 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.746796    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0d81ebf0-5191-4fe8-820b-f16d136e9044-node-exporter-tls\") pod \"node-exporter-9tw2m\" (UID: \"0d81ebf0-5191-4fe8-820b-f16d136e9044\") " pod="openshift-monitoring/node-exporter-9tw2m"
Sep 18 17:16:30.746958 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.746801    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d81ebf0-5191-4fe8-820b-f16d136e9044-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9tw2m\" (UID: \"0d81ebf0-5191-4fe8-820b-f16d136e9044\") " pod="openshift-monitoring/node-exporter-9tw2m"
Sep 18 17:16:30.757812 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.757791    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfcjf\" (UniqueName: \"kubernetes.io/projected/0d81ebf0-5191-4fe8-820b-f16d136e9044-kube-api-access-hfcjf\") pod \"node-exporter-9tw2m\" (UID: \"0d81ebf0-5191-4fe8-820b-f16d136e9044\") " pod="openshift-monitoring/node-exporter-9tw2m"
Sep 18 17:16:30.804913 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:30.804882    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9tw2m"
Sep 18 17:16:30.814987 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:16:30.814961    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d81ebf0_5191_4fe8_820b_f16d136e9044.slice/crio-e32b1da7e4a992064856c8a6de9e10c5b9b9a128e598ce731f30c284c00722d7 WatchSource:0}: Error finding container e32b1da7e4a992064856c8a6de9e10c5b9b9a128e598ce731f30c284c00722d7: Status 404 returned error can't find the container with id e32b1da7e4a992064856c8a6de9e10c5b9b9a128e598ce731f30c284c00722d7
Sep 18 17:16:31.128252 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:31.128174    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-l9knh"
Sep 18 17:16:31.249164 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:31.249135    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9tw2m" event={"ID":"0d81ebf0-5191-4fe8-820b-f16d136e9044","Type":"ContainerStarted","Data":"e32b1da7e4a992064856c8a6de9e10c5b9b9a128e598ce731f30c284c00722d7"}
Sep 18 17:16:32.253718 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:32.253686    2569 generic.go:356] "Generic (PLEG): container finished" podID="0d81ebf0-5191-4fe8-820b-f16d136e9044" containerID="55f1f0c94270d5dcb3ebddc680dbbf12b5e56e6113a0a879e4314c96c127541b" exitCode=0
Sep 18 17:16:32.254120 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:32.253746    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9tw2m" event={"ID":"0d81ebf0-5191-4fe8-820b-f16d136e9044","Type":"ContainerDied","Data":"55f1f0c94270d5dcb3ebddc680dbbf12b5e56e6113a0a879e4314c96c127541b"}
Sep 18 17:16:33.258835 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:33.258801    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9tw2m" event={"ID":"0d81ebf0-5191-4fe8-820b-f16d136e9044","Type":"ContainerStarted","Data":"47dc9169bb1efd77b62a07e0ef9b82a0a7cc931e45ece71455bbcb486603bcfe"}
Sep 18 17:16:33.259221 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:33.258851    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9tw2m" event={"ID":"0d81ebf0-5191-4fe8-820b-f16d136e9044","Type":"ContainerStarted","Data":"d2758bcdadddbaea8d95095dbfcdaf63c5cb70b1676661f78f6de327a109bd4a"}
Sep 18 17:16:33.294939 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:33.294876    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9tw2m" podStartSLOduration=2.156917396 podStartE2EDuration="3.29484278s" totalImagesPullingTime="1.078283363s" totalInitContainerRuntime="59.642021ms" isStatefulPod=true podCreationTimestamp="2026-09-18 17:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:16:33.293473622 +0000 UTC m=+105.987401151" watchObservedRunningTime="2026-09-18 17:16:33.29484278 +0000 UTC m=+105.988770353"
Sep 18 17:16:35.220169 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.220139    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64cf86bd56-w67gw"]
Sep 18 17:16:35.251994 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.251961    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64cf86bd56-w67gw"]
Sep 18 17:16:35.252135 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.252057    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64cf86bd56-w67gw"
Sep 18 17:16:35.381389 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.381349    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxbjn\" (UniqueName: \"kubernetes.io/projected/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-kube-api-access-lxbjn\") pod \"console-64cf86bd56-w67gw\" (UID: \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\") " pod="openshift-console/console-64cf86bd56-w67gw"
Sep 18 17:16:35.381389 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.381388    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-console-config\") pod \"console-64cf86bd56-w67gw\" (UID: \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\") " pod="openshift-console/console-64cf86bd56-w67gw"
Sep 18 17:16:35.381584 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.381423    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-oauth-serving-cert\") pod \"console-64cf86bd56-w67gw\" (UID: \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\") " pod="openshift-console/console-64cf86bd56-w67gw"
Sep 18 17:16:35.381584 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.381495    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-service-ca\") pod \"console-64cf86bd56-w67gw\" (UID: \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\") " pod="openshift-console/console-64cf86bd56-w67gw"
Sep 18 17:16:35.381584 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.381533    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-console-oauth-config\") pod \"console-64cf86bd56-w67gw\" (UID: \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\") " pod="openshift-console/console-64cf86bd56-w67gw"
Sep 18 17:16:35.381584 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.381559    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-trusted-ca-bundle\") pod \"console-64cf86bd56-w67gw\" (UID: \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\") " pod="openshift-console/console-64cf86bd56-w67gw"
Sep 18 17:16:35.381706 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.381586    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-console-serving-cert\") pod \"console-64cf86bd56-w67gw\" (UID: \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\") " pod="openshift-console/console-64cf86bd56-w67gw"
Sep 18 17:16:35.482383 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.482355    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-oauth-serving-cert\") pod \"console-64cf86bd56-w67gw\" (UID: \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\") " pod="openshift-console/console-64cf86bd56-w67gw"
Sep 18 17:16:35.482528 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.482397    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-service-ca\") pod \"console-64cf86bd56-w67gw\" (UID: \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\") " pod="openshift-console/console-64cf86bd56-w67gw"
Sep 18 17:16:35.482528 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.482417    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-console-oauth-config\") pod \"console-64cf86bd56-w67gw\" (UID: \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\") " pod="openshift-console/console-64cf86bd56-w67gw"
Sep 18 17:16:35.482675 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.482656    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-trusted-ca-bundle\") pod \"console-64cf86bd56-w67gw\" (UID: \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\") " pod="openshift-console/console-64cf86bd56-w67gw"
Sep 18 17:16:35.482712 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.482692    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-console-serving-cert\") pod \"console-64cf86bd56-w67gw\" (UID: \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\") " pod="openshift-console/console-64cf86bd56-w67gw"
Sep 18 17:16:35.482758 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.482747    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxbjn\" (UniqueName: \"kubernetes.io/projected/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-kube-api-access-lxbjn\") pod \"console-64cf86bd56-w67gw\" (UID: \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\") " pod="openshift-console/console-64cf86bd56-w67gw"
Sep 18 17:16:35.482792 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.482767    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-console-config\") pod \"console-64cf86bd56-w67gw\" (UID: \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\") " pod="openshift-console/console-64cf86bd56-w67gw"
Sep 18 17:16:35.483241 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.483214    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-service-ca\") pod \"console-64cf86bd56-w67gw\" (UID: \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\") " pod="openshift-console/console-64cf86bd56-w67gw"
Sep 18 17:16:35.483533 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.483513    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-oauth-serving-cert\") pod \"console-64cf86bd56-w67gw\" (UID: \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\") " pod="openshift-console/console-64cf86bd56-w67gw"
Sep 18 17:16:35.483604 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.483581    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-trusted-ca-bundle\") pod \"console-64cf86bd56-w67gw\" (UID: \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\") " pod="openshift-console/console-64cf86bd56-w67gw"
Sep 18 17:16:35.483686 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.483663    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-console-config\") pod \"console-64cf86bd56-w67gw\" (UID: \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\") " pod="openshift-console/console-64cf86bd56-w67gw"
Sep 18 17:16:35.484911 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.484889    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-console-oauth-config\") pod \"console-64cf86bd56-w67gw\" (UID: \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\") " pod="openshift-console/console-64cf86bd56-w67gw"
Sep 18 17:16:35.485205 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.485186    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-console-serving-cert\") pod \"console-64cf86bd56-w67gw\" (UID: \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\") " pod="openshift-console/console-64cf86bd56-w67gw"
Sep 18 17:16:35.491724 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.491698    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxbjn\" (UniqueName: \"kubernetes.io/projected/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-kube-api-access-lxbjn\") pod \"console-64cf86bd56-w67gw\" (UID: \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\") " pod="openshift-console/console-64cf86bd56-w67gw"
Sep 18 17:16:35.560777 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.560739    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64cf86bd56-w67gw"
Sep 18 17:16:35.691434 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.691403    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5995d9b5fc-hj95d"]
Sep 18 17:16:35.694532 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.694510    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5995d9b5fc-hj95d"
Sep 18 17:16:35.697057 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.697036    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 18 17:16:35.698760 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.698742    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 18 17:16:35.698843 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.698826    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-bvb5n\""
Sep 18 17:16:35.698981 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.698963    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 18 17:16:35.699099 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.699070    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 18 17:16:35.699170 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.699131    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 18 17:16:35.704933 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.704916    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 18 17:16:35.714986 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.714964    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5995d9b5fc-hj95d"]
Sep 18 17:16:35.871191 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.871122    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64cf86bd56-w67gw"]
Sep 18 17:16:35.873964 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:16:35.873936    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd48e36f1_1944_4e2b_a9b8_0f5e0f352428.slice/crio-ddd4f9cb3a9b3349c3e3f42ba289a1ac8ca2a369d24df47e0f1bfe6b8916e00d WatchSource:0}: Error finding container ddd4f9cb3a9b3349c3e3f42ba289a1ac8ca2a369d24df47e0f1bfe6b8916e00d: Status 404 returned error can't find the container with id ddd4f9cb3a9b3349c3e3f42ba289a1ac8ca2a369d24df47e0f1bfe6b8916e00d
Sep 18 17:16:35.885788 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.885762    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c0dd879-ad03-42f8-8d1c-44f9dae55b42-serving-certs-ca-bundle\") pod \"telemeter-client-5995d9b5fc-hj95d\" (UID: \"0c0dd879-ad03-42f8-8d1c-44f9dae55b42\") " pod="openshift-monitoring/telemeter-client-5995d9b5fc-hj95d"
Sep 18 17:16:35.885888 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.885816    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c0dd879-ad03-42f8-8d1c-44f9dae55b42-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5995d9b5fc-hj95d\" (UID: \"0c0dd879-ad03-42f8-8d1c-44f9dae55b42\") " pod="openshift-monitoring/telemeter-client-5995d9b5fc-hj95d"
Sep 18 17:16:35.885888 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.885843    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0c0dd879-ad03-42f8-8d1c-44f9dae55b42-telemeter-client-tls\") pod \"telemeter-client-5995d9b5fc-hj95d\" (UID: \"0c0dd879-ad03-42f8-8d1c-44f9dae55b42\") " pod="openshift-monitoring/telemeter-client-5995d9b5fc-hj95d"
Sep 18 17:16:35.886006 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.885913    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0c0dd879-ad03-42f8-8d1c-44f9dae55b42-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5995d9b5fc-hj95d\" (UID: \"0c0dd879-ad03-42f8-8d1c-44f9dae55b42\") " pod="openshift-monitoring/telemeter-client-5995d9b5fc-hj95d"
Sep 18 17:16:35.886006 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.885956    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c0dd879-ad03-42f8-8d1c-44f9dae55b42-metrics-client-ca\") pod \"telemeter-client-5995d9b5fc-hj95d\" (UID: \"0c0dd879-ad03-42f8-8d1c-44f9dae55b42\") " pod="openshift-monitoring/telemeter-client-5995d9b5fc-hj95d"
Sep 18 17:16:35.886006 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.885979    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds6w8\" (UniqueName: \"kubernetes.io/projected/0c0dd879-ad03-42f8-8d1c-44f9dae55b42-kube-api-access-ds6w8\") pod \"telemeter-client-5995d9b5fc-hj95d\" (UID: \"0c0dd879-ad03-42f8-8d1c-44f9dae55b42\") " pod="openshift-monitoring/telemeter-client-5995d9b5fc-hj95d"
Sep 18 17:16:35.886122 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.886004    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0c0dd879-ad03-42f8-8d1c-44f9dae55b42-secret-telemeter-client\") pod \"telemeter-client-5995d9b5fc-hj95d\" (UID: \"0c0dd879-ad03-42f8-8d1c-44f9dae55b42\") " pod="openshift-monitoring/telemeter-client-5995d9b5fc-hj95d"
Sep 18 17:16:35.886122 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.886028    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0c0dd879-ad03-42f8-8d1c-44f9dae55b42-federate-client-tls\") pod \"telemeter-client-5995d9b5fc-hj95d\" (UID: \"0c0dd879-ad03-42f8-8d1c-44f9dae55b42\") " pod="openshift-monitoring/telemeter-client-5995d9b5fc-hj95d"
Sep 18 17:16:35.986607 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.986579    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0c0dd879-ad03-42f8-8d1c-44f9dae55b42-telemeter-client-tls\") pod \"telemeter-client-5995d9b5fc-hj95d\" (UID: \"0c0dd879-ad03-42f8-8d1c-44f9dae55b42\") " pod="openshift-monitoring/telemeter-client-5995d9b5fc-hj95d"
Sep 18 17:16:35.986799 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.986617    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0c0dd879-ad03-42f8-8d1c-44f9dae55b42-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5995d9b5fc-hj95d\" (UID: \"0c0dd879-ad03-42f8-8d1c-44f9dae55b42\") " pod="openshift-monitoring/telemeter-client-5995d9b5fc-hj95d"
Sep 18 17:16:35.986799 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.986640    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c0dd879-ad03-42f8-8d1c-44f9dae55b42-metrics-client-ca\") pod \"telemeter-client-5995d9b5fc-hj95d\" (UID: \"0c0dd879-ad03-42f8-8d1c-44f9dae55b42\") " pod="openshift-monitoring/telemeter-client-5995d9b5fc-hj95d"
Sep 18 17:16:35.986799 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.986661    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds6w8\" (UniqueName: \"kubernetes.io/projected/0c0dd879-ad03-42f8-8d1c-44f9dae55b42-kube-api-access-ds6w8\") pod \"telemeter-client-5995d9b5fc-hj95d\" (UID: \"0c0dd879-ad03-42f8-8d1c-44f9dae55b42\") " pod="openshift-monitoring/telemeter-client-5995d9b5fc-hj95d"
Sep 18 17:16:35.986799 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.986695    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0c0dd879-ad03-42f8-8d1c-44f9dae55b42-secret-telemeter-client\") pod \"telemeter-client-5995d9b5fc-hj95d\" (UID: \"0c0dd879-ad03-42f8-8d1c-44f9dae55b42\") " pod="openshift-monitoring/telemeter-client-5995d9b5fc-hj95d"
Sep 18 17:16:35.986799 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.986724    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0c0dd879-ad03-42f8-8d1c-44f9dae55b42-federate-client-tls\") pod \"telemeter-client-5995d9b5fc-hj95d\" (UID: \"0c0dd879-ad03-42f8-8d1c-44f9dae55b42\") " pod="openshift-monitoring/telemeter-client-5995d9b5fc-hj95d"
Sep 18 17:16:35.986799 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.986789    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c0dd879-ad03-42f8-8d1c-44f9dae55b42-serving-certs-ca-bundle\") pod \"telemeter-client-5995d9b5fc-hj95d\" (UID: \"0c0dd879-ad03-42f8-8d1c-44f9dae55b42\") " pod="openshift-monitoring/telemeter-client-5995d9b5fc-hj95d"
Sep 18 17:16:35.987139 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.986827    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c0dd879-ad03-42f8-8d1c-44f9dae55b42-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5995d9b5fc-hj95d\" (UID: \"0c0dd879-ad03-42f8-8d1c-44f9dae55b42\") " pod="openshift-monitoring/telemeter-client-5995d9b5fc-hj95d"
Sep 18 17:16:35.987606 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.987578    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c0dd879-ad03-42f8-8d1c-44f9dae55b42-metrics-client-ca\") pod \"telemeter-client-5995d9b5fc-hj95d\" (UID: \"0c0dd879-ad03-42f8-8d1c-44f9dae55b42\") " pod="openshift-monitoring/telemeter-client-5995d9b5fc-hj95d"
Sep 18 17:16:35.987771 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.987717    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c0dd879-ad03-42f8-8d1c-44f9dae55b42-serving-certs-ca-bundle\") pod \"telemeter-client-5995d9b5fc-hj95d\" (UID: \"0c0dd879-ad03-42f8-8d1c-44f9dae55b42\") " pod="openshift-monitoring/telemeter-client-5995d9b5fc-hj95d"
Sep 18 17:16:35.987918 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.987820    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c0dd879-ad03-42f8-8d1c-44f9dae55b42-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5995d9b5fc-hj95d\" (UID: \"0c0dd879-ad03-42f8-8d1c-44f9dae55b42\") " pod="openshift-monitoring/telemeter-client-5995d9b5fc-hj95d"
Sep 18 17:16:35.989233 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.989211    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0c0dd879-ad03-42f8-8d1c-44f9dae55b42-telemeter-client-tls\") pod \"telemeter-client-5995d9b5fc-hj95d\" (UID: \"0c0dd879-ad03-42f8-8d1c-44f9dae55b42\") " pod="openshift-monitoring/telemeter-client-5995d9b5fc-hj95d"
Sep 18 17:16:35.989402 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.989378    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0c0dd879-ad03-42f8-8d1c-44f9dae55b42-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5995d9b5fc-hj95d\" (UID: \"0c0dd879-ad03-42f8-8d1c-44f9dae55b42\") " pod="openshift-monitoring/telemeter-client-5995d9b5fc-hj95d"
Sep 18 17:16:35.989483 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.989464    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0c0dd879-ad03-42f8-8d1c-44f9dae55b42-secret-telemeter-client\") pod \"telemeter-client-5995d9b5fc-hj95d\" (UID: \"0c0dd879-ad03-42f8-8d1c-44f9dae55b42\") " pod="openshift-monitoring/telemeter-client-5995d9b5fc-hj95d"
Sep 18 17:16:35.989597 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.989576    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0c0dd879-ad03-42f8-8d1c-44f9dae55b42-federate-client-tls\") pod \"telemeter-client-5995d9b5fc-hj95d\" (UID: \"0c0dd879-ad03-42f8-8d1c-44f9dae55b42\") " pod="openshift-monitoring/telemeter-client-5995d9b5fc-hj95d"
Sep 18 17:16:35.997105 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:35.997086    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds6w8\" (UniqueName: \"kubernetes.io/projected/0c0dd879-ad03-42f8-8d1c-44f9dae55b42-kube-api-access-ds6w8\") pod \"telemeter-client-5995d9b5fc-hj95d\" (UID: \"0c0dd879-ad03-42f8-8d1c-44f9dae55b42\") " pod="openshift-monitoring/telemeter-client-5995d9b5fc-hj95d"
Sep 18 17:16:36.004914 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:36.004882    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5995d9b5fc-hj95d"
Sep 18 17:16:36.270163 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:36.270130    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64cf86bd56-w67gw" event={"ID":"d48e36f1-1944-4e2b-a9b8-0f5e0f352428","Type":"ContainerStarted","Data":"e040dd47d437b9411909fb431987c9215cbbb47c9b958c6b3116eca9ac427131"}
Sep 18 17:16:36.270531 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:36.270181    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64cf86bd56-w67gw" event={"ID":"d48e36f1-1944-4e2b-a9b8-0f5e0f352428","Type":"ContainerStarted","Data":"ddd4f9cb3a9b3349c3e3f42ba289a1ac8ca2a369d24df47e0f1bfe6b8916e00d"}
Sep 18 17:16:36.292984 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:36.292952    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5995d9b5fc-hj95d"]
Sep 18 17:16:36.296344 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:16:36.296320    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c0dd879_ad03_42f8_8d1c_44f9dae55b42.slice/crio-9b0cb69859de6f0a93dfcc0ba2234c2bb4bdb07d4a027d63b0b32a0f011f0e77 WatchSource:0}: Error finding container 9b0cb69859de6f0a93dfcc0ba2234c2bb4bdb07d4a027d63b0b32a0f011f0e77: Status 404 returned error can't find the container with id 9b0cb69859de6f0a93dfcc0ba2234c2bb4bdb07d4a027d63b0b32a0f011f0e77
Sep 18 17:16:37.277348 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:37.277311    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5995d9b5fc-hj95d" event={"ID":"0c0dd879-ad03-42f8-8d1c-44f9dae55b42","Type":"ContainerStarted","Data":"9b0cb69859de6f0a93dfcc0ba2234c2bb4bdb07d4a027d63b0b32a0f011f0e77"}
Sep 18 17:16:38.281981 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:38.281943    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5995d9b5fc-hj95d" event={"ID":"0c0dd879-ad03-42f8-8d1c-44f9dae55b42","Type":"ContainerStarted","Data":"4d2b99987f9b5cd720d620d42e7d75662b7ccbc350565be199b30481cd224f0f"}
Sep 18 17:16:40.290366 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:40.290328    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5995d9b5fc-hj95d" event={"ID":"0c0dd879-ad03-42f8-8d1c-44f9dae55b42","Type":"ContainerStarted","Data":"9dfb0d3425f556b46359a779724b05b6214e93bef2a5de70c9dd3d11eb5c0fa5"}
Sep 18 17:16:40.290834 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:40.290398    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5995d9b5fc-hj95d" event={"ID":"0c0dd879-ad03-42f8-8d1c-44f9dae55b42","Type":"ContainerStarted","Data":"415c1e198fb6662beaf1aafa57755f0244994b164b97f70cf11d0c5780bd4f8d"}
Sep 18 17:16:40.333276 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:40.333222    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5995d9b5fc-hj95d" podStartSLOduration=2.422736702 podStartE2EDuration="5.333208336s" totalImagesPullingTime="2.910471634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:16:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:16:40.329414689 +0000 UTC m=+113.023342219" watchObservedRunningTime="2026-09-18 17:16:40.333208336 +0000 UTC m=+113.027135864"
Sep 18 17:16:40.333613 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:40.333586    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64cf86bd56-w67gw" podStartSLOduration=5.333576547 podStartE2EDuration="5.333576547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:16:36.318909217 +0000 UTC m=+109.012836913" watchObservedRunningTime="2026-09-18 17:16:40.333576547 +0000 UTC m=+113.027504074"
Sep 18 17:16:40.464986 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:40.464947    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64cf86bd56-w67gw"]
Sep 18 17:16:40.524176 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:40.524141    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59f46f87c6-kp9kh"]
Sep 18 17:16:40.526200 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:40.526178    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59f46f87c6-kp9kh"
Sep 18 17:16:40.544332 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:40.544282    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59f46f87c6-kp9kh"]
Sep 18 17:16:40.621080 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:40.621052    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1a496d5d-4251-404c-9635-93c694b4e647-oauth-serving-cert\") pod \"console-59f46f87c6-kp9kh\" (UID: \"1a496d5d-4251-404c-9635-93c694b4e647\") " pod="openshift-console/console-59f46f87c6-kp9kh"
Sep 18 17:16:40.621243 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:40.621110    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1a496d5d-4251-404c-9635-93c694b4e647-console-config\") pod \"console-59f46f87c6-kp9kh\" (UID: \"1a496d5d-4251-404c-9635-93c694b4e647\") " pod="openshift-console/console-59f46f87c6-kp9kh"
Sep 18 17:16:40.621243 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:40.621163    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a496d5d-4251-404c-9635-93c694b4e647-trusted-ca-bundle\") pod \"console-59f46f87c6-kp9kh\" (UID: \"1a496d5d-4251-404c-9635-93c694b4e647\") " pod="openshift-console/console-59f46f87c6-kp9kh"
Sep 18 17:16:40.621243 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:40.621193    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1a496d5d-4251-404c-9635-93c694b4e647-console-oauth-config\") pod \"console-59f46f87c6-kp9kh\" (UID: \"1a496d5d-4251-404c-9635-93c694b4e647\") " pod="openshift-console/console-59f46f87c6-kp9kh"
Sep 18 17:16:40.621364 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:40.621249    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt8j9\" (UniqueName: \"kubernetes.io/projected/1a496d5d-4251-404c-9635-93c694b4e647-kube-api-access-gt8j9\") pod \"console-59f46f87c6-kp9kh\" (UID: \"1a496d5d-4251-404c-9635-93c694b4e647\") " pod="openshift-console/console-59f46f87c6-kp9kh"
Sep 18 17:16:40.621364 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:40.621281    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a496d5d-4251-404c-9635-93c694b4e647-console-serving-cert\") pod \"console-59f46f87c6-kp9kh\" (UID: \"1a496d5d-4251-404c-9635-93c694b4e647\") " pod="openshift-console/console-59f46f87c6-kp9kh"
Sep 18 17:16:40.621364 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:40.621301    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a496d5d-4251-404c-9635-93c694b4e647-service-ca\") pod \"console-59f46f87c6-kp9kh\" (UID: \"1a496d5d-4251-404c-9635-93c694b4e647\") " pod="openshift-console/console-59f46f87c6-kp9kh"
Sep 18 17:16:40.722618 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:40.722580    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt8j9\" (UniqueName: \"kubernetes.io/projected/1a496d5d-4251-404c-9635-93c694b4e647-kube-api-access-gt8j9\") pod \"console-59f46f87c6-kp9kh\" (UID: \"1a496d5d-4251-404c-9635-93c694b4e647\") " pod="openshift-console/console-59f46f87c6-kp9kh"
Sep 18 17:16:40.722618 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:40.722623    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a496d5d-4251-404c-9635-93c694b4e647-console-serving-cert\") pod \"console-59f46f87c6-kp9kh\" (UID: \"1a496d5d-4251-404c-9635-93c694b4e647\") " pod="openshift-console/console-59f46f87c6-kp9kh"
Sep 18 17:16:40.722824 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:40.722641    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a496d5d-4251-404c-9635-93c694b4e647-service-ca\") pod \"console-59f46f87c6-kp9kh\" (UID: \"1a496d5d-4251-404c-9635-93c694b4e647\") " pod="openshift-console/console-59f46f87c6-kp9kh"
Sep 18 17:16:40.722824 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:40.722665    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1a496d5d-4251-404c-9635-93c694b4e647-oauth-serving-cert\") pod \"console-59f46f87c6-kp9kh\" (UID: \"1a496d5d-4251-404c-9635-93c694b4e647\") " pod="openshift-console/console-59f46f87c6-kp9kh"
Sep 18 17:16:40.722824 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:40.722729    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1a496d5d-4251-404c-9635-93c694b4e647-console-config\") pod \"console-59f46f87c6-kp9kh\" (UID: \"1a496d5d-4251-404c-9635-93c694b4e647\") " pod="openshift-console/console-59f46f87c6-kp9kh"
Sep 18 17:16:40.722824 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:40.722773    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a496d5d-4251-404c-9635-93c694b4e647-trusted-ca-bundle\") pod \"console-59f46f87c6-kp9kh\" (UID: \"1a496d5d-4251-404c-9635-93c694b4e647\") " pod="openshift-console/console-59f46f87c6-kp9kh"
Sep 18 17:16:40.722824 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:40.722799    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1a496d5d-4251-404c-9635-93c694b4e647-console-oauth-config\") pod \"console-59f46f87c6-kp9kh\" (UID: \"1a496d5d-4251-404c-9635-93c694b4e647\") " pod="openshift-console/console-59f46f87c6-kp9kh"
Sep 18 17:16:40.723490 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:40.723465    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a496d5d-4251-404c-9635-93c694b4e647-service-ca\") pod \"console-59f46f87c6-kp9kh\" (UID: \"1a496d5d-4251-404c-9635-93c694b4e647\") " pod="openshift-console/console-59f46f87c6-kp9kh"
Sep 18 17:16:40.723587 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:40.723464    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1a496d5d-4251-404c-9635-93c694b4e647-oauth-serving-cert\") pod \"console-59f46f87c6-kp9kh\" (UID: \"1a496d5d-4251-404c-9635-93c694b4e647\") " pod="openshift-console/console-59f46f87c6-kp9kh"
Sep 18 17:16:40.723669 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:40.723645    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1a496d5d-4251-404c-9635-93c694b4e647-console-config\") pod \"console-59f46f87c6-kp9kh\" (UID: \"1a496d5d-4251-404c-9635-93c694b4e647\") " pod="openshift-console/console-59f46f87c6-kp9kh"
Sep 18 17:16:40.723810 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:40.723784    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a496d5d-4251-404c-9635-93c694b4e647-trusted-ca-bundle\") pod \"console-59f46f87c6-kp9kh\" (UID: \"1a496d5d-4251-404c-9635-93c694b4e647\") " pod="openshift-console/console-59f46f87c6-kp9kh"
Sep 18 17:16:40.725241 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:40.725225    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1a496d5d-4251-404c-9635-93c694b4e647-console-oauth-config\") pod \"console-59f46f87c6-kp9kh\" (UID: \"1a496d5d-4251-404c-9635-93c694b4e647\") " pod="openshift-console/console-59f46f87c6-kp9kh"
Sep 18 17:16:40.725302 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:40.725285    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a496d5d-4251-404c-9635-93c694b4e647-console-serving-cert\") pod \"console-59f46f87c6-kp9kh\" (UID: \"1a496d5d-4251-404c-9635-93c694b4e647\") " pod="openshift-console/console-59f46f87c6-kp9kh"
Sep 18 17:16:40.734399 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:40.734368    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt8j9\" (UniqueName: \"kubernetes.io/projected/1a496d5d-4251-404c-9635-93c694b4e647-kube-api-access-gt8j9\") pod \"console-59f46f87c6-kp9kh\" (UID: \"1a496d5d-4251-404c-9635-93c694b4e647\") " pod="openshift-console/console-59f46f87c6-kp9kh"
Sep 18 17:16:40.834725 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:40.834653    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59f46f87c6-kp9kh"
Sep 18 17:16:41.112938 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:41.112853    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59f46f87c6-kp9kh"]
Sep 18 17:16:41.116450 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:16:41.116427    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a496d5d_4251_404c_9635_93c694b4e647.slice/crio-6cc1a982f1a22d20675ba92aab97652178a6dd16cc73a395a8eff15bba1b5af8 WatchSource:0}: Error finding container 6cc1a982f1a22d20675ba92aab97652178a6dd16cc73a395a8eff15bba1b5af8: Status 404 returned error can't find the container with id 6cc1a982f1a22d20675ba92aab97652178a6dd16cc73a395a8eff15bba1b5af8
Sep 18 17:16:41.295180 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:41.295147    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59f46f87c6-kp9kh" event={"ID":"1a496d5d-4251-404c-9635-93c694b4e647","Type":"ContainerStarted","Data":"96ea983fcfc2c3da0f123995977b7f2e5b32c1642dca785043f17dde54ab2be5"}
Sep 18 17:16:41.295534 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:41.295201    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59f46f87c6-kp9kh" event={"ID":"1a496d5d-4251-404c-9635-93c694b4e647","Type":"ContainerStarted","Data":"6cc1a982f1a22d20675ba92aab97652178a6dd16cc73a395a8eff15bba1b5af8"}
Sep 18 17:16:41.317686 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:41.317637    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59f46f87c6-kp9kh" podStartSLOduration=1.317622277 podStartE2EDuration="1.317622277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:16:41.316574378 +0000 UTC m=+114.010501918" watchObservedRunningTime="2026-09-18 17:16:41.317622277 +0000 UTC m=+114.011549805"
Sep 18 17:16:45.560955 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:45.560920    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64cf86bd56-w67gw"
Sep 18 17:16:48.320924 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:48.320891    2569 generic.go:356] "Generic (PLEG): container finished" podID="cdd63e16-6865-47a1-86ca-c1870d2eae7d" containerID="fd1716625ddfcb7359a2b917ea2ec546dbb3186a2b5de7e5cf9db99ad08995a1" exitCode=0
Sep 18 17:16:48.321314 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:48.320936    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-v4nfn" event={"ID":"cdd63e16-6865-47a1-86ca-c1870d2eae7d","Type":"ContainerDied","Data":"fd1716625ddfcb7359a2b917ea2ec546dbb3186a2b5de7e5cf9db99ad08995a1"}
Sep 18 17:16:48.321314 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:48.321239    2569 scope.go:117] "RemoveContainer" containerID="fd1716625ddfcb7359a2b917ea2ec546dbb3186a2b5de7e5cf9db99ad08995a1"
Sep 18 17:16:48.926805 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:48.926778    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76465d565b-r2rdc_63db05d0-62bd-4fe6-999f-0d19ef2ff266/router/0.log"
Sep 18 17:16:48.968907 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:48.968884    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nwncv_1549ca4a-cf93-4295-8f95-c869e2b6a1c6/serve-healthcheck-canary/0.log"
Sep 18 17:16:49.326749 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:49.326712    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-v4nfn" event={"ID":"cdd63e16-6865-47a1-86ca-c1870d2eae7d","Type":"ContainerStarted","Data":"536457f6622f24f605fa58c205ecb8666a5139270e5f7bfc8be2dba40ec60140"}
Sep 18 17:16:50.834967 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:50.834931    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59f46f87c6-kp9kh"
Sep 18 17:16:50.835354 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:50.835260    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59f46f87c6-kp9kh"
Sep 18 17:16:50.840031 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:50.840010    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59f46f87c6-kp9kh"
Sep 18 17:16:51.337473 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:51.337446    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59f46f87c6-kp9kh"
Sep 18 17:16:51.410977 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:51.410948    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84d7d579f4-7ht29"]
Sep 18 17:16:52.338596 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:52.338565    2569 generic.go:356] "Generic (PLEG): container finished" podID="00c0c1f1-e495-4fc3-8dd6-ec7a4d316728" containerID="7e825c901e36d330eff38c145316774443c58017e4371adc5464556fb379c524" exitCode=0
Sep 18 17:16:52.339006 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:52.338640    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5kvpm" event={"ID":"00c0c1f1-e495-4fc3-8dd6-ec7a4d316728","Type":"ContainerDied","Data":"7e825c901e36d330eff38c145316774443c58017e4371adc5464556fb379c524"}
Sep 18 17:16:52.339085 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:52.339071    2569 scope.go:117] "RemoveContainer" containerID="7e825c901e36d330eff38c145316774443c58017e4371adc5464556fb379c524"
Sep 18 17:16:53.343850 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:53.343813    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5kvpm" event={"ID":"00c0c1f1-e495-4fc3-8dd6-ec7a4d316728","Type":"ContainerStarted","Data":"96a2b7a67e7c4330e9d53e70516f325730b4d8d29d496392136b09df71a1294e"}
Sep 18 17:16:57.572509 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:57.572468    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91a9651b-affa-44e8-8996-1ea02fa0c2c4-metrics-certs\") pod \"network-metrics-daemon-ltwd4\" (UID: \"91a9651b-affa-44e8-8996-1ea02fa0c2c4\") " pod="openshift-multus/network-metrics-daemon-ltwd4"
Sep 18 17:16:57.574837 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:57.574813    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91a9651b-affa-44e8-8996-1ea02fa0c2c4-metrics-certs\") pod \"network-metrics-daemon-ltwd4\" (UID: \"91a9651b-affa-44e8-8996-1ea02fa0c2c4\") " pod="openshift-multus/network-metrics-daemon-ltwd4"
Sep 18 17:16:57.785733 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:57.785706    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2h6n2\""
Sep 18 17:16:57.793206 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:57.793186    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltwd4"
Sep 18 17:16:58.072459 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:58.072371    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ltwd4"]
Sep 18 17:16:58.075206 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:16:58.075181    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91a9651b_affa_44e8_8996_1ea02fa0c2c4.slice/crio-db82f4c4245ad58a99c2c405d6d03eea0d21bd156c070f279ba7a485aa45a164 WatchSource:0}: Error finding container db82f4c4245ad58a99c2c405d6d03eea0d21bd156c070f279ba7a485aa45a164: Status 404 returned error can't find the container with id db82f4c4245ad58a99c2c405d6d03eea0d21bd156c070f279ba7a485aa45a164
Sep 18 17:16:58.363616 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:58.363579    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ltwd4" event={"ID":"91a9651b-affa-44e8-8996-1ea02fa0c2c4","Type":"ContainerStarted","Data":"db82f4c4245ad58a99c2c405d6d03eea0d21bd156c070f279ba7a485aa45a164"}
Sep 18 17:16:59.370625 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:59.370587    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ltwd4" event={"ID":"91a9651b-affa-44e8-8996-1ea02fa0c2c4","Type":"ContainerStarted","Data":"c4d28d631c2323846cc264e6246183acc5091a1e9780898441d9d39a8d0f5680"}
Sep 18 17:16:59.371105 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:59.370654    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ltwd4" event={"ID":"91a9651b-affa-44e8-8996-1ea02fa0c2c4","Type":"ContainerStarted","Data":"dbd37d14ca6c5babff23d3021c62cfab7ffc6e43181a3dcdaa36824ac1880d46"}
Sep 18 17:16:59.392423 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:16:59.392371    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ltwd4" podStartSLOduration=130.321916818 podStartE2EDuration="2m11.392353977s" totalImagesPullingTime="1.070437159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:14:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:16:59.391190868 +0000 UTC m=+132.085118413" watchObservedRunningTime="2026-09-18 17:16:59.392353977 +0000 UTC m=+132.086281504"
Sep 18 17:17:05.485135 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:05.485065    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-64cf86bd56-w67gw" podUID="d48e36f1-1944-4e2b-a9b8-0f5e0f352428" containerName="console" containerID="cri-o://e040dd47d437b9411909fb431987c9215cbbb47c9b958c6b3116eca9ac427131" gracePeriod=15
Sep 18 17:17:05.720764 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:05.720739    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64cf86bd56-w67gw"
Sep 18 17:17:05.842276 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:05.842192    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-console-oauth-config\") pod \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\" (UID: \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\") "
Sep 18 17:17:05.842276 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:05.842231    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-console-serving-cert\") pod \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\" (UID: \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\") "
Sep 18 17:17:05.842276 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:05.842256    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxbjn\" (UniqueName: \"kubernetes.io/projected/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-kube-api-access-lxbjn\") pod \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\" (UID: \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\") "
Sep 18 17:17:05.842276 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:05.842277    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-console-config\") pod \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\" (UID: \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\") "
Sep 18 17:17:05.842595 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:05.842455    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-trusted-ca-bundle\") pod \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\" (UID: \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\") "
Sep 18 17:17:05.842595 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:05.842503    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-oauth-serving-cert\") pod \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\" (UID: \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\") "
Sep 18 17:17:05.842595 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:05.842552    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-service-ca\") pod \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\" (UID: \"d48e36f1-1944-4e2b-a9b8-0f5e0f352428\") "
Sep 18 17:17:05.842754 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:05.842725    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-console-config" (OuterVolumeSpecName: "console-config") pod "d48e36f1-1944-4e2b-a9b8-0f5e0f352428" (UID: "d48e36f1-1944-4e2b-a9b8-0f5e0f352428"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:17:05.842851 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:05.842833    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-console-config\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:17:05.842851 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:05.842843    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d48e36f1-1944-4e2b-a9b8-0f5e0f352428" (UID: "d48e36f1-1944-4e2b-a9b8-0f5e0f352428"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:17:05.842999 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:05.842931    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d48e36f1-1944-4e2b-a9b8-0f5e0f352428" (UID: "d48e36f1-1944-4e2b-a9b8-0f5e0f352428"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:17:05.843079 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:05.843051    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-service-ca" (OuterVolumeSpecName: "service-ca") pod "d48e36f1-1944-4e2b-a9b8-0f5e0f352428" (UID: "d48e36f1-1944-4e2b-a9b8-0f5e0f352428"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:17:05.844597 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:05.844567    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d48e36f1-1944-4e2b-a9b8-0f5e0f352428" (UID: "d48e36f1-1944-4e2b-a9b8-0f5e0f352428"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:17:05.844597 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:05.844586    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d48e36f1-1944-4e2b-a9b8-0f5e0f352428" (UID: "d48e36f1-1944-4e2b-a9b8-0f5e0f352428"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:17:05.844727 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:05.844636    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-kube-api-access-lxbjn" (OuterVolumeSpecName: "kube-api-access-lxbjn") pod "d48e36f1-1944-4e2b-a9b8-0f5e0f352428" (UID: "d48e36f1-1944-4e2b-a9b8-0f5e0f352428"). InnerVolumeSpecName "kube-api-access-lxbjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:17:05.943316 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:05.943286    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-console-oauth-config\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:17:05.943316 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:05.943309    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-console-serving-cert\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:17:05.943316 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:05.943319    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxbjn\" (UniqueName: \"kubernetes.io/projected/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-kube-api-access-lxbjn\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:17:05.943519 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:05.943329    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-trusted-ca-bundle\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:17:05.943519 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:05.943336    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-oauth-serving-cert\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:17:05.943519 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:05.943345    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d48e36f1-1944-4e2b-a9b8-0f5e0f352428-service-ca\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:17:06.401661 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:06.401625    2569 generic.go:356] "Generic (PLEG): container finished" podID="d48e36f1-1944-4e2b-a9b8-0f5e0f352428" containerID="e040dd47d437b9411909fb431987c9215cbbb47c9b958c6b3116eca9ac427131" exitCode=0
Sep 18 17:17:06.401882 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:06.401675    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64cf86bd56-w67gw"
Sep 18 17:17:06.401882 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:06.401679    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64cf86bd56-w67gw" event={"ID":"d48e36f1-1944-4e2b-a9b8-0f5e0f352428","Type":"ContainerDied","Data":"e040dd47d437b9411909fb431987c9215cbbb47c9b958c6b3116eca9ac427131"}
Sep 18 17:17:06.401882 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:06.401796    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64cf86bd56-w67gw" event={"ID":"d48e36f1-1944-4e2b-a9b8-0f5e0f352428","Type":"ContainerDied","Data":"ddd4f9cb3a9b3349c3e3f42ba289a1ac8ca2a369d24df47e0f1bfe6b8916e00d"}
Sep 18 17:17:06.401882 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:06.401825    2569 scope.go:117] "RemoveContainer" containerID="e040dd47d437b9411909fb431987c9215cbbb47c9b958c6b3116eca9ac427131"
Sep 18 17:17:06.410257 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:06.410243    2569 scope.go:117] "RemoveContainer" containerID="e040dd47d437b9411909fb431987c9215cbbb47c9b958c6b3116eca9ac427131"
Sep 18 17:17:06.410507 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:17:06.410491    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e040dd47d437b9411909fb431987c9215cbbb47c9b958c6b3116eca9ac427131\": container with ID starting with e040dd47d437b9411909fb431987c9215cbbb47c9b958c6b3116eca9ac427131 not found: ID does not exist" containerID="e040dd47d437b9411909fb431987c9215cbbb47c9b958c6b3116eca9ac427131"
Sep 18 17:17:06.410548 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:06.410516    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e040dd47d437b9411909fb431987c9215cbbb47c9b958c6b3116eca9ac427131"} err="failed to get container status \"e040dd47d437b9411909fb431987c9215cbbb47c9b958c6b3116eca9ac427131\": rpc error: code = NotFound desc = could not find container \"e040dd47d437b9411909fb431987c9215cbbb47c9b958c6b3116eca9ac427131\": container with ID starting with e040dd47d437b9411909fb431987c9215cbbb47c9b958c6b3116eca9ac427131 not found: ID does not exist"
Sep 18 17:17:06.427949 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:06.427927    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64cf86bd56-w67gw"]
Sep 18 17:17:06.430598 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:06.430574    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64cf86bd56-w67gw"]
Sep 18 17:17:07.763000 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:07.762961    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d48e36f1-1944-4e2b-a9b8-0f5e0f352428" path="/var/lib/kubelet/pods/d48e36f1-1944-4e2b-a9b8-0f5e0f352428/volumes"
Sep 18 17:17:16.435822 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:16.435758    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-84d7d579f4-7ht29" podUID="c2429234-6e97-4b9c-8528-bb0463f8da5c" containerName="console" containerID="cri-o://33ec74ceaa70799c5bd81070da923d83754f8e05727f66e920e3a1677cc77d97" gracePeriod=15
Sep 18 17:17:16.678251 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:16.678228    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d7d579f4-7ht29"
Sep 18 17:17:16.833522 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:16.833487    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c2429234-6e97-4b9c-8528-bb0463f8da5c-oauth-serving-cert\") pod \"c2429234-6e97-4b9c-8528-bb0463f8da5c\" (UID: \"c2429234-6e97-4b9c-8528-bb0463f8da5c\") "
Sep 18 17:17:16.833522 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:16.833524    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2429234-6e97-4b9c-8528-bb0463f8da5c-trusted-ca-bundle\") pod \"c2429234-6e97-4b9c-8528-bb0463f8da5c\" (UID: \"c2429234-6e97-4b9c-8528-bb0463f8da5c\") "
Sep 18 17:17:16.833742 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:16.833586    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c2429234-6e97-4b9c-8528-bb0463f8da5c-service-ca\") pod \"c2429234-6e97-4b9c-8528-bb0463f8da5c\" (UID: \"c2429234-6e97-4b9c-8528-bb0463f8da5c\") "
Sep 18 17:17:16.833796 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:16.833734    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tt5ps\" (UniqueName: \"kubernetes.io/projected/c2429234-6e97-4b9c-8528-bb0463f8da5c-kube-api-access-tt5ps\") pod \"c2429234-6e97-4b9c-8528-bb0463f8da5c\" (UID: \"c2429234-6e97-4b9c-8528-bb0463f8da5c\") "
Sep 18 17:17:16.833796 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:16.833778    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c2429234-6e97-4b9c-8528-bb0463f8da5c-console-serving-cert\") pod \"c2429234-6e97-4b9c-8528-bb0463f8da5c\" (UID: \"c2429234-6e97-4b9c-8528-bb0463f8da5c\") "
Sep 18 17:17:16.833932 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:16.833808    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c2429234-6e97-4b9c-8528-bb0463f8da5c-console-oauth-config\") pod \"c2429234-6e97-4b9c-8528-bb0463f8da5c\" (UID: \"c2429234-6e97-4b9c-8528-bb0463f8da5c\") "
Sep 18 17:17:16.833932 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:16.833853    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c2429234-6e97-4b9c-8528-bb0463f8da5c-console-config\") pod \"c2429234-6e97-4b9c-8528-bb0463f8da5c\" (UID: \"c2429234-6e97-4b9c-8528-bb0463f8da5c\") "
Sep 18 17:17:16.834029 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:16.833968    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2429234-6e97-4b9c-8528-bb0463f8da5c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c2429234-6e97-4b9c-8528-bb0463f8da5c" (UID: "c2429234-6e97-4b9c-8528-bb0463f8da5c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:17:16.834029 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:16.834011    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2429234-6e97-4b9c-8528-bb0463f8da5c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c2429234-6e97-4b9c-8528-bb0463f8da5c" (UID: "c2429234-6e97-4b9c-8528-bb0463f8da5c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:17:16.834029 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:16.833972    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2429234-6e97-4b9c-8528-bb0463f8da5c-service-ca" (OuterVolumeSpecName: "service-ca") pod "c2429234-6e97-4b9c-8528-bb0463f8da5c" (UID: "c2429234-6e97-4b9c-8528-bb0463f8da5c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:17:16.834274 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:16.834251    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c2429234-6e97-4b9c-8528-bb0463f8da5c-oauth-serving-cert\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:17:16.834274 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:16.834277    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2429234-6e97-4b9c-8528-bb0463f8da5c-trusted-ca-bundle\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:17:16.834425 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:16.834289    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c2429234-6e97-4b9c-8528-bb0463f8da5c-service-ca\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:17:16.834425 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:16.834296    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2429234-6e97-4b9c-8528-bb0463f8da5c-console-config" (OuterVolumeSpecName: "console-config") pod "c2429234-6e97-4b9c-8528-bb0463f8da5c" (UID: "c2429234-6e97-4b9c-8528-bb0463f8da5c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:17:16.835900 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:16.835877    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2429234-6e97-4b9c-8528-bb0463f8da5c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c2429234-6e97-4b9c-8528-bb0463f8da5c" (UID: "c2429234-6e97-4b9c-8528-bb0463f8da5c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:17:16.835957 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:16.835926    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2429234-6e97-4b9c-8528-bb0463f8da5c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c2429234-6e97-4b9c-8528-bb0463f8da5c" (UID: "c2429234-6e97-4b9c-8528-bb0463f8da5c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:17:16.835992 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:16.835977    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2429234-6e97-4b9c-8528-bb0463f8da5c-kube-api-access-tt5ps" (OuterVolumeSpecName: "kube-api-access-tt5ps") pod "c2429234-6e97-4b9c-8528-bb0463f8da5c" (UID: "c2429234-6e97-4b9c-8528-bb0463f8da5c"). InnerVolumeSpecName "kube-api-access-tt5ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:17:16.935088 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:16.935047    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt5ps\" (UniqueName: \"kubernetes.io/projected/c2429234-6e97-4b9c-8528-bb0463f8da5c-kube-api-access-tt5ps\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:17:16.935088 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:16.935081    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c2429234-6e97-4b9c-8528-bb0463f8da5c-console-serving-cert\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:17:16.935088 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:16.935093    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c2429234-6e97-4b9c-8528-bb0463f8da5c-console-oauth-config\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:17:16.935318 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:16.935107    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c2429234-6e97-4b9c-8528-bb0463f8da5c-console-config\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:17:17.446497 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:17.446463    2569 generic.go:356] "Generic (PLEG): container finished" podID="c2429234-6e97-4b9c-8528-bb0463f8da5c" containerID="33ec74ceaa70799c5bd81070da923d83754f8e05727f66e920e3a1677cc77d97" exitCode=0
Sep 18 17:17:17.446984 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:17.446508    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d7d579f4-7ht29"
Sep 18 17:17:17.446984 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:17.446548    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d7d579f4-7ht29" event={"ID":"c2429234-6e97-4b9c-8528-bb0463f8da5c","Type":"ContainerDied","Data":"33ec74ceaa70799c5bd81070da923d83754f8e05727f66e920e3a1677cc77d97"}
Sep 18 17:17:17.446984 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:17.446604    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d7d579f4-7ht29" event={"ID":"c2429234-6e97-4b9c-8528-bb0463f8da5c","Type":"ContainerDied","Data":"b86e7c942dedb3e87cae556db7822c91082783f43f786989eb721baf612a3554"}
Sep 18 17:17:17.446984 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:17.446634    2569 scope.go:117] "RemoveContainer" containerID="33ec74ceaa70799c5bd81070da923d83754f8e05727f66e920e3a1677cc77d97"
Sep 18 17:17:17.455195 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:17.455175    2569 scope.go:117] "RemoveContainer" containerID="33ec74ceaa70799c5bd81070da923d83754f8e05727f66e920e3a1677cc77d97"
Sep 18 17:17:17.455445 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:17:17.455427    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33ec74ceaa70799c5bd81070da923d83754f8e05727f66e920e3a1677cc77d97\": container with ID starting with 33ec74ceaa70799c5bd81070da923d83754f8e05727f66e920e3a1677cc77d97 not found: ID does not exist" containerID="33ec74ceaa70799c5bd81070da923d83754f8e05727f66e920e3a1677cc77d97"
Sep 18 17:17:17.455497 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:17.455453    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33ec74ceaa70799c5bd81070da923d83754f8e05727f66e920e3a1677cc77d97"} err="failed to get container status \"33ec74ceaa70799c5bd81070da923d83754f8e05727f66e920e3a1677cc77d97\": rpc error: code = NotFound desc = could not find container \"33ec74ceaa70799c5bd81070da923d83754f8e05727f66e920e3a1677cc77d97\": container with ID starting with 33ec74ceaa70799c5bd81070da923d83754f8e05727f66e920e3a1677cc77d97 not found: ID does not exist"
Sep 18 17:17:17.479834 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:17.479809    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84d7d579f4-7ht29"]
Sep 18 17:17:17.484031 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:17.484011    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84d7d579f4-7ht29"]
Sep 18 17:17:17.763730 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:17.763701    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2429234-6e97-4b9c-8528-bb0463f8da5c" path="/var/lib/kubelet/pods/c2429234-6e97-4b9c-8528-bb0463f8da5c/volumes"
Sep 18 17:17:50.671493 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:50.669441    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c7b8789b8-qpmdj"]
Sep 18 17:17:50.671493 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:50.670042    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2429234-6e97-4b9c-8528-bb0463f8da5c" containerName="console"
Sep 18 17:17:50.671493 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:50.670066    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2429234-6e97-4b9c-8528-bb0463f8da5c" containerName="console"
Sep 18 17:17:50.671493 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:50.670085    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d48e36f1-1944-4e2b-a9b8-0f5e0f352428" containerName="console"
Sep 18 17:17:50.671493 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:50.670100    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d48e36f1-1944-4e2b-a9b8-0f5e0f352428" containerName="console"
Sep 18 17:17:50.671493 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:50.670227    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d48e36f1-1944-4e2b-a9b8-0f5e0f352428" containerName="console"
Sep 18 17:17:50.671493 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:50.670238    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2429234-6e97-4b9c-8528-bb0463f8da5c" containerName="console"
Sep 18 17:17:50.673764 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:50.673674    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c7b8789b8-qpmdj"
Sep 18 17:17:50.687305 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:50.687279    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlvs9\" (UniqueName: \"kubernetes.io/projected/a4d1d6b4-46c4-4171-bddb-f707707c0b65-kube-api-access-zlvs9\") pod \"console-6c7b8789b8-qpmdj\" (UID: \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\") " pod="openshift-console/console-6c7b8789b8-qpmdj"
Sep 18 17:17:50.687414 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:50.687333    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4d1d6b4-46c4-4171-bddb-f707707c0b65-console-config\") pod \"console-6c7b8789b8-qpmdj\" (UID: \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\") " pod="openshift-console/console-6c7b8789b8-qpmdj"
Sep 18 17:17:50.687414 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:50.687360    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4d1d6b4-46c4-4171-bddb-f707707c0b65-oauth-serving-cert\") pod \"console-6c7b8789b8-qpmdj\" (UID: \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\") " pod="openshift-console/console-6c7b8789b8-qpmdj"
Sep 18 17:17:50.687528 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:50.687466    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4d1d6b4-46c4-4171-bddb-f707707c0b65-console-serving-cert\") pod \"console-6c7b8789b8-qpmdj\" (UID: \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\") " pod="openshift-console/console-6c7b8789b8-qpmdj"
Sep 18 17:17:50.687528 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:50.687502    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4d1d6b4-46c4-4171-bddb-f707707c0b65-console-oauth-config\") pod \"console-6c7b8789b8-qpmdj\" (UID: \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\") " pod="openshift-console/console-6c7b8789b8-qpmdj"
Sep 18 17:17:50.687528 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:50.687525    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4d1d6b4-46c4-4171-bddb-f707707c0b65-service-ca\") pod \"console-6c7b8789b8-qpmdj\" (UID: \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\") " pod="openshift-console/console-6c7b8789b8-qpmdj"
Sep 18 17:17:50.687671 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:50.687554    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4d1d6b4-46c4-4171-bddb-f707707c0b65-trusted-ca-bundle\") pod \"console-6c7b8789b8-qpmdj\" (UID: \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\") " pod="openshift-console/console-6c7b8789b8-qpmdj"
Sep 18 17:17:50.689457 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:50.689431    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c7b8789b8-qpmdj"]
Sep 18 17:17:50.788187 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:50.788150    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4d1d6b4-46c4-4171-bddb-f707707c0b65-trusted-ca-bundle\") pod \"console-6c7b8789b8-qpmdj\" (UID: \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\") " pod="openshift-console/console-6c7b8789b8-qpmdj"
Sep 18 17:17:50.788187 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:50.788191    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlvs9\" (UniqueName: \"kubernetes.io/projected/a4d1d6b4-46c4-4171-bddb-f707707c0b65-kube-api-access-zlvs9\") pod \"console-6c7b8789b8-qpmdj\" (UID: \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\") " pod="openshift-console/console-6c7b8789b8-qpmdj"
Sep 18 17:17:50.788467 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:50.788235    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4d1d6b4-46c4-4171-bddb-f707707c0b65-console-config\") pod \"console-6c7b8789b8-qpmdj\" (UID: \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\") " pod="openshift-console/console-6c7b8789b8-qpmdj"
Sep 18 17:17:50.788467 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:50.788259    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4d1d6b4-46c4-4171-bddb-f707707c0b65-oauth-serving-cert\") pod \"console-6c7b8789b8-qpmdj\" (UID: \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\") " pod="openshift-console/console-6c7b8789b8-qpmdj"
Sep 18 17:17:50.788467 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:50.788304    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4d1d6b4-46c4-4171-bddb-f707707c0b65-console-serving-cert\") pod \"console-6c7b8789b8-qpmdj\" (UID: \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\") " pod="openshift-console/console-6c7b8789b8-qpmdj"
Sep 18 17:17:50.788467 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:50.788337    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4d1d6b4-46c4-4171-bddb-f707707c0b65-console-oauth-config\") pod \"console-6c7b8789b8-qpmdj\" (UID: \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\") " pod="openshift-console/console-6c7b8789b8-qpmdj"
Sep 18 17:17:50.788467 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:50.788357    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4d1d6b4-46c4-4171-bddb-f707707c0b65-service-ca\") pod \"console-6c7b8789b8-qpmdj\" (UID: \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\") " pod="openshift-console/console-6c7b8789b8-qpmdj"
Sep 18 17:17:50.789127 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:50.789089    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4d1d6b4-46c4-4171-bddb-f707707c0b65-console-config\") pod \"console-6c7b8789b8-qpmdj\" (UID: \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\") " pod="openshift-console/console-6c7b8789b8-qpmdj"
Sep 18 17:17:50.789262 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:50.789141    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4d1d6b4-46c4-4171-bddb-f707707c0b65-oauth-serving-cert\") pod \"console-6c7b8789b8-qpmdj\" (UID: \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\") " pod="openshift-console/console-6c7b8789b8-qpmdj"
Sep 18 17:17:50.789262 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:50.789204    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4d1d6b4-46c4-4171-bddb-f707707c0b65-service-ca\") pod \"console-6c7b8789b8-qpmdj\" (UID: \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\") " pod="openshift-console/console-6c7b8789b8-qpmdj"
Sep 18 17:17:50.789378 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:50.789282    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4d1d6b4-46c4-4171-bddb-f707707c0b65-trusted-ca-bundle\") pod \"console-6c7b8789b8-qpmdj\" (UID: \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\") " pod="openshift-console/console-6c7b8789b8-qpmdj"
Sep 18 17:17:50.790971 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:50.790948    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4d1d6b4-46c4-4171-bddb-f707707c0b65-console-oauth-config\") pod \"console-6c7b8789b8-qpmdj\" (UID: \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\") " pod="openshift-console/console-6c7b8789b8-qpmdj"
Sep 18 17:17:50.791072 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:50.790948    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4d1d6b4-46c4-4171-bddb-f707707c0b65-console-serving-cert\") pod \"console-6c7b8789b8-qpmdj\" (UID: \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\") " pod="openshift-console/console-6c7b8789b8-qpmdj"
Sep 18 17:17:50.804011 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:50.803985    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlvs9\" (UniqueName: \"kubernetes.io/projected/a4d1d6b4-46c4-4171-bddb-f707707c0b65-kube-api-access-zlvs9\") pod \"console-6c7b8789b8-qpmdj\" (UID: \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\") " pod="openshift-console/console-6c7b8789b8-qpmdj"
Sep 18 17:17:50.984353 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:50.984321    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c7b8789b8-qpmdj"
Sep 18 17:17:51.254337 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:51.254252    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c7b8789b8-qpmdj"]
Sep 18 17:17:51.257780 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:17:51.257750    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4d1d6b4_46c4_4171_bddb_f707707c0b65.slice/crio-66fc629684ec8db3bc3e923c2a1e72b2e6ef79e4ff949b8b1a3bae82d395ae4a WatchSource:0}: Error finding container 66fc629684ec8db3bc3e923c2a1e72b2e6ef79e4ff949b8b1a3bae82d395ae4a: Status 404 returned error can't find the container with id 66fc629684ec8db3bc3e923c2a1e72b2e6ef79e4ff949b8b1a3bae82d395ae4a
Sep 18 17:17:51.563759 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:51.563670    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c7b8789b8-qpmdj" event={"ID":"a4d1d6b4-46c4-4171-bddb-f707707c0b65","Type":"ContainerStarted","Data":"f73e55c8b089c73a23701f69d2c412e92ecd55cd222e386b51632d264fa56374"}
Sep 18 17:17:51.563759 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:51.563723    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c7b8789b8-qpmdj" event={"ID":"a4d1d6b4-46c4-4171-bddb-f707707c0b65","Type":"ContainerStarted","Data":"66fc629684ec8db3bc3e923c2a1e72b2e6ef79e4ff949b8b1a3bae82d395ae4a"}
Sep 18 17:17:51.586764 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:17:51.586712    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c7b8789b8-qpmdj" podStartSLOduration=1.5866885640000001 podStartE2EDuration="1.586688564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:17:51.58575933 +0000 UTC m=+184.279686857" watchObservedRunningTime="2026-09-18 17:17:51.586688564 +0000 UTC m=+184.280616092"
Sep 18 17:18:00.985417 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:00.985369    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c7b8789b8-qpmdj"
Sep 18 17:18:00.985907 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:00.985439    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c7b8789b8-qpmdj"
Sep 18 17:18:00.990077 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:00.990056    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c7b8789b8-qpmdj"
Sep 18 17:18:01.603424 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:01.603395    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c7b8789b8-qpmdj"
Sep 18 17:18:01.656136 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:01.656105    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59f46f87c6-kp9kh"]
Sep 18 17:18:26.677503 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:26.677461    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-59f46f87c6-kp9kh" podUID="1a496d5d-4251-404c-9635-93c694b4e647" containerName="console" containerID="cri-o://96ea983fcfc2c3da0f123995977b7f2e5b32c1642dca785043f17dde54ab2be5" gracePeriod=15
Sep 18 17:18:26.910852 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:26.910826    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59f46f87c6-kp9kh"
Sep 18 17:18:27.088857 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:27.088810    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1a496d5d-4251-404c-9635-93c694b4e647-console-oauth-config\") pod \"1a496d5d-4251-404c-9635-93c694b4e647\" (UID: \"1a496d5d-4251-404c-9635-93c694b4e647\") "
Sep 18 17:18:27.088857 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:27.088885    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gt8j9\" (UniqueName: \"kubernetes.io/projected/1a496d5d-4251-404c-9635-93c694b4e647-kube-api-access-gt8j9\") pod \"1a496d5d-4251-404c-9635-93c694b4e647\" (UID: \"1a496d5d-4251-404c-9635-93c694b4e647\") "
Sep 18 17:18:27.089114 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:27.088919    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1a496d5d-4251-404c-9635-93c694b4e647-console-config\") pod \"1a496d5d-4251-404c-9635-93c694b4e647\" (UID: \"1a496d5d-4251-404c-9635-93c694b4e647\") "
Sep 18 17:18:27.089114 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:27.088966    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1a496d5d-4251-404c-9635-93c694b4e647-oauth-serving-cert\") pod \"1a496d5d-4251-404c-9635-93c694b4e647\" (UID: \"1a496d5d-4251-404c-9635-93c694b4e647\") "
Sep 18 17:18:27.089114 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:27.089008    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a496d5d-4251-404c-9635-93c694b4e647-service-ca\") pod \"1a496d5d-4251-404c-9635-93c694b4e647\" (UID: \"1a496d5d-4251-404c-9635-93c694b4e647\") "
Sep 18 17:18:27.089114 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:27.089028    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a496d5d-4251-404c-9635-93c694b4e647-trusted-ca-bundle\") pod \"1a496d5d-4251-404c-9635-93c694b4e647\" (UID: \"1a496d5d-4251-404c-9635-93c694b4e647\") "
Sep 18 17:18:27.089114 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:27.089046    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a496d5d-4251-404c-9635-93c694b4e647-console-serving-cert\") pod \"1a496d5d-4251-404c-9635-93c694b4e647\" (UID: \"1a496d5d-4251-404c-9635-93c694b4e647\") "
Sep 18 17:18:27.089447 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:27.089414    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a496d5d-4251-404c-9635-93c694b4e647-console-config" (OuterVolumeSpecName: "console-config") pod "1a496d5d-4251-404c-9635-93c694b4e647" (UID: "1a496d5d-4251-404c-9635-93c694b4e647"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:18:27.089531 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:27.089437    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a496d5d-4251-404c-9635-93c694b4e647-service-ca" (OuterVolumeSpecName: "service-ca") pod "1a496d5d-4251-404c-9635-93c694b4e647" (UID: "1a496d5d-4251-404c-9635-93c694b4e647"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:18:27.089531 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:27.089462    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a496d5d-4251-404c-9635-93c694b4e647-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1a496d5d-4251-404c-9635-93c694b4e647" (UID: "1a496d5d-4251-404c-9635-93c694b4e647"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:18:27.089531 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:27.089490    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a496d5d-4251-404c-9635-93c694b4e647-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1a496d5d-4251-404c-9635-93c694b4e647" (UID: "1a496d5d-4251-404c-9635-93c694b4e647"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:18:27.091134 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:27.091107    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a496d5d-4251-404c-9635-93c694b4e647-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1a496d5d-4251-404c-9635-93c694b4e647" (UID: "1a496d5d-4251-404c-9635-93c694b4e647"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:18:27.091236 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:27.091200    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a496d5d-4251-404c-9635-93c694b4e647-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1a496d5d-4251-404c-9635-93c694b4e647" (UID: "1a496d5d-4251-404c-9635-93c694b4e647"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:18:27.091236 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:27.091196    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a496d5d-4251-404c-9635-93c694b4e647-kube-api-access-gt8j9" (OuterVolumeSpecName: "kube-api-access-gt8j9") pod "1a496d5d-4251-404c-9635-93c694b4e647" (UID: "1a496d5d-4251-404c-9635-93c694b4e647"). InnerVolumeSpecName "kube-api-access-gt8j9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:18:27.190201 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:27.190172    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1a496d5d-4251-404c-9635-93c694b4e647-oauth-serving-cert\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:18:27.190201 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:27.190196    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a496d5d-4251-404c-9635-93c694b4e647-service-ca\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:18:27.190201 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:27.190205    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a496d5d-4251-404c-9635-93c694b4e647-trusted-ca-bundle\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:18:27.190442 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:27.190214    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a496d5d-4251-404c-9635-93c694b4e647-console-serving-cert\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:18:27.190442 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:27.190223    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1a496d5d-4251-404c-9635-93c694b4e647-console-oauth-config\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:18:27.190442 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:27.190244    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt8j9\" (UniqueName: \"kubernetes.io/projected/1a496d5d-4251-404c-9635-93c694b4e647-kube-api-access-gt8j9\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:18:27.190442 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:27.190254    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1a496d5d-4251-404c-9635-93c694b4e647-console-config\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:18:27.693180 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:27.693144    2569 generic.go:356] "Generic (PLEG): container finished" podID="1a496d5d-4251-404c-9635-93c694b4e647" containerID="96ea983fcfc2c3da0f123995977b7f2e5b32c1642dca785043f17dde54ab2be5" exitCode=0
Sep 18 17:18:27.693642 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:27.693190    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59f46f87c6-kp9kh"
Sep 18 17:18:27.693642 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:27.693225    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59f46f87c6-kp9kh" event={"ID":"1a496d5d-4251-404c-9635-93c694b4e647","Type":"ContainerDied","Data":"96ea983fcfc2c3da0f123995977b7f2e5b32c1642dca785043f17dde54ab2be5"}
Sep 18 17:18:27.693642 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:27.693265    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59f46f87c6-kp9kh" event={"ID":"1a496d5d-4251-404c-9635-93c694b4e647","Type":"ContainerDied","Data":"6cc1a982f1a22d20675ba92aab97652178a6dd16cc73a395a8eff15bba1b5af8"}
Sep 18 17:18:27.693642 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:27.693295    2569 scope.go:117] "RemoveContainer" containerID="96ea983fcfc2c3da0f123995977b7f2e5b32c1642dca785043f17dde54ab2be5"
Sep 18 17:18:27.701532 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:27.701511    2569 scope.go:117] "RemoveContainer" containerID="96ea983fcfc2c3da0f123995977b7f2e5b32c1642dca785043f17dde54ab2be5"
Sep 18 17:18:27.701789 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:18:27.701773    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96ea983fcfc2c3da0f123995977b7f2e5b32c1642dca785043f17dde54ab2be5\": container with ID starting with 96ea983fcfc2c3da0f123995977b7f2e5b32c1642dca785043f17dde54ab2be5 not found: ID does not exist" containerID="96ea983fcfc2c3da0f123995977b7f2e5b32c1642dca785043f17dde54ab2be5"
Sep 18 17:18:27.701844 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:27.701796    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96ea983fcfc2c3da0f123995977b7f2e5b32c1642dca785043f17dde54ab2be5"} err="failed to get container status \"96ea983fcfc2c3da0f123995977b7f2e5b32c1642dca785043f17dde54ab2be5\": rpc error: code = NotFound desc = could not find container \"96ea983fcfc2c3da0f123995977b7f2e5b32c1642dca785043f17dde54ab2be5\": container with ID starting with 96ea983fcfc2c3da0f123995977b7f2e5b32c1642dca785043f17dde54ab2be5 not found: ID does not exist"
Sep 18 17:18:27.714832 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:27.714808    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59f46f87c6-kp9kh"]
Sep 18 17:18:27.718089 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:27.718068    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59f46f87c6-kp9kh"]
Sep 18 17:18:27.763667 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:18:27.763638    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a496d5d-4251-404c-9635-93c694b4e647" path="/var/lib/kubelet/pods/1a496d5d-4251-404c-9635-93c694b4e647/volumes"
Sep 18 17:19:24.367313 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:24.367238    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvh6nf"]
Sep 18 17:19:24.367784 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:24.367486    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a496d5d-4251-404c-9635-93c694b4e647" containerName="console"
Sep 18 17:19:24.367784 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:24.367497    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a496d5d-4251-404c-9635-93c694b4e647" containerName="console"
Sep 18 17:19:24.367784 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:24.367554    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a496d5d-4251-404c-9635-93c694b4e647" containerName="console"
Sep 18 17:19:24.369375 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:24.369360    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvh6nf"
Sep 18 17:19:24.371900 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:24.371879    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 17:19:24.373637 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:24.373623    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 17:19:24.374755 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:24.374737    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ft7jf\""
Sep 18 17:19:24.383855 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:24.383828    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvh6nf"]
Sep 18 17:19:24.500424 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:24.500389    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4f76\" (UniqueName: \"kubernetes.io/projected/a7252957-024c-4c9c-aa5c-cfdbd4acb761-kube-api-access-x4f76\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvh6nf\" (UID: \"a7252957-024c-4c9c-aa5c-cfdbd4acb761\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvh6nf"
Sep 18 17:19:24.500579 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:24.500434    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7252957-024c-4c9c-aa5c-cfdbd4acb761-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvh6nf\" (UID: \"a7252957-024c-4c9c-aa5c-cfdbd4acb761\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvh6nf"
Sep 18 17:19:24.500579 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:24.500499    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7252957-024c-4c9c-aa5c-cfdbd4acb761-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvh6nf\" (UID: \"a7252957-024c-4c9c-aa5c-cfdbd4acb761\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvh6nf"
Sep 18 17:19:24.600846 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:24.600816    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4f76\" (UniqueName: \"kubernetes.io/projected/a7252957-024c-4c9c-aa5c-cfdbd4acb761-kube-api-access-x4f76\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvh6nf\" (UID: \"a7252957-024c-4c9c-aa5c-cfdbd4acb761\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvh6nf"
Sep 18 17:19:24.601037 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:24.600857    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7252957-024c-4c9c-aa5c-cfdbd4acb761-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvh6nf\" (UID: \"a7252957-024c-4c9c-aa5c-cfdbd4acb761\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvh6nf"
Sep 18 17:19:24.601037 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:24.600913    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7252957-024c-4c9c-aa5c-cfdbd4acb761-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvh6nf\" (UID: \"a7252957-024c-4c9c-aa5c-cfdbd4acb761\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvh6nf"
Sep 18 17:19:24.601235 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:24.601218    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7252957-024c-4c9c-aa5c-cfdbd4acb761-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvh6nf\" (UID: \"a7252957-024c-4c9c-aa5c-cfdbd4acb761\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvh6nf"
Sep 18 17:19:24.601280 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:24.601254    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7252957-024c-4c9c-aa5c-cfdbd4acb761-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvh6nf\" (UID: \"a7252957-024c-4c9c-aa5c-cfdbd4acb761\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvh6nf"
Sep 18 17:19:24.621182 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:24.621129    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4f76\" (UniqueName: \"kubernetes.io/projected/a7252957-024c-4c9c-aa5c-cfdbd4acb761-kube-api-access-x4f76\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvh6nf\" (UID: \"a7252957-024c-4c9c-aa5c-cfdbd4acb761\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvh6nf"
Sep 18 17:19:24.678487 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:24.678458    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvh6nf"
Sep 18 17:19:24.962694 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:24.962606    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvh6nf"]
Sep 18 17:19:24.970289 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:19:24.970257    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7252957_024c_4c9c_aa5c_cfdbd4acb761.slice/crio-efc76af04a2b2c54c6b00ae2b139870d4e7b926a2fb42d35473fcb2326983fdc WatchSource:0}: Error finding container efc76af04a2b2c54c6b00ae2b139870d4e7b926a2fb42d35473fcb2326983fdc: Status 404 returned error can't find the container with id efc76af04a2b2c54c6b00ae2b139870d4e7b926a2fb42d35473fcb2326983fdc
Sep 18 17:19:25.898657 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:25.898619    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvh6nf" event={"ID":"a7252957-024c-4c9c-aa5c-cfdbd4acb761","Type":"ContainerStarted","Data":"efc76af04a2b2c54c6b00ae2b139870d4e7b926a2fb42d35473fcb2326983fdc"}
Sep 18 17:19:30.920414 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:30.920325    2569 generic.go:356] "Generic (PLEG): container finished" podID="a7252957-024c-4c9c-aa5c-cfdbd4acb761" containerID="3f8010820c5c4b1981839206c563803b8ebb3b9d76e92bd8e4a7e5bf8b88e2bf" exitCode=0
Sep 18 17:19:30.920780 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:30.920416    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvh6nf" event={"ID":"a7252957-024c-4c9c-aa5c-cfdbd4acb761","Type":"ContainerDied","Data":"3f8010820c5c4b1981839206c563803b8ebb3b9d76e92bd8e4a7e5bf8b88e2bf"}
Sep 18 17:19:32.929426 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:32.929330    2569 generic.go:356] "Generic (PLEG): container finished" podID="a7252957-024c-4c9c-aa5c-cfdbd4acb761" containerID="dd3a35c8e12532d50bed7e2a8f07344e33eca63447c6ccbcef4935f3193451bf" exitCode=0
Sep 18 17:19:32.929426 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:32.929402    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvh6nf" event={"ID":"a7252957-024c-4c9c-aa5c-cfdbd4acb761","Type":"ContainerDied","Data":"dd3a35c8e12532d50bed7e2a8f07344e33eca63447c6ccbcef4935f3193451bf"}
Sep 18 17:19:35.020773 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:35.020738    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xp2jh"]
Sep 18 17:19:35.024068 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:35.024043    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-xp2jh"
Sep 18 17:19:35.027931 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:35.027908    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 18 17:19:35.027931 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:35.027924    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 18 17:19:35.028304 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:35.028288    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-qqpx4\""
Sep 18 17:19:35.037219 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:35.037194    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xp2jh"]
Sep 18 17:19:35.192529 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:35.192490    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70c09f58-68a3-4e87-b382-6ab2af9d2afc-bound-sa-token\") pod \"cert-manager-55b4d954d5-xp2jh\" (UID: \"70c09f58-68a3-4e87-b382-6ab2af9d2afc\") " pod="cert-manager/cert-manager-55b4d954d5-xp2jh"
Sep 18 17:19:35.192720 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:35.192547    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttn59\" (UniqueName: \"kubernetes.io/projected/70c09f58-68a3-4e87-b382-6ab2af9d2afc-kube-api-access-ttn59\") pod \"cert-manager-55b4d954d5-xp2jh\" (UID: \"70c09f58-68a3-4e87-b382-6ab2af9d2afc\") " pod="cert-manager/cert-manager-55b4d954d5-xp2jh"
Sep 18 17:19:35.293754 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:35.293663    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70c09f58-68a3-4e87-b382-6ab2af9d2afc-bound-sa-token\") pod \"cert-manager-55b4d954d5-xp2jh\" (UID: \"70c09f58-68a3-4e87-b382-6ab2af9d2afc\") " pod="cert-manager/cert-manager-55b4d954d5-xp2jh"
Sep 18 17:19:35.293754 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:35.293733    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttn59\" (UniqueName: \"kubernetes.io/projected/70c09f58-68a3-4e87-b382-6ab2af9d2afc-kube-api-access-ttn59\") pod \"cert-manager-55b4d954d5-xp2jh\" (UID: \"70c09f58-68a3-4e87-b382-6ab2af9d2afc\") " pod="cert-manager/cert-manager-55b4d954d5-xp2jh"
Sep 18 17:19:35.304750 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:35.304719    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70c09f58-68a3-4e87-b382-6ab2af9d2afc-bound-sa-token\") pod \"cert-manager-55b4d954d5-xp2jh\" (UID: \"70c09f58-68a3-4e87-b382-6ab2af9d2afc\") " pod="cert-manager/cert-manager-55b4d954d5-xp2jh"
Sep 18 17:19:35.304891 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:35.304814    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttn59\" (UniqueName: \"kubernetes.io/projected/70c09f58-68a3-4e87-b382-6ab2af9d2afc-kube-api-access-ttn59\") pod \"cert-manager-55b4d954d5-xp2jh\" (UID: \"70c09f58-68a3-4e87-b382-6ab2af9d2afc\") " pod="cert-manager/cert-manager-55b4d954d5-xp2jh"
Sep 18 17:19:35.335743 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:35.335708    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-xp2jh"
Sep 18 17:19:35.643159 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:35.643089    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xp2jh"]
Sep 18 17:19:35.646434 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:19:35.646399    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70c09f58_68a3_4e87_b382_6ab2af9d2afc.slice/crio-f464798094c99b027e1a170bd735e39ff28852024e2b19ec1104f10cfe268356 WatchSource:0}: Error finding container f464798094c99b027e1a170bd735e39ff28852024e2b19ec1104f10cfe268356: Status 404 returned error can't find the container with id f464798094c99b027e1a170bd735e39ff28852024e2b19ec1104f10cfe268356
Sep 18 17:19:35.943876 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:35.943782    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xp2jh" event={"ID":"70c09f58-68a3-4e87-b382-6ab2af9d2afc","Type":"ContainerStarted","Data":"f464798094c99b027e1a170bd735e39ff28852024e2b19ec1104f10cfe268356"}
Sep 18 17:19:41.971224 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:41.971191    2569 generic.go:356] "Generic (PLEG): container finished" podID="a7252957-024c-4c9c-aa5c-cfdbd4acb761" containerID="6e3117b7d932afa545101c7c5ccad16409e8e31250f998ce3af7c28431122f07" exitCode=0
Sep 18 17:19:41.971664 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:41.971279    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvh6nf" event={"ID":"a7252957-024c-4c9c-aa5c-cfdbd4acb761","Type":"ContainerDied","Data":"6e3117b7d932afa545101c7c5ccad16409e8e31250f998ce3af7c28431122f07"}
Sep 18 17:19:41.972813 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:41.972793    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xp2jh" event={"ID":"70c09f58-68a3-4e87-b382-6ab2af9d2afc","Type":"ContainerStarted","Data":"a0b228df7afe812ff0031b01384e2ece58143183641aa0dfdff2504834fefab6"}
Sep 18 17:19:42.027140 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:42.027092    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-xp2jh" podStartSLOduration=1.5430321 podStartE2EDuration="7.027075107s" totalImagesPullingTime="5.484043007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:19:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:19:42.024976524 +0000 UTC m=+294.718904052" watchObservedRunningTime="2026-09-18 17:19:42.027075107 +0000 UTC m=+294.721002636"
Sep 18 17:19:43.096275 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:43.096248    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvh6nf"
Sep 18 17:19:43.263797 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:43.263763    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7252957-024c-4c9c-aa5c-cfdbd4acb761-bundle\") pod \"a7252957-024c-4c9c-aa5c-cfdbd4acb761\" (UID: \"a7252957-024c-4c9c-aa5c-cfdbd4acb761\") "
Sep 18 17:19:43.264011 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:43.263822    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4f76\" (UniqueName: \"kubernetes.io/projected/a7252957-024c-4c9c-aa5c-cfdbd4acb761-kube-api-access-x4f76\") pod \"a7252957-024c-4c9c-aa5c-cfdbd4acb761\" (UID: \"a7252957-024c-4c9c-aa5c-cfdbd4acb761\") "
Sep 18 17:19:43.264011 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:43.263856    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7252957-024c-4c9c-aa5c-cfdbd4acb761-util\") pod \"a7252957-024c-4c9c-aa5c-cfdbd4acb761\" (UID: \"a7252957-024c-4c9c-aa5c-cfdbd4acb761\") "
Sep 18 17:19:43.264245 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:43.264218    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7252957-024c-4c9c-aa5c-cfdbd4acb761-bundle" (OuterVolumeSpecName: "bundle") pod "a7252957-024c-4c9c-aa5c-cfdbd4acb761" (UID: "a7252957-024c-4c9c-aa5c-cfdbd4acb761"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:19:43.266111 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:43.266089    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7252957-024c-4c9c-aa5c-cfdbd4acb761-kube-api-access-x4f76" (OuterVolumeSpecName: "kube-api-access-x4f76") pod "a7252957-024c-4c9c-aa5c-cfdbd4acb761" (UID: "a7252957-024c-4c9c-aa5c-cfdbd4acb761"). InnerVolumeSpecName "kube-api-access-x4f76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:19:43.268190 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:43.268157    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7252957-024c-4c9c-aa5c-cfdbd4acb761-util" (OuterVolumeSpecName: "util") pod "a7252957-024c-4c9c-aa5c-cfdbd4acb761" (UID: "a7252957-024c-4c9c-aa5c-cfdbd4acb761"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:19:43.364671 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:43.364618    2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7252957-024c-4c9c-aa5c-cfdbd4acb761-util\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:19:43.364671 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:43.364664    2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7252957-024c-4c9c-aa5c-cfdbd4acb761-bundle\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:19:43.364671 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:43.364675    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4f76\" (UniqueName: \"kubernetes.io/projected/a7252957-024c-4c9c-aa5c-cfdbd4acb761-kube-api-access-x4f76\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:19:43.981801 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:43.981766    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvh6nf" event={"ID":"a7252957-024c-4c9c-aa5c-cfdbd4acb761","Type":"ContainerDied","Data":"efc76af04a2b2c54c6b00ae2b139870d4e7b926a2fb42d35473fcb2326983fdc"}
Sep 18 17:19:43.981972 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:43.981820    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvh6nf"
Sep 18 17:19:43.981972 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:43.981828    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efc76af04a2b2c54c6b00ae2b139870d4e7b926a2fb42d35473fcb2326983fdc"
Sep 18 17:19:46.486267 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:46.486230    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-76k8x"]
Sep 18 17:19:46.486739 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:46.486603    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7252957-024c-4c9c-aa5c-cfdbd4acb761" containerName="extract"
Sep 18 17:19:46.486739 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:46.486621    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7252957-024c-4c9c-aa5c-cfdbd4acb761" containerName="extract"
Sep 18 17:19:46.486739 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:46.486635    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7252957-024c-4c9c-aa5c-cfdbd4acb761" containerName="util"
Sep 18 17:19:46.486739 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:46.486643    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7252957-024c-4c9c-aa5c-cfdbd4acb761" containerName="util"
Sep 18 17:19:46.486739 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:46.486667    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7252957-024c-4c9c-aa5c-cfdbd4acb761" containerName="pull"
Sep 18 17:19:46.486739 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:46.486674    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7252957-024c-4c9c-aa5c-cfdbd4acb761" containerName="pull"
Sep 18 17:19:46.487070 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:46.486778    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7252957-024c-4c9c-aa5c-cfdbd4acb761" containerName="extract"
Sep 18 17:19:46.490149 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:46.490127    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-76k8x"
Sep 18 17:19:46.493294 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:46.493275    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 18 17:19:46.494275 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:46.494255    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-6xvn4\""
Sep 18 17:19:46.494362 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:46.494255    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 18 17:19:46.501169 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:46.501143    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-76k8x"]
Sep 18 17:19:46.589204 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:46.589169    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d578c9ff-051b-453e-936c-d5d1f409a042-tmp\") pod \"openshift-lws-operator-fd8ccff4c-76k8x\" (UID: \"d578c9ff-051b-453e-936c-d5d1f409a042\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-76k8x"
Sep 18 17:19:46.589352 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:46.589225    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz6nv\" (UniqueName: \"kubernetes.io/projected/d578c9ff-051b-453e-936c-d5d1f409a042-kube-api-access-pz6nv\") pod \"openshift-lws-operator-fd8ccff4c-76k8x\" (UID: \"d578c9ff-051b-453e-936c-d5d1f409a042\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-76k8x"
Sep 18 17:19:46.689615 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:46.689567    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz6nv\" (UniqueName: \"kubernetes.io/projected/d578c9ff-051b-453e-936c-d5d1f409a042-kube-api-access-pz6nv\") pod \"openshift-lws-operator-fd8ccff4c-76k8x\" (UID: \"d578c9ff-051b-453e-936c-d5d1f409a042\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-76k8x"
Sep 18 17:19:46.689746 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:46.689632    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d578c9ff-051b-453e-936c-d5d1f409a042-tmp\") pod \"openshift-lws-operator-fd8ccff4c-76k8x\" (UID: \"d578c9ff-051b-453e-936c-d5d1f409a042\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-76k8x"
Sep 18 17:19:46.690007 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:46.689989    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d578c9ff-051b-453e-936c-d5d1f409a042-tmp\") pod \"openshift-lws-operator-fd8ccff4c-76k8x\" (UID: \"d578c9ff-051b-453e-936c-d5d1f409a042\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-76k8x"
Sep 18 17:19:46.699999 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:46.699978    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz6nv\" (UniqueName: \"kubernetes.io/projected/d578c9ff-051b-453e-936c-d5d1f409a042-kube-api-access-pz6nv\") pod \"openshift-lws-operator-fd8ccff4c-76k8x\" (UID: \"d578c9ff-051b-453e-936c-d5d1f409a042\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-76k8x"
Sep 18 17:19:46.798397 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:46.798320    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-76k8x"
Sep 18 17:19:47.087753 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:47.087671    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-76k8x"]
Sep 18 17:19:47.089838 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:19:47.089816    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd578c9ff_051b_453e_936c_d5d1f409a042.slice/crio-342840c6dee5e40e1f3ae90148597fd3ce5d3a6cac6d6a74ad11b999169d052f WatchSource:0}: Error finding container 342840c6dee5e40e1f3ae90148597fd3ce5d3a6cac6d6a74ad11b999169d052f: Status 404 returned error can't find the container with id 342840c6dee5e40e1f3ae90148597fd3ce5d3a6cac6d6a74ad11b999169d052f
Sep 18 17:19:47.653057 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:47.653028    2569 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 17:19:47.996951 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:47.996916    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-76k8x" event={"ID":"d578c9ff-051b-453e-936c-d5d1f409a042","Type":"ContainerStarted","Data":"342840c6dee5e40e1f3ae90148597fd3ce5d3a6cac6d6a74ad11b999169d052f"}
Sep 18 17:19:49.003140 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:49.003101    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-76k8x" event={"ID":"d578c9ff-051b-453e-936c-d5d1f409a042","Type":"ContainerStarted","Data":"0480e5934d731b460dda806e61b5c13a4a4cfc7f37feda9142843246c60f3c64"}
Sep 18 17:19:49.030435 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:19:49.030335    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-76k8x" podStartSLOduration=1.345335432 podStartE2EDuration="3.030316522s" totalImagesPullingTime="1.68498109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:19:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:19:49.029428269 +0000 UTC m=+301.723355797" watchObservedRunningTime="2026-09-18 17:19:49.030316522 +0000 UTC m=+301.724244053"
Sep 18 17:20:08.542925 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.542888    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07b5h7"]
Sep 18 17:20:08.547212 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.547192    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07b5h7"
Sep 18 17:20:08.552082 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.552060    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 17:20:08.553606 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.553072    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 17:20:08.553606 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.553305    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ft7jf\""
Sep 18 17:20:08.555467 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.555444    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9496889b9-8mhvz"]
Sep 18 17:20:08.557442 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.557422    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-8mhvz"
Sep 18 17:20:08.559677 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.559659    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 18 17:20:08.560791 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.560767    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 18 17:20:08.561383 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.561362    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 18 17:20:08.561480 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.561439    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dlwd7\""
Sep 18 17:20:08.561542 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.561522    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 18 17:20:08.562574 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.562551    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07b5h7"]
Sep 18 17:20:08.594293 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.594267    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9496889b9-8mhvz"]
Sep 18 17:20:08.657625 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.657589    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f0748df9-cc4f-4426-9035-21be3c6519e7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9496889b9-8mhvz\" (UID: \"f0748df9-cc4f-4426-9035-21be3c6519e7\") " pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-8mhvz"
Sep 18 17:20:08.657818 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.657643    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjvbh\" (UniqueName: \"kubernetes.io/projected/aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed-kube-api-access-wjvbh\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07b5h7\" (UID: \"aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07b5h7"
Sep 18 17:20:08.657818 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.657667    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f0748df9-cc4f-4426-9035-21be3c6519e7-manifests\") pod \"opendatahub-operator-controller-manager-9496889b9-8mhvz\" (UID: \"f0748df9-cc4f-4426-9035-21be3c6519e7\") " pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-8mhvz"
Sep 18 17:20:08.657818 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.657742    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07b5h7\" (UID: \"aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07b5h7"
Sep 18 17:20:08.657818 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.657785    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0748df9-cc4f-4426-9035-21be3c6519e7-webhook-cert\") pod \"opendatahub-operator-controller-manager-9496889b9-8mhvz\" (UID: \"f0748df9-cc4f-4426-9035-21be3c6519e7\") " pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-8mhvz"
Sep 18 17:20:08.658038 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.657822    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk9qv\" (UniqueName: \"kubernetes.io/projected/f0748df9-cc4f-4426-9035-21be3c6519e7-kube-api-access-nk9qv\") pod \"opendatahub-operator-controller-manager-9496889b9-8mhvz\" (UID: \"f0748df9-cc4f-4426-9035-21be3c6519e7\") " pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-8mhvz"
Sep 18 17:20:08.658038 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.657881    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0748df9-cc4f-4426-9035-21be3c6519e7-tmp\") pod \"opendatahub-operator-controller-manager-9496889b9-8mhvz\" (UID: \"f0748df9-cc4f-4426-9035-21be3c6519e7\") " pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-8mhvz"
Sep 18 17:20:08.658038 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.657919    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07b5h7\" (UID: \"aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07b5h7"
Sep 18 17:20:08.758783 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.758718    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07b5h7\" (UID: \"aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07b5h7"
Sep 18 17:20:08.758977 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.758827    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f0748df9-cc4f-4426-9035-21be3c6519e7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9496889b9-8mhvz\" (UID: \"f0748df9-cc4f-4426-9035-21be3c6519e7\") " pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-8mhvz"
Sep 18 17:20:08.759047 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.759017    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjvbh\" (UniqueName: \"kubernetes.io/projected/aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed-kube-api-access-wjvbh\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07b5h7\" (UID: \"aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07b5h7"
Sep 18 17:20:08.759097 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.759069    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f0748df9-cc4f-4426-9035-21be3c6519e7-manifests\") pod \"opendatahub-operator-controller-manager-9496889b9-8mhvz\" (UID: \"f0748df9-cc4f-4426-9035-21be3c6519e7\") " pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-8mhvz"
Sep 18 17:20:08.759148 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.759112    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07b5h7\" (UID: \"aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07b5h7"
Sep 18 17:20:08.759198 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.759147    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0748df9-cc4f-4426-9035-21be3c6519e7-webhook-cert\") pod \"opendatahub-operator-controller-manager-9496889b9-8mhvz\" (UID: \"f0748df9-cc4f-4426-9035-21be3c6519e7\") " pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-8mhvz"
Sep 18 17:20:08.759313 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.759283    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk9qv\" (UniqueName: \"kubernetes.io/projected/f0748df9-cc4f-4426-9035-21be3c6519e7-kube-api-access-nk9qv\") pod \"opendatahub-operator-controller-manager-9496889b9-8mhvz\" (UID: \"f0748df9-cc4f-4426-9035-21be3c6519e7\") " pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-8mhvz"
Sep 18 17:20:08.759392 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.759349    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0748df9-cc4f-4426-9035-21be3c6519e7-tmp\") pod \"opendatahub-operator-controller-manager-9496889b9-8mhvz\" (UID: \"f0748df9-cc4f-4426-9035-21be3c6519e7\") " pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-8mhvz"
Sep 18 17:20:08.759453 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.759296    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07b5h7\" (UID: \"aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07b5h7"
Sep 18 17:20:08.759502 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.759448    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07b5h7\" (UID: \"aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07b5h7"
Sep 18 17:20:08.759550 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.759516    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f0748df9-cc4f-4426-9035-21be3c6519e7-manifests\") pod \"opendatahub-operator-controller-manager-9496889b9-8mhvz\" (UID: \"f0748df9-cc4f-4426-9035-21be3c6519e7\") " pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-8mhvz"
Sep 18 17:20:08.761384 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.761361    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0748df9-cc4f-4426-9035-21be3c6519e7-tmp\") pod \"opendatahub-operator-controller-manager-9496889b9-8mhvz\" (UID: \"f0748df9-cc4f-4426-9035-21be3c6519e7\") " pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-8mhvz"
Sep 18 17:20:08.761580 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.761565    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0748df9-cc4f-4426-9035-21be3c6519e7-webhook-cert\") pod \"opendatahub-operator-controller-manager-9496889b9-8mhvz\" (UID: \"f0748df9-cc4f-4426-9035-21be3c6519e7\") " pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-8mhvz"
Sep 18 17:20:08.761626 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.761565    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f0748df9-cc4f-4426-9035-21be3c6519e7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9496889b9-8mhvz\" (UID: \"f0748df9-cc4f-4426-9035-21be3c6519e7\") " pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-8mhvz"
Sep 18 17:20:08.767690 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.767665    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjvbh\" (UniqueName: \"kubernetes.io/projected/aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed-kube-api-access-wjvbh\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07b5h7\" (UID: \"aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07b5h7"
Sep 18 17:20:08.768238 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.768217    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk9qv\" (UniqueName: \"kubernetes.io/projected/f0748df9-cc4f-4426-9035-21be3c6519e7-kube-api-access-nk9qv\") pod \"opendatahub-operator-controller-manager-9496889b9-8mhvz\" (UID: \"f0748df9-cc4f-4426-9035-21be3c6519e7\") " pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-8mhvz"
Sep 18 17:20:08.857230 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.857155    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07b5h7"
Sep 18 17:20:08.867486 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:08.867443    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-8mhvz"
Sep 18 17:20:09.133379 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:09.133302    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07b5h7"]
Sep 18 17:20:09.136432 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:20:09.136405    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa04a0bc_c9c5_4c95_af16_d1d6bf2c9eed.slice/crio-aad50cebcbe15250973b232f8c9dba4ef4759d398fa05c9d151a055cf6aed893 WatchSource:0}: Error finding container aad50cebcbe15250973b232f8c9dba4ef4759d398fa05c9d151a055cf6aed893: Status 404 returned error can't find the container with id aad50cebcbe15250973b232f8c9dba4ef4759d398fa05c9d151a055cf6aed893
Sep 18 17:20:09.150660 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:09.150640    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9496889b9-8mhvz"]
Sep 18 17:20:09.153842 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:20:09.153813    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0748df9_cc4f_4426_9035_21be3c6519e7.slice/crio-908e02a3dcf5976c0161bd7b48a4c1b7757d1635781d040e2cbb90c933510f4c WatchSource:0}: Error finding container 908e02a3dcf5976c0161bd7b48a4c1b7757d1635781d040e2cbb90c933510f4c: Status 404 returned error can't find the container with id 908e02a3dcf5976c0161bd7b48a4c1b7757d1635781d040e2cbb90c933510f4c
Sep 18 17:20:09.156055 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:09.156038    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 17:20:10.079028 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:10.078985    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-8mhvz" event={"ID":"f0748df9-cc4f-4426-9035-21be3c6519e7","Type":"ContainerStarted","Data":"908e02a3dcf5976c0161bd7b48a4c1b7757d1635781d040e2cbb90c933510f4c"}
Sep 18 17:20:10.081042 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:10.081012    2569 generic.go:356] "Generic (PLEG): container finished" podID="aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed" containerID="572b88e6a3fdfc008f170efe1698590a62a3186c33529e709123769da6212b56" exitCode=0
Sep 18 17:20:10.081177 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:10.081066    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07b5h7" event={"ID":"aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed","Type":"ContainerDied","Data":"572b88e6a3fdfc008f170efe1698590a62a3186c33529e709123769da6212b56"}
Sep 18 17:20:10.081177 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:10.081104    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07b5h7" event={"ID":"aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed","Type":"ContainerStarted","Data":"aad50cebcbe15250973b232f8c9dba4ef4759d398fa05c9d151a055cf6aed893"}
Sep 18 17:20:13.098144 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:13.098059    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-8mhvz" event={"ID":"f0748df9-cc4f-4426-9035-21be3c6519e7","Type":"ContainerStarted","Data":"e0ff5be2969d41b2c1d29ef00e7af89230359272244299321e9f2c8625c6c3a6"}
Sep 18 17:20:13.099978 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:13.099947    2569 generic.go:356] "Generic (PLEG): container finished" podID="aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed" containerID="20313cd63d66a1fdcaea16504413d3cdf1f8728f88be22705dd13b48c6bead1e" exitCode=0
Sep 18 17:20:13.100108 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:13.100029    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07b5h7" event={"ID":"aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed","Type":"ContainerDied","Data":"20313cd63d66a1fdcaea16504413d3cdf1f8728f88be22705dd13b48c6bead1e"}
Sep 18 17:20:14.106366 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:14.106326    2569 generic.go:356] "Generic (PLEG): container finished" podID="aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed" containerID="634c1a43f77a8d156787fac966a4b299f732b93a115ec3ca6d97e30065a5dd0e" exitCode=0
Sep 18 17:20:14.106775 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:14.106402    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07b5h7" event={"ID":"aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed","Type":"ContainerDied","Data":"634c1a43f77a8d156787fac966a4b299f732b93a115ec3ca6d97e30065a5dd0e"}
Sep 18 17:20:15.113050 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:15.113016    2569 generic.go:356] "Generic (PLEG): container finished" podID="f0748df9-cc4f-4426-9035-21be3c6519e7" containerID="e0ff5be2969d41b2c1d29ef00e7af89230359272244299321e9f2c8625c6c3a6" exitCode=0
Sep 18 17:20:15.113554 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:15.113097    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-8mhvz" event={"ID":"f0748df9-cc4f-4426-9035-21be3c6519e7","Type":"ContainerDied","Data":"e0ff5be2969d41b2c1d29ef00e7af89230359272244299321e9f2c8625c6c3a6"}
Sep 18 17:20:15.261193 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:15.261169    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07b5h7"
Sep 18 17:20:15.313334 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:15.313302    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed-util\") pod \"aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed\" (UID: \"aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed\") "
Sep 18 17:20:15.313476 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:15.313368    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed-bundle\") pod \"aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed\" (UID: \"aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed\") "
Sep 18 17:20:15.313476 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:15.313438    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjvbh\" (UniqueName: \"kubernetes.io/projected/aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed-kube-api-access-wjvbh\") pod \"aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed\" (UID: \"aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed\") "
Sep 18 17:20:15.314069 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:15.314041    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed-bundle" (OuterVolumeSpecName: "bundle") pod "aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed" (UID: "aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:20:15.315425 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:15.315402    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed-kube-api-access-wjvbh" (OuterVolumeSpecName: "kube-api-access-wjvbh") pod "aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed" (UID: "aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed"). InnerVolumeSpecName "kube-api-access-wjvbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:20:15.317996 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:15.317959    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed-util" (OuterVolumeSpecName: "util") pod "aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed" (UID: "aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:20:15.414103 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:15.414036    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjvbh\" (UniqueName: \"kubernetes.io/projected/aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed-kube-api-access-wjvbh\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:20:15.414103 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:15.414060    2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed-util\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:20:15.414103 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:15.414069    2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed-bundle\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:20:16.123332 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:16.123300    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-8mhvz" event={"ID":"f0748df9-cc4f-4426-9035-21be3c6519e7","Type":"ContainerStarted","Data":"badc7cae0f9abeb0680dab1e453e36a536e66716088502cc5887287043911b7b"}
Sep 18 17:20:16.123791 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:16.123526    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-8mhvz"
Sep 18 17:20:16.125289 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:16.125261    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07b5h7" event={"ID":"aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed","Type":"ContainerDied","Data":"aad50cebcbe15250973b232f8c9dba4ef4759d398fa05c9d151a055cf6aed893"}
Sep 18 17:20:16.125400 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:16.125309    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aad50cebcbe15250973b232f8c9dba4ef4759d398fa05c9d151a055cf6aed893"
Sep 18 17:20:16.125400 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:16.125274    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07b5h7"
Sep 18 17:20:16.147638 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:16.147593    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-8mhvz" podStartSLOduration=2.816179492 podStartE2EDuration="8.147578283s" totalImagesPullingTime="4.16718787s" totalInitContainerRuntime="1.164210921s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:20:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:20:16.146418925 +0000 UTC m=+328.840346471" watchObservedRunningTime="2026-09-18 17:20:16.147578283 +0000 UTC m=+328.841505814"
Sep 18 17:20:27.131986 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:27.131948    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-8mhvz"
Sep 18 17:20:29.453114 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:29.453079    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-65b7cd5988-ng2jw"]
Sep 18 17:20:29.453491 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:29.453369    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed" containerName="util"
Sep 18 17:20:29.453491 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:29.453379    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed" containerName="util"
Sep 18 17:20:29.453491 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:29.453395    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed" containerName="pull"
Sep 18 17:20:29.453491 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:29.453400    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed" containerName="pull"
Sep 18 17:20:29.453491 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:29.453407    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed" containerName="extract"
Sep 18 17:20:29.453491 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:29.453412    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed" containerName="extract"
Sep 18 17:20:29.453491 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:29.453454    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa04a0bc-c9c5-4c95-af16-d1d6bf2c9eed" containerName="extract"
Sep 18 17:20:29.458118 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:29.458099    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65b7cd5988-ng2jw"
Sep 18 17:20:29.463756 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:29.463726    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 18 17:20:29.463877 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:29.463785    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-bx9gx\""
Sep 18 17:20:29.464710 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:29.464693    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 18 17:20:29.478306 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:29.478278    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65b7cd5988-ng2jw"]
Sep 18 17:20:29.534316 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:29.534281    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/812e1c0f-7f7a-495a-96c7-7e03d0b2599c-tls-certs\") pod \"kube-auth-proxy-65b7cd5988-ng2jw\" (UID: \"812e1c0f-7f7a-495a-96c7-7e03d0b2599c\") " pod="openshift-ingress/kube-auth-proxy-65b7cd5988-ng2jw"
Sep 18 17:20:29.534480 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:29.534329    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sws8t\" (UniqueName: \"kubernetes.io/projected/812e1c0f-7f7a-495a-96c7-7e03d0b2599c-kube-api-access-sws8t\") pod \"kube-auth-proxy-65b7cd5988-ng2jw\" (UID: \"812e1c0f-7f7a-495a-96c7-7e03d0b2599c\") " pod="openshift-ingress/kube-auth-proxy-65b7cd5988-ng2jw"
Sep 18 17:20:29.534480 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:29.534382    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/812e1c0f-7f7a-495a-96c7-7e03d0b2599c-tmp\") pod \"kube-auth-proxy-65b7cd5988-ng2jw\" (UID: \"812e1c0f-7f7a-495a-96c7-7e03d0b2599c\") " pod="openshift-ingress/kube-auth-proxy-65b7cd5988-ng2jw"
Sep 18 17:20:29.635767 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:29.635728    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sws8t\" (UniqueName: \"kubernetes.io/projected/812e1c0f-7f7a-495a-96c7-7e03d0b2599c-kube-api-access-sws8t\") pod \"kube-auth-proxy-65b7cd5988-ng2jw\" (UID: \"812e1c0f-7f7a-495a-96c7-7e03d0b2599c\") " pod="openshift-ingress/kube-auth-proxy-65b7cd5988-ng2jw"
Sep 18 17:20:29.635767 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:29.635768    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/812e1c0f-7f7a-495a-96c7-7e03d0b2599c-tmp\") pod \"kube-auth-proxy-65b7cd5988-ng2jw\" (UID: \"812e1c0f-7f7a-495a-96c7-7e03d0b2599c\") " pod="openshift-ingress/kube-auth-proxy-65b7cd5988-ng2jw"
Sep 18 17:20:29.636001 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:29.635825    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/812e1c0f-7f7a-495a-96c7-7e03d0b2599c-tls-certs\") pod \"kube-auth-proxy-65b7cd5988-ng2jw\" (UID: \"812e1c0f-7f7a-495a-96c7-7e03d0b2599c\") " pod="openshift-ingress/kube-auth-proxy-65b7cd5988-ng2jw"
Sep 18 17:20:29.638043 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:29.638018    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/812e1c0f-7f7a-495a-96c7-7e03d0b2599c-tmp\") pod \"kube-auth-proxy-65b7cd5988-ng2jw\" (UID: \"812e1c0f-7f7a-495a-96c7-7e03d0b2599c\") " pod="openshift-ingress/kube-auth-proxy-65b7cd5988-ng2jw"
Sep 18 17:20:29.638241 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:29.638224    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/812e1c0f-7f7a-495a-96c7-7e03d0b2599c-tls-certs\") pod \"kube-auth-proxy-65b7cd5988-ng2jw\" (UID: \"812e1c0f-7f7a-495a-96c7-7e03d0b2599c\") " pod="openshift-ingress/kube-auth-proxy-65b7cd5988-ng2jw"
Sep 18 17:20:29.668190 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:29.668158    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sws8t\" (UniqueName: \"kubernetes.io/projected/812e1c0f-7f7a-495a-96c7-7e03d0b2599c-kube-api-access-sws8t\") pod \"kube-auth-proxy-65b7cd5988-ng2jw\" (UID: \"812e1c0f-7f7a-495a-96c7-7e03d0b2599c\") " pod="openshift-ingress/kube-auth-proxy-65b7cd5988-ng2jw"
Sep 18 17:20:29.767433 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:29.767409    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65b7cd5988-ng2jw"
Sep 18 17:20:30.079810 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:20:30.079779    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod812e1c0f_7f7a_495a_96c7_7e03d0b2599c.slice/crio-dcd464d1a26eca3c67120b5b49f14dc2915ee0add2f8d03c0932806334f8fbe5 WatchSource:0}: Error finding container dcd464d1a26eca3c67120b5b49f14dc2915ee0add2f8d03c0932806334f8fbe5: Status 404 returned error can't find the container with id dcd464d1a26eca3c67120b5b49f14dc2915ee0add2f8d03c0932806334f8fbe5
Sep 18 17:20:30.081209 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:30.081187    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65b7cd5988-ng2jw"]
Sep 18 17:20:30.182234 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:30.182198    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65b7cd5988-ng2jw" event={"ID":"812e1c0f-7f7a-495a-96c7-7e03d0b2599c","Type":"ContainerStarted","Data":"dcd464d1a26eca3c67120b5b49f14dc2915ee0add2f8d03c0932806334f8fbe5"}
Sep 18 17:20:31.374465 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:31.374424    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn"]
Sep 18 17:20:31.377108 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:31.377086    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn"
Sep 18 17:20:31.382118 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:31.381900    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 18 17:20:31.382240 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:31.382142    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 17:20:31.382240 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:31.382179    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 18 17:20:31.382355 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:31.382276    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 18 17:20:31.385277 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:31.384977    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7z85n\""
Sep 18 17:20:31.398460 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:31.398399    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn"]
Sep 18 17:20:31.453128 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:31.453089    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2bc1a269-1867-43f2-8642-328d40015e3b-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-d4rwn\" (UID: \"2bc1a269-1867-43f2-8642-328d40015e3b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn"
Sep 18 17:20:31.453299 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:31.453201    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2bc1a269-1867-43f2-8642-328d40015e3b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-d4rwn\" (UID: \"2bc1a269-1867-43f2-8642-328d40015e3b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn"
Sep 18 17:20:31.453299 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:31.453246    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2bc1a269-1867-43f2-8642-328d40015e3b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-d4rwn\" (UID: \"2bc1a269-1867-43f2-8642-328d40015e3b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn"
Sep 18 17:20:31.453299 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:31.453287    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rjb4\" (UniqueName: \"kubernetes.io/projected/2bc1a269-1867-43f2-8642-328d40015e3b-kube-api-access-9rjb4\") pod \"istiod-openshift-gateway-7b7995d5cd-d4rwn\" (UID: \"2bc1a269-1867-43f2-8642-328d40015e3b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn"
Sep 18 17:20:31.453448 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:31.453359    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bc1a269-1867-43f2-8642-328d40015e3b-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-d4rwn\" (UID: \"2bc1a269-1867-43f2-8642-328d40015e3b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn"
Sep 18 17:20:31.453448 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:31.453390    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bc1a269-1867-43f2-8642-328d40015e3b-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-d4rwn\" (UID: \"2bc1a269-1867-43f2-8642-328d40015e3b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn"
Sep 18 17:20:31.453543 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:31.453443    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2bc1a269-1867-43f2-8642-328d40015e3b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-d4rwn\" (UID: \"2bc1a269-1867-43f2-8642-328d40015e3b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn"
Sep 18 17:20:31.554881 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:31.554839    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2bc1a269-1867-43f2-8642-328d40015e3b-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-d4rwn\" (UID: \"2bc1a269-1867-43f2-8642-328d40015e3b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn"
Sep 18 17:20:31.555054 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:31.554932    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2bc1a269-1867-43f2-8642-328d40015e3b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-d4rwn\" (UID: \"2bc1a269-1867-43f2-8642-328d40015e3b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn"
Sep 18 17:20:31.555054 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:31.554965    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2bc1a269-1867-43f2-8642-328d40015e3b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-d4rwn\" (UID: \"2bc1a269-1867-43f2-8642-328d40015e3b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn"
Sep 18 17:20:31.555054 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:31.555002    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rjb4\" (UniqueName: \"kubernetes.io/projected/2bc1a269-1867-43f2-8642-328d40015e3b-kube-api-access-9rjb4\") pod \"istiod-openshift-gateway-7b7995d5cd-d4rwn\" (UID: \"2bc1a269-1867-43f2-8642-328d40015e3b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn"
Sep 18 17:20:31.555054 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:31.555051    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bc1a269-1867-43f2-8642-328d40015e3b-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-d4rwn\" (UID: \"2bc1a269-1867-43f2-8642-328d40015e3b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn"
Sep 18 17:20:31.555276 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:31.555095    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bc1a269-1867-43f2-8642-328d40015e3b-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-d4rwn\" (UID: \"2bc1a269-1867-43f2-8642-328d40015e3b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn"
Sep 18 17:20:31.555276 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:31.555119    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2bc1a269-1867-43f2-8642-328d40015e3b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-d4rwn\" (UID: \"2bc1a269-1867-43f2-8642-328d40015e3b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn"
Sep 18 17:20:31.555807 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:31.555752    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2bc1a269-1867-43f2-8642-328d40015e3b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-d4rwn\" (UID: \"2bc1a269-1867-43f2-8642-328d40015e3b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn"
Sep 18 17:20:31.557944 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:31.557910    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2bc1a269-1867-43f2-8642-328d40015e3b-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-d4rwn\" (UID: \"2bc1a269-1867-43f2-8642-328d40015e3b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn"
Sep 18 17:20:31.558067 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:31.557989    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bc1a269-1867-43f2-8642-328d40015e3b-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-d4rwn\" (UID: \"2bc1a269-1867-43f2-8642-328d40015e3b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn"
Sep 18 17:20:31.558653 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:31.558627    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2bc1a269-1867-43f2-8642-328d40015e3b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-d4rwn\" (UID: \"2bc1a269-1867-43f2-8642-328d40015e3b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn"
Sep 18 17:20:31.559242 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:31.559221    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2bc1a269-1867-43f2-8642-328d40015e3b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-d4rwn\" (UID: \"2bc1a269-1867-43f2-8642-328d40015e3b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn"
Sep 18 17:20:31.569631 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:31.569605    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bc1a269-1867-43f2-8642-328d40015e3b-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-d4rwn\" (UID: \"2bc1a269-1867-43f2-8642-328d40015e3b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn"
Sep 18 17:20:31.570693 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:31.570670    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rjb4\" (UniqueName: \"kubernetes.io/projected/2bc1a269-1867-43f2-8642-328d40015e3b-kube-api-access-9rjb4\") pod \"istiod-openshift-gateway-7b7995d5cd-d4rwn\" (UID: \"2bc1a269-1867-43f2-8642-328d40015e3b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn"
Sep 18 17:20:31.691151 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:31.691072    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn"
Sep 18 17:20:33.129124 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:33.129045    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn"]
Sep 18 17:20:33.132607 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:20:33.132570    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc1a269_1867_43f2_8642_328d40015e3b.slice/crio-f2a483b7caecd508b18d3804dfc2d72b99a3131bc8ad02911590aa8875d2aadb WatchSource:0}: Error finding container f2a483b7caecd508b18d3804dfc2d72b99a3131bc8ad02911590aa8875d2aadb: Status 404 returned error can't find the container with id f2a483b7caecd508b18d3804dfc2d72b99a3131bc8ad02911590aa8875d2aadb
Sep 18 17:20:33.197849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:33.197812    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65b7cd5988-ng2jw" event={"ID":"812e1c0f-7f7a-495a-96c7-7e03d0b2599c","Type":"ContainerStarted","Data":"dbb2ee860ec735e9404f716df641544ed35334299a781801900bd6a1e585f7f0"}
Sep 18 17:20:33.199155 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:33.199128    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn" event={"ID":"2bc1a269-1867-43f2-8642-328d40015e3b","Type":"ContainerStarted","Data":"f2a483b7caecd508b18d3804dfc2d72b99a3131bc8ad02911590aa8875d2aadb"}
Sep 18 17:20:33.233943 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:33.233899    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-65b7cd5988-ng2jw" podStartSLOduration=1.434657718 podStartE2EDuration="4.233884514s" totalImagesPullingTime="2.799226796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:20:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:20:33.233627724 +0000 UTC m=+345.927555251" watchObservedRunningTime="2026-09-18 17:20:33.233884514 +0000 UTC m=+345.927812036"
Sep 18 17:20:35.727130 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:35.727093    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 17:20:35.727432 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:35.727161    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 17:20:36.214504 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:36.214464    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn" event={"ID":"2bc1a269-1867-43f2-8642-328d40015e3b","Type":"ContainerStarted","Data":"4e89b91fde4b7fd5eebfc503b760719c7f3cbf7457d3497ebde2a787f11117ab"}
Sep 18 17:20:36.214698 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:36.214680    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn"
Sep 18 17:20:36.216340 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:36.216311    2569 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-d4rwn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 17:20:36.216453 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:36.216372    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn" podUID="2bc1a269-1867-43f2-8642-328d40015e3b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:20:36.239784 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:36.239731    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn" podStartSLOduration=2.647432956 podStartE2EDuration="5.239715971s" totalImagesPullingTime="2.592283015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:20:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:20:36.23928142 +0000 UTC m=+348.933208947" watchObservedRunningTime="2026-09-18 17:20:36.239715971 +0000 UTC m=+348.933643498"
Sep 18 17:20:37.219653 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:37.219625    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn"
Sep 18 17:20:48.492559 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:48.492422    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b67bbc446-crkqn"]
Sep 18 17:20:48.500059 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:48.500038    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-crkqn"
Sep 18 17:20:48.503644 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:48.503620    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 18 17:20:48.503922 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:48.503899    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 18 17:20:48.503922 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:48.503715    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-qb9bx\""
Sep 18 17:20:48.504113 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:48.503726    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 18 17:20:48.506263 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:48.506243    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b67bbc446-crkqn"]
Sep 18 17:20:48.599056 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:48.599018    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/01955b9b-e9a1-456b-b671-a20d74c803d8-manager-config\") pod \"lws-controller-manager-5b67bbc446-crkqn\" (UID: \"01955b9b-e9a1-456b-b671-a20d74c803d8\") " pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-crkqn"
Sep 18 17:20:48.599056 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:48.599055    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/01955b9b-e9a1-456b-b671-a20d74c803d8-metrics-cert\") pod \"lws-controller-manager-5b67bbc446-crkqn\" (UID: \"01955b9b-e9a1-456b-b671-a20d74c803d8\") " pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-crkqn"
Sep 18 17:20:48.599314 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:48.599077    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgr4k\" (UniqueName: \"kubernetes.io/projected/01955b9b-e9a1-456b-b671-a20d74c803d8-kube-api-access-rgr4k\") pod \"lws-controller-manager-5b67bbc446-crkqn\" (UID: \"01955b9b-e9a1-456b-b671-a20d74c803d8\") " pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-crkqn"
Sep 18 17:20:48.599314 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:48.599186    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01955b9b-e9a1-456b-b671-a20d74c803d8-cert\") pod \"lws-controller-manager-5b67bbc446-crkqn\" (UID: \"01955b9b-e9a1-456b-b671-a20d74c803d8\") " pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-crkqn"
Sep 18 17:20:48.700070 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:48.700028    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/01955b9b-e9a1-456b-b671-a20d74c803d8-manager-config\") pod \"lws-controller-manager-5b67bbc446-crkqn\" (UID: \"01955b9b-e9a1-456b-b671-a20d74c803d8\") " pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-crkqn"
Sep 18 17:20:48.700070 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:48.700072    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/01955b9b-e9a1-456b-b671-a20d74c803d8-metrics-cert\") pod \"lws-controller-manager-5b67bbc446-crkqn\" (UID: \"01955b9b-e9a1-456b-b671-a20d74c803d8\") " pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-crkqn"
Sep 18 17:20:48.700330 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:48.700102    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgr4k\" (UniqueName: \"kubernetes.io/projected/01955b9b-e9a1-456b-b671-a20d74c803d8-kube-api-access-rgr4k\") pod \"lws-controller-manager-5b67bbc446-crkqn\" (UID: \"01955b9b-e9a1-456b-b671-a20d74c803d8\") " pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-crkqn"
Sep 18 17:20:48.700330 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:48.700159    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01955b9b-e9a1-456b-b671-a20d74c803d8-cert\") pod \"lws-controller-manager-5b67bbc446-crkqn\" (UID: \"01955b9b-e9a1-456b-b671-a20d74c803d8\") " pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-crkqn"
Sep 18 17:20:48.700698 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:48.700675    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/01955b9b-e9a1-456b-b671-a20d74c803d8-manager-config\") pod \"lws-controller-manager-5b67bbc446-crkqn\" (UID: \"01955b9b-e9a1-456b-b671-a20d74c803d8\") " pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-crkqn"
Sep 18 17:20:48.702569 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:48.702546    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01955b9b-e9a1-456b-b671-a20d74c803d8-cert\") pod \"lws-controller-manager-5b67bbc446-crkqn\" (UID: \"01955b9b-e9a1-456b-b671-a20d74c803d8\") " pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-crkqn"
Sep 18 17:20:48.702664 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:48.702616    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/01955b9b-e9a1-456b-b671-a20d74c803d8-metrics-cert\") pod \"lws-controller-manager-5b67bbc446-crkqn\" (UID: \"01955b9b-e9a1-456b-b671-a20d74c803d8\") " pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-crkqn"
Sep 18 17:20:48.717935 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:48.717910    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgr4k\" (UniqueName: \"kubernetes.io/projected/01955b9b-e9a1-456b-b671-a20d74c803d8-kube-api-access-rgr4k\") pod \"lws-controller-manager-5b67bbc446-crkqn\" (UID: \"01955b9b-e9a1-456b-b671-a20d74c803d8\") " pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-crkqn"
Sep 18 17:20:48.809673 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:48.809596    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-crkqn"
Sep 18 17:20:49.101584 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:49.098334    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b67bbc446-crkqn"]
Sep 18 17:20:49.271281 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:49.271243    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-crkqn" event={"ID":"01955b9b-e9a1-456b-b671-a20d74c803d8","Type":"ContainerStarted","Data":"dc89c9498d2da63accc48ffb081e8eeed223cfab88baac94090a9bcc07d149f3"}
Sep 18 17:20:51.282977 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:51.282941    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-crkqn" event={"ID":"01955b9b-e9a1-456b-b671-a20d74c803d8","Type":"ContainerStarted","Data":"7876fdd96922762eafd0bfb429b1fab7fd7f334335d11080c451c5a7e54a3e93"}
Sep 18 17:20:51.283366 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:51.283137    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-crkqn"
Sep 18 17:20:51.332779 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:51.332724    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-crkqn" podStartSLOduration=1.764514191 podStartE2EDuration="3.332707753s" totalImagesPullingTime="1.568193562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:20:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:20:51.331304456 +0000 UTC m=+364.025232010" watchObservedRunningTime="2026-09-18 17:20:51.332707753 +0000 UTC m=+364.026635281"
Sep 18 17:20:52.165481 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:52.165440    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn"]
Sep 18 17:20:52.165658 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:52.165627    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn" podUID="2bc1a269-1867-43f2-8642-328d40015e3b" containerName="discovery" containerID="cri-o://4e89b91fde4b7fd5eebfc503b760719c7f3cbf7457d3497ebde2a787f11117ab" gracePeriod=30
Sep 18 17:20:52.219555 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:52.219524    2569 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-d4rwn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.27:8080/ready\": dial tcp 10.133.0.27:8080: connect: connection refused" start-of-body=
Sep 18 17:20:52.219668 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:52.219578    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn" podUID="2bc1a269-1867-43f2-8642-328d40015e3b" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.27:8080/ready\": dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 18 17:20:52.290274 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:52.290246    2569 generic.go:356] "Generic (PLEG): container finished" podID="2bc1a269-1867-43f2-8642-328d40015e3b" containerID="4e89b91fde4b7fd5eebfc503b760719c7f3cbf7457d3497ebde2a787f11117ab" exitCode=0
Sep 18 17:20:52.290600 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:52.290319    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn" event={"ID":"2bc1a269-1867-43f2-8642-328d40015e3b","Type":"ContainerDied","Data":"4e89b91fde4b7fd5eebfc503b760719c7f3cbf7457d3497ebde2a787f11117ab"}
Sep 18 17:20:52.403732 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:52.403707    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn"
Sep 18 17:20:52.433368 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:52.433297    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2bc1a269-1867-43f2-8642-328d40015e3b-istio-csr-dns-cert\") pod \"2bc1a269-1867-43f2-8642-328d40015e3b\" (UID: \"2bc1a269-1867-43f2-8642-328d40015e3b\") "
Sep 18 17:20:52.433368 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:52.433362    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rjb4\" (UniqueName: \"kubernetes.io/projected/2bc1a269-1867-43f2-8642-328d40015e3b-kube-api-access-9rjb4\") pod \"2bc1a269-1867-43f2-8642-328d40015e3b\" (UID: \"2bc1a269-1867-43f2-8642-328d40015e3b\") "
Sep 18 17:20:52.433525 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:52.433396    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bc1a269-1867-43f2-8642-328d40015e3b-istio-token\") pod \"2bc1a269-1867-43f2-8642-328d40015e3b\" (UID: \"2bc1a269-1867-43f2-8642-328d40015e3b\") "
Sep 18 17:20:52.433525 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:52.433435    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2bc1a269-1867-43f2-8642-328d40015e3b-cacerts\") pod \"2bc1a269-1867-43f2-8642-328d40015e3b\" (UID: \"2bc1a269-1867-43f2-8642-328d40015e3b\") "
Sep 18 17:20:52.433525 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:52.433474    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2bc1a269-1867-43f2-8642-328d40015e3b-istio-kubeconfig\") pod \"2bc1a269-1867-43f2-8642-328d40015e3b\" (UID: \"2bc1a269-1867-43f2-8642-328d40015e3b\") "
Sep 18 17:20:52.433525 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:52.433518    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bc1a269-1867-43f2-8642-328d40015e3b-local-certs\") pod \"2bc1a269-1867-43f2-8642-328d40015e3b\" (UID: \"2bc1a269-1867-43f2-8642-328d40015e3b\") "
Sep 18 17:20:52.433729 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:52.433643    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2bc1a269-1867-43f2-8642-328d40015e3b-istio-csr-ca-configmap\") pod \"2bc1a269-1867-43f2-8642-328d40015e3b\" (UID: \"2bc1a269-1867-43f2-8642-328d40015e3b\") "
Sep 18 17:20:52.434268 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:52.434105    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bc1a269-1867-43f2-8642-328d40015e3b-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "2bc1a269-1867-43f2-8642-328d40015e3b" (UID: "2bc1a269-1867-43f2-8642-328d40015e3b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:20:52.436080 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:52.436041    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bc1a269-1867-43f2-8642-328d40015e3b-istio-token" (OuterVolumeSpecName: "istio-token") pod "2bc1a269-1867-43f2-8642-328d40015e3b" (UID: "2bc1a269-1867-43f2-8642-328d40015e3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:20:52.436268 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:52.436246    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bc1a269-1867-43f2-8642-328d40015e3b-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "2bc1a269-1867-43f2-8642-328d40015e3b" (UID: "2bc1a269-1867-43f2-8642-328d40015e3b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:20:52.436365 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:52.436344    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bc1a269-1867-43f2-8642-328d40015e3b-cacerts" (OuterVolumeSpecName: "cacerts") pod "2bc1a269-1867-43f2-8642-328d40015e3b" (UID: "2bc1a269-1867-43f2-8642-328d40015e3b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:20:52.436438 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:52.436411    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bc1a269-1867-43f2-8642-328d40015e3b-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "2bc1a269-1867-43f2-8642-328d40015e3b" (UID: "2bc1a269-1867-43f2-8642-328d40015e3b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:20:52.436585 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:52.436569    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bc1a269-1867-43f2-8642-328d40015e3b-kube-api-access-9rjb4" (OuterVolumeSpecName: "kube-api-access-9rjb4") pod "2bc1a269-1867-43f2-8642-328d40015e3b" (UID: "2bc1a269-1867-43f2-8642-328d40015e3b"). InnerVolumeSpecName "kube-api-access-9rjb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:20:52.436639 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:52.436624    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bc1a269-1867-43f2-8642-328d40015e3b-local-certs" (OuterVolumeSpecName: "local-certs") pod "2bc1a269-1867-43f2-8642-328d40015e3b" (UID: "2bc1a269-1867-43f2-8642-328d40015e3b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:20:52.535222 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:52.535176    2569 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2bc1a269-1867-43f2-8642-328d40015e3b-istio-csr-dns-cert\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:20:52.535222 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:52.535215    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rjb4\" (UniqueName: \"kubernetes.io/projected/2bc1a269-1867-43f2-8642-328d40015e3b-kube-api-access-9rjb4\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:20:52.535222 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:52.535225    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bc1a269-1867-43f2-8642-328d40015e3b-istio-token\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:20:52.535222 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:52.535233    2569 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2bc1a269-1867-43f2-8642-328d40015e3b-cacerts\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:20:52.535465 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:52.535241    2569 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2bc1a269-1867-43f2-8642-328d40015e3b-istio-kubeconfig\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:20:52.535465 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:52.535249    2569 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bc1a269-1867-43f2-8642-328d40015e3b-local-certs\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:20:52.535465 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:52.535260    2569 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2bc1a269-1867-43f2-8642-328d40015e3b-istio-csr-ca-configmap\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:20:53.296467 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:53.296440    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn"
Sep 18 17:20:53.296905 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:53.296437    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn" event={"ID":"2bc1a269-1867-43f2-8642-328d40015e3b","Type":"ContainerDied","Data":"f2a483b7caecd508b18d3804dfc2d72b99a3131bc8ad02911590aa8875d2aadb"}
Sep 18 17:20:53.296905 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:53.296574    2569 scope.go:117] "RemoveContainer" containerID="4e89b91fde4b7fd5eebfc503b760719c7f3cbf7457d3497ebde2a787f11117ab"
Sep 18 17:20:53.340953 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:53.340917    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn"]
Sep 18 17:20:53.345238 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:53.345211    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4rwn"]
Sep 18 17:20:53.764582 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:20:53.764549    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bc1a269-1867-43f2-8642-328d40015e3b" path="/var/lib/kubelet/pods/2bc1a269-1867-43f2-8642-328d40015e3b/volumes"
Sep 18 17:21:02.292705 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:21:02.292669    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-crkqn"
Sep 18 17:21:36.879151 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:21:36.879113    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c7b8789b8-qpmdj"]
Sep 18 17:22:01.898036 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:01.897989    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6c7b8789b8-qpmdj" podUID="a4d1d6b4-46c4-4171-bddb-f707707c0b65" containerName="console" containerID="cri-o://f73e55c8b089c73a23701f69d2c412e92ecd55cd222e386b51632d264fa56374" gracePeriod=15
Sep 18 17:22:02.137336 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:02.137309    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c7b8789b8-qpmdj"
Sep 18 17:22:02.199306 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:02.199234    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4d1d6b4-46c4-4171-bddb-f707707c0b65-oauth-serving-cert\") pod \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\" (UID: \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\") "
Sep 18 17:22:02.199306 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:02.199269    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlvs9\" (UniqueName: \"kubernetes.io/projected/a4d1d6b4-46c4-4171-bddb-f707707c0b65-kube-api-access-zlvs9\") pod \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\" (UID: \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\") "
Sep 18 17:22:02.199306 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:02.199289    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4d1d6b4-46c4-4171-bddb-f707707c0b65-service-ca\") pod \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\" (UID: \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\") "
Sep 18 17:22:02.199549 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:02.199310    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4d1d6b4-46c4-4171-bddb-f707707c0b65-console-config\") pod \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\" (UID: \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\") "
Sep 18 17:22:02.199549 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:02.199370    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4d1d6b4-46c4-4171-bddb-f707707c0b65-console-serving-cert\") pod \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\" (UID: \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\") "
Sep 18 17:22:02.199549 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:02.199404    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4d1d6b4-46c4-4171-bddb-f707707c0b65-console-oauth-config\") pod \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\" (UID: \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\") "
Sep 18 17:22:02.199549 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:02.199443    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4d1d6b4-46c4-4171-bddb-f707707c0b65-trusted-ca-bundle\") pod \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\" (UID: \"a4d1d6b4-46c4-4171-bddb-f707707c0b65\") "
Sep 18 17:22:02.199724 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:02.199677    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4d1d6b4-46c4-4171-bddb-f707707c0b65-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a4d1d6b4-46c4-4171-bddb-f707707c0b65" (UID: "a4d1d6b4-46c4-4171-bddb-f707707c0b65"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:22:02.199724 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:02.199690    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4d1d6b4-46c4-4171-bddb-f707707c0b65-service-ca" (OuterVolumeSpecName: "service-ca") pod "a4d1d6b4-46c4-4171-bddb-f707707c0b65" (UID: "a4d1d6b4-46c4-4171-bddb-f707707c0b65"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:22:02.199825 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:02.199781    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4d1d6b4-46c4-4171-bddb-f707707c0b65-oauth-serving-cert\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:22:02.200074 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:02.200052    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4d1d6b4-46c4-4171-bddb-f707707c0b65-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a4d1d6b4-46c4-4171-bddb-f707707c0b65" (UID: "a4d1d6b4-46c4-4171-bddb-f707707c0b65"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:22:02.200193 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:02.200081    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4d1d6b4-46c4-4171-bddb-f707707c0b65-console-config" (OuterVolumeSpecName: "console-config") pod "a4d1d6b4-46c4-4171-bddb-f707707c0b65" (UID: "a4d1d6b4-46c4-4171-bddb-f707707c0b65"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:22:02.201505 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:02.201484    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4d1d6b4-46c4-4171-bddb-f707707c0b65-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a4d1d6b4-46c4-4171-bddb-f707707c0b65" (UID: "a4d1d6b4-46c4-4171-bddb-f707707c0b65"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:22:02.201679 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:02.201654    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4d1d6b4-46c4-4171-bddb-f707707c0b65-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a4d1d6b4-46c4-4171-bddb-f707707c0b65" (UID: "a4d1d6b4-46c4-4171-bddb-f707707c0b65"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:22:02.201957 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:02.201937    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4d1d6b4-46c4-4171-bddb-f707707c0b65-kube-api-access-zlvs9" (OuterVolumeSpecName: "kube-api-access-zlvs9") pod "a4d1d6b4-46c4-4171-bddb-f707707c0b65" (UID: "a4d1d6b4-46c4-4171-bddb-f707707c0b65"). InnerVolumeSpecName "kube-api-access-zlvs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:22:02.301094 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:02.301061    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4d1d6b4-46c4-4171-bddb-f707707c0b65-console-serving-cert\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:22:02.301094 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:02.301088    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4d1d6b4-46c4-4171-bddb-f707707c0b65-console-oauth-config\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:22:02.301094 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:02.301101    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4d1d6b4-46c4-4171-bddb-f707707c0b65-trusted-ca-bundle\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:22:02.301321 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:02.301113    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlvs9\" (UniqueName: \"kubernetes.io/projected/a4d1d6b4-46c4-4171-bddb-f707707c0b65-kube-api-access-zlvs9\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:22:02.301321 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:02.301127    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4d1d6b4-46c4-4171-bddb-f707707c0b65-service-ca\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:22:02.301321 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:02.301140    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4d1d6b4-46c4-4171-bddb-f707707c0b65-console-config\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:22:02.602707 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:02.602670    2569 generic.go:356] "Generic (PLEG): container finished" podID="a4d1d6b4-46c4-4171-bddb-f707707c0b65" containerID="f73e55c8b089c73a23701f69d2c412e92ecd55cd222e386b51632d264fa56374" exitCode=0
Sep 18 17:22:02.602909 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:02.602724    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c7b8789b8-qpmdj"
Sep 18 17:22:02.602909 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:02.602749    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c7b8789b8-qpmdj" event={"ID":"a4d1d6b4-46c4-4171-bddb-f707707c0b65","Type":"ContainerDied","Data":"f73e55c8b089c73a23701f69d2c412e92ecd55cd222e386b51632d264fa56374"}
Sep 18 17:22:02.602909 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:02.602789    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c7b8789b8-qpmdj" event={"ID":"a4d1d6b4-46c4-4171-bddb-f707707c0b65","Type":"ContainerDied","Data":"66fc629684ec8db3bc3e923c2a1e72b2e6ef79e4ff949b8b1a3bae82d395ae4a"}
Sep 18 17:22:02.602909 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:02.602810    2569 scope.go:117] "RemoveContainer" containerID="f73e55c8b089c73a23701f69d2c412e92ecd55cd222e386b51632d264fa56374"
Sep 18 17:22:02.614293 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:02.614267    2569 scope.go:117] "RemoveContainer" containerID="f73e55c8b089c73a23701f69d2c412e92ecd55cd222e386b51632d264fa56374"
Sep 18 17:22:02.614594 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:22:02.614566    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f73e55c8b089c73a23701f69d2c412e92ecd55cd222e386b51632d264fa56374\": container with ID starting with f73e55c8b089c73a23701f69d2c412e92ecd55cd222e386b51632d264fa56374 not found: ID does not exist" containerID="f73e55c8b089c73a23701f69d2c412e92ecd55cd222e386b51632d264fa56374"
Sep 18 17:22:02.614674 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:02.614604    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f73e55c8b089c73a23701f69d2c412e92ecd55cd222e386b51632d264fa56374"} err="failed to get container status \"f73e55c8b089c73a23701f69d2c412e92ecd55cd222e386b51632d264fa56374\": rpc error: code = NotFound desc = could not find container \"f73e55c8b089c73a23701f69d2c412e92ecd55cd222e386b51632d264fa56374\": container with ID starting with f73e55c8b089c73a23701f69d2c412e92ecd55cd222e386b51632d264fa56374 not found: ID does not exist"
Sep 18 17:22:02.649319 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:02.649293    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c7b8789b8-qpmdj"]
Sep 18 17:22:02.655315 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:02.655290    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c7b8789b8-qpmdj"]
Sep 18 17:22:03.763975 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:03.763941    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4d1d6b4-46c4-4171-bddb-f707707c0b65" path="/var/lib/kubelet/pods/a4d1d6b4-46c4-4171-bddb-f707707c0b65/volumes"
Sep 18 17:22:15.246483 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:15.246402    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-7vgst"]
Sep 18 17:22:15.246950 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:15.246808    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bc1a269-1867-43f2-8642-328d40015e3b" containerName="discovery"
Sep 18 17:22:15.246950 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:15.246825    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bc1a269-1867-43f2-8642-328d40015e3b" containerName="discovery"
Sep 18 17:22:15.246950 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:15.246878    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4d1d6b4-46c4-4171-bddb-f707707c0b65" containerName="console"
Sep 18 17:22:15.246950 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:15.246887    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4d1d6b4-46c4-4171-bddb-f707707c0b65" containerName="console"
Sep 18 17:22:15.247167 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:15.246961    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bc1a269-1867-43f2-8642-328d40015e3b" containerName="discovery"
Sep 18 17:22:15.247167 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:15.246976    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4d1d6b4-46c4-4171-bddb-f707707c0b65" containerName="console"
Sep 18 17:22:15.248852 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:15.248830    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7vgst"
Sep 18 17:22:15.251420 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:15.251400    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 17:22:15.251522 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:15.251402    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 18 17:22:15.251522 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:15.251402    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 17:22:15.251636 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:15.251407    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 18 17:22:15.252341 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:15.252322    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-flxjs\""
Sep 18 17:22:15.257993 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:15.257972    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-7vgst"]
Sep 18 17:22:15.303209 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:15.303173    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a2253623-1c4c-4134-92b7-ac254c2c0569-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-7vgst\" (UID: \"a2253623-1c4c-4134-92b7-ac254c2c0569\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7vgst"
Sep 18 17:22:15.303356 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:15.303217    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ndsj\" (UniqueName: \"kubernetes.io/projected/a2253623-1c4c-4134-92b7-ac254c2c0569-kube-api-access-4ndsj\") pod \"kuadrant-console-plugin-fbdd4f445-7vgst\" (UID: \"a2253623-1c4c-4134-92b7-ac254c2c0569\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7vgst"
Sep 18 17:22:15.303356 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:15.303308    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a2253623-1c4c-4134-92b7-ac254c2c0569-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-7vgst\" (UID: \"a2253623-1c4c-4134-92b7-ac254c2c0569\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7vgst"
Sep 18 17:22:15.403899 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:15.403839    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a2253623-1c4c-4134-92b7-ac254c2c0569-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-7vgst\" (UID: \"a2253623-1c4c-4134-92b7-ac254c2c0569\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7vgst"
Sep 18 17:22:15.404079 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:15.403940    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a2253623-1c4c-4134-92b7-ac254c2c0569-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-7vgst\" (UID: \"a2253623-1c4c-4134-92b7-ac254c2c0569\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7vgst"
Sep 18 17:22:15.404079 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:15.403973    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ndsj\" (UniqueName: \"kubernetes.io/projected/a2253623-1c4c-4134-92b7-ac254c2c0569-kube-api-access-4ndsj\") pod \"kuadrant-console-plugin-fbdd4f445-7vgst\" (UID: \"a2253623-1c4c-4134-92b7-ac254c2c0569\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7vgst"
Sep 18 17:22:15.404234 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:22:15.404083    2569 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 18 17:22:15.404234 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:22:15.404178    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2253623-1c4c-4134-92b7-ac254c2c0569-plugin-serving-cert podName:a2253623-1c4c-4134-92b7-ac254c2c0569 nodeName:}" failed. No retries permitted until 2026-09-18 17:22:15.904152971 +0000 UTC m=+448.598080494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/a2253623-1c4c-4134-92b7-ac254c2c0569-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-7vgst" (UID: "a2253623-1c4c-4134-92b7-ac254c2c0569") : secret "plugin-serving-cert" not found
Sep 18 17:22:15.404593 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:15.404572    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a2253623-1c4c-4134-92b7-ac254c2c0569-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-7vgst\" (UID: \"a2253623-1c4c-4134-92b7-ac254c2c0569\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7vgst"
Sep 18 17:22:15.415534 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:15.415506    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ndsj\" (UniqueName: \"kubernetes.io/projected/a2253623-1c4c-4134-92b7-ac254c2c0569-kube-api-access-4ndsj\") pod \"kuadrant-console-plugin-fbdd4f445-7vgst\" (UID: \"a2253623-1c4c-4134-92b7-ac254c2c0569\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7vgst"
Sep 18 17:22:15.907809 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:15.907759    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a2253623-1c4c-4134-92b7-ac254c2c0569-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-7vgst\" (UID: \"a2253623-1c4c-4134-92b7-ac254c2c0569\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7vgst"
Sep 18 17:22:15.910187 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:15.910160    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a2253623-1c4c-4134-92b7-ac254c2c0569-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-7vgst\" (UID: \"a2253623-1c4c-4134-92b7-ac254c2c0569\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7vgst"
Sep 18 17:22:16.159150 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:16.159074    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7vgst"
Sep 18 17:22:16.434789 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:16.434698    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-7vgst"]
Sep 18 17:22:16.437496 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:22:16.437459    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2253623_1c4c_4134_92b7_ac254c2c0569.slice/crio-1a017c5824d8fa5962312219d0e149af494d0d09c8460db5bf37ff2329574b35 WatchSource:0}: Error finding container 1a017c5824d8fa5962312219d0e149af494d0d09c8460db5bf37ff2329574b35: Status 404 returned error can't find the container with id 1a017c5824d8fa5962312219d0e149af494d0d09c8460db5bf37ff2329574b35
Sep 18 17:22:16.664609 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:16.664569    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7vgst" event={"ID":"a2253623-1c4c-4134-92b7-ac254c2c0569","Type":"ContainerStarted","Data":"1a017c5824d8fa5962312219d0e149af494d0d09c8460db5bf37ff2329574b35"}
Sep 18 17:22:20.684511 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:20.684475    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7vgst" event={"ID":"a2253623-1c4c-4134-92b7-ac254c2c0569","Type":"ContainerStarted","Data":"dbce08ecfa5dc5e791718bab427db048b1586d37bbdc8e825de01848b3913249"}
Sep 18 17:22:20.708345 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:22:20.708245    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7vgst" podStartSLOduration=1.69173061 podStartE2EDuration="5.70822837s" totalImagesPullingTime="4.01649776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:22:20.706461748 +0000 UTC m=+453.400389275" watchObservedRunningTime="2026-09-18 17:22:20.70822837 +0000 UTC m=+453.402155899"
Sep 18 17:23:09.852240 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:09.852200    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-vl8mv"]
Sep 18 17:23:09.856905 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:09.856882    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-vl8mv"
Sep 18 17:23:09.860089 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:09.860067    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dfz6w\""
Sep 18 17:23:09.863100 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:09.863077    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-vl8mv"]
Sep 18 17:23:09.964415 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:09.964377    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-6srx9"]
Sep 18 17:23:09.964760 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:09.964738    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv6ml\" (UniqueName: \"kubernetes.io/projected/47b52251-949f-4e89-abf1-01dd61bcebfd-kube-api-access-hv6ml\") pod \"authorino-7bcfd8f9cd-vl8mv\" (UID: \"47b52251-949f-4e89-abf1-01dd61bcebfd\") " pod="kuadrant-system/authorino-7bcfd8f9cd-vl8mv"
Sep 18 17:23:09.967655 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:09.967639    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-6srx9"
Sep 18 17:23:09.973151 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:09.973127    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6srx9"]
Sep 18 17:23:10.065531 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:10.065495    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhx58\" (UniqueName: \"kubernetes.io/projected/6554c194-ff32-4cf0-9308-86e12a32dd1b-kube-api-access-dhx58\") pod \"authorino-59c974bd48-6srx9\" (UID: \"6554c194-ff32-4cf0-9308-86e12a32dd1b\") " pod="kuadrant-system/authorino-59c974bd48-6srx9"
Sep 18 17:23:10.065711 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:10.065555    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv6ml\" (UniqueName: \"kubernetes.io/projected/47b52251-949f-4e89-abf1-01dd61bcebfd-kube-api-access-hv6ml\") pod \"authorino-7bcfd8f9cd-vl8mv\" (UID: \"47b52251-949f-4e89-abf1-01dd61bcebfd\") " pod="kuadrant-system/authorino-7bcfd8f9cd-vl8mv"
Sep 18 17:23:10.074451 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:10.074428    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv6ml\" (UniqueName: \"kubernetes.io/projected/47b52251-949f-4e89-abf1-01dd61bcebfd-kube-api-access-hv6ml\") pod \"authorino-7bcfd8f9cd-vl8mv\" (UID: \"47b52251-949f-4e89-abf1-01dd61bcebfd\") " pod="kuadrant-system/authorino-7bcfd8f9cd-vl8mv"
Sep 18 17:23:10.166524 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:10.166440    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhx58\" (UniqueName: \"kubernetes.io/projected/6554c194-ff32-4cf0-9308-86e12a32dd1b-kube-api-access-dhx58\") pod \"authorino-59c974bd48-6srx9\" (UID: \"6554c194-ff32-4cf0-9308-86e12a32dd1b\") " pod="kuadrant-system/authorino-59c974bd48-6srx9"
Sep 18 17:23:10.166664 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:10.166553    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-vl8mv"
Sep 18 17:23:10.175962 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:10.175937    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhx58\" (UniqueName: \"kubernetes.io/projected/6554c194-ff32-4cf0-9308-86e12a32dd1b-kube-api-access-dhx58\") pod \"authorino-59c974bd48-6srx9\" (UID: \"6554c194-ff32-4cf0-9308-86e12a32dd1b\") " pod="kuadrant-system/authorino-59c974bd48-6srx9"
Sep 18 17:23:10.276768 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:10.276737    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-6srx9"
Sep 18 17:23:10.435207 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:10.435170    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-vl8mv"]
Sep 18 17:23:10.437361 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:23:10.437331    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47b52251_949f_4e89_abf1_01dd61bcebfd.slice/crio-b192bd7525742381b7bcb95c854ca21f05bce6c00eb458776538deb7b0ac7fd7 WatchSource:0}: Error finding container b192bd7525742381b7bcb95c854ca21f05bce6c00eb458776538deb7b0ac7fd7: Status 404 returned error can't find the container with id b192bd7525742381b7bcb95c854ca21f05bce6c00eb458776538deb7b0ac7fd7
Sep 18 17:23:10.544414 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:10.544387    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6srx9"]
Sep 18 17:23:10.546644 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:23:10.546615    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6554c194_ff32_4cf0_9308_86e12a32dd1b.slice/crio-73e2edb250daf825266970870f3f2a349be81739a9036d30b42153269453e43e WatchSource:0}: Error finding container 73e2edb250daf825266970870f3f2a349be81739a9036d30b42153269453e43e: Status 404 returned error can't find the container with id 73e2edb250daf825266970870f3f2a349be81739a9036d30b42153269453e43e
Sep 18 17:23:10.893596 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:10.893558    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-6srx9" event={"ID":"6554c194-ff32-4cf0-9308-86e12a32dd1b","Type":"ContainerStarted","Data":"73e2edb250daf825266970870f3f2a349be81739a9036d30b42153269453e43e"}
Sep 18 17:23:10.894697 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:10.894677    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-vl8mv" event={"ID":"47b52251-949f-4e89-abf1-01dd61bcebfd","Type":"ContainerStarted","Data":"b192bd7525742381b7bcb95c854ca21f05bce6c00eb458776538deb7b0ac7fd7"}
Sep 18 17:23:13.912977 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:13.912941    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-6srx9" event={"ID":"6554c194-ff32-4cf0-9308-86e12a32dd1b","Type":"ContainerStarted","Data":"6ace29aae66bd87574d2e5d229b0836ce4ab32dd830b8aa32681056a05b2890c"}
Sep 18 17:23:13.914348 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:13.914311    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-vl8mv" event={"ID":"47b52251-949f-4e89-abf1-01dd61bcebfd","Type":"ContainerStarted","Data":"35b0792b412fb7ae8d137c61cb0f47b5799822e3e57e31528d3c8c1e89d52daa"}
Sep 18 17:23:13.937642 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:13.937592    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-6srx9" podStartSLOduration=2.065724851 podStartE2EDuration="4.937579104s" totalImagesPullingTime="2.871854253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:23:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:23:13.937500704 +0000 UTC m=+506.631428233" watchObservedRunningTime="2026-09-18 17:23:13.937579104 +0000 UTC m=+506.631506632"
Sep 18 17:23:13.975642 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:13.975587    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-vl8mv" podStartSLOduration=2.003883156 podStartE2EDuration="4.975568002s" totalImagesPullingTime="2.971684846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:23:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:23:13.974293821 +0000 UTC m=+506.668221349" watchObservedRunningTime="2026-09-18 17:23:13.975568002 +0000 UTC m=+506.669495530"
Sep 18 17:23:13.985023 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:13.984993    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-vl8mv"]
Sep 18 17:23:15.923151 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:15.923112    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-vl8mv" podUID="47b52251-949f-4e89-abf1-01dd61bcebfd" containerName="authorino" containerID="cri-o://35b0792b412fb7ae8d137c61cb0f47b5799822e3e57e31528d3c8c1e89d52daa" gracePeriod=30
Sep 18 17:23:16.160487 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:16.160463    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-vl8mv"
Sep 18 17:23:16.218779 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:16.218707    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hv6ml\" (UniqueName: \"kubernetes.io/projected/47b52251-949f-4e89-abf1-01dd61bcebfd-kube-api-access-hv6ml\") pod \"47b52251-949f-4e89-abf1-01dd61bcebfd\" (UID: \"47b52251-949f-4e89-abf1-01dd61bcebfd\") "
Sep 18 17:23:16.220828 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:16.220798    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47b52251-949f-4e89-abf1-01dd61bcebfd-kube-api-access-hv6ml" (OuterVolumeSpecName: "kube-api-access-hv6ml") pod "47b52251-949f-4e89-abf1-01dd61bcebfd" (UID: "47b52251-949f-4e89-abf1-01dd61bcebfd"). InnerVolumeSpecName "kube-api-access-hv6ml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:23:16.319643 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:16.319593    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv6ml\" (UniqueName: \"kubernetes.io/projected/47b52251-949f-4e89-abf1-01dd61bcebfd-kube-api-access-hv6ml\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:23:16.928243 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:16.928208    2569 generic.go:356] "Generic (PLEG): container finished" podID="47b52251-949f-4e89-abf1-01dd61bcebfd" containerID="35b0792b412fb7ae8d137c61cb0f47b5799822e3e57e31528d3c8c1e89d52daa" exitCode=0
Sep 18 17:23:16.928769 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:16.928249    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-vl8mv"
Sep 18 17:23:16.928769 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:16.928286    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-vl8mv" event={"ID":"47b52251-949f-4e89-abf1-01dd61bcebfd","Type":"ContainerDied","Data":"35b0792b412fb7ae8d137c61cb0f47b5799822e3e57e31528d3c8c1e89d52daa"}
Sep 18 17:23:16.928769 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:16.928329    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-vl8mv" event={"ID":"47b52251-949f-4e89-abf1-01dd61bcebfd","Type":"ContainerDied","Data":"b192bd7525742381b7bcb95c854ca21f05bce6c00eb458776538deb7b0ac7fd7"}
Sep 18 17:23:16.928769 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:16.928345    2569 scope.go:117] "RemoveContainer" containerID="35b0792b412fb7ae8d137c61cb0f47b5799822e3e57e31528d3c8c1e89d52daa"
Sep 18 17:23:16.937001 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:16.936984    2569 scope.go:117] "RemoveContainer" containerID="35b0792b412fb7ae8d137c61cb0f47b5799822e3e57e31528d3c8c1e89d52daa"
Sep 18 17:23:16.937253 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:23:16.937233    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35b0792b412fb7ae8d137c61cb0f47b5799822e3e57e31528d3c8c1e89d52daa\": container with ID starting with 35b0792b412fb7ae8d137c61cb0f47b5799822e3e57e31528d3c8c1e89d52daa not found: ID does not exist" containerID="35b0792b412fb7ae8d137c61cb0f47b5799822e3e57e31528d3c8c1e89d52daa"
Sep 18 17:23:16.937318 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:16.937261    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35b0792b412fb7ae8d137c61cb0f47b5799822e3e57e31528d3c8c1e89d52daa"} err="failed to get container status \"35b0792b412fb7ae8d137c61cb0f47b5799822e3e57e31528d3c8c1e89d52daa\": rpc error: code = NotFound desc = could not find container \"35b0792b412fb7ae8d137c61cb0f47b5799822e3e57e31528d3c8c1e89d52daa\": container with ID starting with 35b0792b412fb7ae8d137c61cb0f47b5799822e3e57e31528d3c8c1e89d52daa not found: ID does not exist"
Sep 18 17:23:16.958928 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:16.958905    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-vl8mv"]
Sep 18 17:23:16.966272 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:16.966253    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-vl8mv"]
Sep 18 17:23:17.764589 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:17.764556    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47b52251-949f-4e89-abf1-01dd61bcebfd" path="/var/lib/kubelet/pods/47b52251-949f-4e89-abf1-01dd61bcebfd/volumes"
Sep 18 17:23:42.610140 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:42.610064    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-54wvk"]
Sep 18 17:23:42.610547 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:42.610407    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47b52251-949f-4e89-abf1-01dd61bcebfd" containerName="authorino"
Sep 18 17:23:42.610547 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:42.610420    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="47b52251-949f-4e89-abf1-01dd61bcebfd" containerName="authorino"
Sep 18 17:23:42.610547 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:42.610468    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="47b52251-949f-4e89-abf1-01dd61bcebfd" containerName="authorino"
Sep 18 17:23:42.612718 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:42.612701    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-54wvk"
Sep 18 17:23:42.634913 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:42.634882    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5c82\" (UniqueName: \"kubernetes.io/projected/e9c610f6-b3ee-4177-bf56-36f95c991ccb-kube-api-access-w5c82\") pod \"authorino-557f876f69-54wvk\" (UID: \"e9c610f6-b3ee-4177-bf56-36f95c991ccb\") " pod="kuadrant-system/authorino-557f876f69-54wvk"
Sep 18 17:23:42.635393 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:42.635371    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-54wvk"]
Sep 18 17:23:42.698693 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:42.698663    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-54wvk"]
Sep 18 17:23:42.698956 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:23:42.698936    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-w5c82], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-54wvk" podUID="e9c610f6-b3ee-4177-bf56-36f95c991ccb"
Sep 18 17:23:42.735307 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:42.735273    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5c82\" (UniqueName: \"kubernetes.io/projected/e9c610f6-b3ee-4177-bf56-36f95c991ccb-kube-api-access-w5c82\") pod \"authorino-557f876f69-54wvk\" (UID: \"e9c610f6-b3ee-4177-bf56-36f95c991ccb\") " pod="kuadrant-system/authorino-557f876f69-54wvk"
Sep 18 17:23:42.759529 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:42.759502    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5c82\" (UniqueName: \"kubernetes.io/projected/e9c610f6-b3ee-4177-bf56-36f95c991ccb-kube-api-access-w5c82\") pod \"authorino-557f876f69-54wvk\" (UID: \"e9c610f6-b3ee-4177-bf56-36f95c991ccb\") " pod="kuadrant-system/authorino-557f876f69-54wvk"
Sep 18 17:23:42.760915 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:42.760889    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-khmzm"]
Sep 18 17:23:42.763538 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:42.763518    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-khmzm"
Sep 18 17:23:42.767453 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:42.767431    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 18 17:23:42.779225 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:42.779201    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-khmzm"]
Sep 18 17:23:42.813275 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:42.813249    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-khmzm"]
Sep 18 17:23:42.813490 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:23:42.813471    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-lw8gc tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-lw8gc tls-cert]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-khmzm" podUID="cec52271-ee94-49f4-afef-acf23c374ffe"
Sep 18 17:23:42.836440 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:42.836407    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw8gc\" (UniqueName: \"kubernetes.io/projected/cec52271-ee94-49f4-afef-acf23c374ffe-kube-api-access-lw8gc\") pod \"authorino-6d4dfc649f-khmzm\" (UID: \"cec52271-ee94-49f4-afef-acf23c374ffe\") " pod="kuadrant-system/authorino-6d4dfc649f-khmzm"
Sep 18 17:23:42.836560 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:42.836540    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cec52271-ee94-49f4-afef-acf23c374ffe-tls-cert\") pod \"authorino-6d4dfc649f-khmzm\" (UID: \"cec52271-ee94-49f4-afef-acf23c374ffe\") " pod="kuadrant-system/authorino-6d4dfc649f-khmzm"
Sep 18 17:23:42.859747 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:42.859712    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-88d9d4659-8vk4h"]
Sep 18 17:23:42.862337 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:42.862284    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-88d9d4659-8vk4h"
Sep 18 17:23:42.878748 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:42.878725    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-88d9d4659-8vk4h"]
Sep 18 17:23:42.936930 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:42.936894    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw8gc\" (UniqueName: \"kubernetes.io/projected/cec52271-ee94-49f4-afef-acf23c374ffe-kube-api-access-lw8gc\") pod \"authorino-6d4dfc649f-khmzm\" (UID: \"cec52271-ee94-49f4-afef-acf23c374ffe\") " pod="kuadrant-system/authorino-6d4dfc649f-khmzm"
Sep 18 17:23:42.937076 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:42.936954    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/707cc028-82b8-465b-b891-28fd16b18e04-tls-cert\") pod \"authorino-88d9d4659-8vk4h\" (UID: \"707cc028-82b8-465b-b891-28fd16b18e04\") " pod="kuadrant-system/authorino-88d9d4659-8vk4h"
Sep 18 17:23:42.937076 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:42.936972    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7njch\" (UniqueName: \"kubernetes.io/projected/707cc028-82b8-465b-b891-28fd16b18e04-kube-api-access-7njch\") pod \"authorino-88d9d4659-8vk4h\" (UID: \"707cc028-82b8-465b-b891-28fd16b18e04\") " pod="kuadrant-system/authorino-88d9d4659-8vk4h"
Sep 18 17:23:42.937076 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:42.936996    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cec52271-ee94-49f4-afef-acf23c374ffe-tls-cert\") pod \"authorino-6d4dfc649f-khmzm\" (UID: \"cec52271-ee94-49f4-afef-acf23c374ffe\") " pod="kuadrant-system/authorino-6d4dfc649f-khmzm"
Sep 18 17:23:42.939402 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:42.939379    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cec52271-ee94-49f4-afef-acf23c374ffe-tls-cert\") pod \"authorino-6d4dfc649f-khmzm\" (UID: \"cec52271-ee94-49f4-afef-acf23c374ffe\") " pod="kuadrant-system/authorino-6d4dfc649f-khmzm"
Sep 18 17:23:42.946549 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:42.946525    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw8gc\" (UniqueName: \"kubernetes.io/projected/cec52271-ee94-49f4-afef-acf23c374ffe-kube-api-access-lw8gc\") pod \"authorino-6d4dfc649f-khmzm\" (UID: \"cec52271-ee94-49f4-afef-acf23c374ffe\") " pod="kuadrant-system/authorino-6d4dfc649f-khmzm"
Sep 18 17:23:43.036143 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:43.036114    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-khmzm"
Sep 18 17:23:43.036291 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:43.036115    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-54wvk"
Sep 18 17:23:43.037392 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:43.037371    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/707cc028-82b8-465b-b891-28fd16b18e04-tls-cert\") pod \"authorino-88d9d4659-8vk4h\" (UID: \"707cc028-82b8-465b-b891-28fd16b18e04\") " pod="kuadrant-system/authorino-88d9d4659-8vk4h"
Sep 18 17:23:43.037506 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:43.037405    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7njch\" (UniqueName: \"kubernetes.io/projected/707cc028-82b8-465b-b891-28fd16b18e04-kube-api-access-7njch\") pod \"authorino-88d9d4659-8vk4h\" (UID: \"707cc028-82b8-465b-b891-28fd16b18e04\") " pod="kuadrant-system/authorino-88d9d4659-8vk4h"
Sep 18 17:23:43.039929 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:43.039909    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/707cc028-82b8-465b-b891-28fd16b18e04-tls-cert\") pod \"authorino-88d9d4659-8vk4h\" (UID: \"707cc028-82b8-465b-b891-28fd16b18e04\") " pod="kuadrant-system/authorino-88d9d4659-8vk4h"
Sep 18 17:23:43.042888 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:43.042858    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-khmzm"
Sep 18 17:23:43.046608 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:43.046590    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7njch\" (UniqueName: \"kubernetes.io/projected/707cc028-82b8-465b-b891-28fd16b18e04-kube-api-access-7njch\") pod \"authorino-88d9d4659-8vk4h\" (UID: \"707cc028-82b8-465b-b891-28fd16b18e04\") " pod="kuadrant-system/authorino-88d9d4659-8vk4h"
Sep 18 17:23:43.058597 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:43.058582    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-54wvk"
Sep 18 17:23:43.138110 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:43.138017    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lw8gc\" (UniqueName: \"kubernetes.io/projected/cec52271-ee94-49f4-afef-acf23c374ffe-kube-api-access-lw8gc\") pod \"cec52271-ee94-49f4-afef-acf23c374ffe\" (UID: \"cec52271-ee94-49f4-afef-acf23c374ffe\") "
Sep 18 17:23:43.138110 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:43.138065    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5c82\" (UniqueName: \"kubernetes.io/projected/e9c610f6-b3ee-4177-bf56-36f95c991ccb-kube-api-access-w5c82\") pod \"e9c610f6-b3ee-4177-bf56-36f95c991ccb\" (UID: \"e9c610f6-b3ee-4177-bf56-36f95c991ccb\") "
Sep 18 17:23:43.138110 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:43.138090    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cec52271-ee94-49f4-afef-acf23c374ffe-tls-cert\") pod \"cec52271-ee94-49f4-afef-acf23c374ffe\" (UID: \"cec52271-ee94-49f4-afef-acf23c374ffe\") "
Sep 18 17:23:43.140245 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:43.140211    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cec52271-ee94-49f4-afef-acf23c374ffe-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "cec52271-ee94-49f4-afef-acf23c374ffe" (UID: "cec52271-ee94-49f4-afef-acf23c374ffe"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:23:43.140359 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:43.140287    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec52271-ee94-49f4-afef-acf23c374ffe-kube-api-access-lw8gc" (OuterVolumeSpecName: "kube-api-access-lw8gc") pod "cec52271-ee94-49f4-afef-acf23c374ffe" (UID: "cec52271-ee94-49f4-afef-acf23c374ffe"). InnerVolumeSpecName "kube-api-access-lw8gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:23:43.140359 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:43.140317    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c610f6-b3ee-4177-bf56-36f95c991ccb-kube-api-access-w5c82" (OuterVolumeSpecName: "kube-api-access-w5c82") pod "e9c610f6-b3ee-4177-bf56-36f95c991ccb" (UID: "e9c610f6-b3ee-4177-bf56-36f95c991ccb"). InnerVolumeSpecName "kube-api-access-w5c82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:23:43.170400 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:43.170372    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-88d9d4659-8vk4h"
Sep 18 17:23:43.238581 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:43.238547    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw8gc\" (UniqueName: \"kubernetes.io/projected/cec52271-ee94-49f4-afef-acf23c374ffe-kube-api-access-lw8gc\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:23:43.238581 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:43.238579    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5c82\" (UniqueName: \"kubernetes.io/projected/e9c610f6-b3ee-4177-bf56-36f95c991ccb-kube-api-access-w5c82\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:23:43.238738 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:43.238588    2569 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cec52271-ee94-49f4-afef-acf23c374ffe-tls-cert\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:23:43.446920 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:43.446833    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-88d9d4659-8vk4h"]
Sep 18 17:23:43.449881 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:23:43.449831    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod707cc028_82b8_465b_b891_28fd16b18e04.slice/crio-e7cb2d73e8027e74c4c92b0da82d2caaa4b7e62da50ac5cfc39af777a9124135 WatchSource:0}: Error finding container e7cb2d73e8027e74c4c92b0da82d2caaa4b7e62da50ac5cfc39af777a9124135: Status 404 returned error can't find the container with id e7cb2d73e8027e74c4c92b0da82d2caaa4b7e62da50ac5cfc39af777a9124135
Sep 18 17:23:44.043720 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:44.043634    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-54wvk"
Sep 18 17:23:44.043720 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:44.043670    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-khmzm"
Sep 18 17:23:44.043720 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:44.043631    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-88d9d4659-8vk4h" event={"ID":"707cc028-82b8-465b-b891-28fd16b18e04","Type":"ContainerStarted","Data":"dafa4e1ea2593c6d57a629ef6249f429ea12c31292070e977ac7b4d8bf82fdf9"}
Sep 18 17:23:44.044208 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:44.043740    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-88d9d4659-8vk4h" event={"ID":"707cc028-82b8-465b-b891-28fd16b18e04","Type":"ContainerStarted","Data":"e7cb2d73e8027e74c4c92b0da82d2caaa4b7e62da50ac5cfc39af777a9124135"}
Sep 18 17:23:44.062579 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:44.062532    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-88d9d4659-8vk4h" podStartSLOduration=1.6104429580000001 podStartE2EDuration="2.062516874s" totalImagesPullingTime="452.073916ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:23:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:23:44.060388524 +0000 UTC m=+536.754316051" watchObservedRunningTime="2026-09-18 17:23:44.062516874 +0000 UTC m=+536.756444402"
Sep 18 17:23:44.093408 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:44.093374    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6srx9"]
Sep 18 17:23:44.093598 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:44.093573    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-6srx9" podUID="6554c194-ff32-4cf0-9308-86e12a32dd1b" containerName="authorino" containerID="cri-o://6ace29aae66bd87574d2e5d229b0836ce4ab32dd830b8aa32681056a05b2890c" gracePeriod=30
Sep 18 17:23:44.102335 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:44.102303    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-54wvk"]
Sep 18 17:23:44.108856 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:44.108833    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-54wvk"]
Sep 18 17:23:44.131257 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:44.131224    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-khmzm"]
Sep 18 17:23:44.132739 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:44.132714    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-khmzm"]
Sep 18 17:23:44.320708 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:44.320684    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-6srx9"
Sep 18 17:23:44.345162 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:44.345135    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhx58\" (UniqueName: \"kubernetes.io/projected/6554c194-ff32-4cf0-9308-86e12a32dd1b-kube-api-access-dhx58\") pod \"6554c194-ff32-4cf0-9308-86e12a32dd1b\" (UID: \"6554c194-ff32-4cf0-9308-86e12a32dd1b\") "
Sep 18 17:23:44.347166 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:44.347136    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6554c194-ff32-4cf0-9308-86e12a32dd1b-kube-api-access-dhx58" (OuterVolumeSpecName: "kube-api-access-dhx58") pod "6554c194-ff32-4cf0-9308-86e12a32dd1b" (UID: "6554c194-ff32-4cf0-9308-86e12a32dd1b"). InnerVolumeSpecName "kube-api-access-dhx58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:23:44.446595 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:44.446568    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhx58\" (UniqueName: \"kubernetes.io/projected/6554c194-ff32-4cf0-9308-86e12a32dd1b-kube-api-access-dhx58\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:23:45.049153 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:45.049064    2569 generic.go:356] "Generic (PLEG): container finished" podID="6554c194-ff32-4cf0-9308-86e12a32dd1b" containerID="6ace29aae66bd87574d2e5d229b0836ce4ab32dd830b8aa32681056a05b2890c" exitCode=0
Sep 18 17:23:45.049153 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:45.049093    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-6srx9"
Sep 18 17:23:45.049631 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:45.049143    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-6srx9" event={"ID":"6554c194-ff32-4cf0-9308-86e12a32dd1b","Type":"ContainerDied","Data":"6ace29aae66bd87574d2e5d229b0836ce4ab32dd830b8aa32681056a05b2890c"}
Sep 18 17:23:45.049631 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:45.049183    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-6srx9" event={"ID":"6554c194-ff32-4cf0-9308-86e12a32dd1b","Type":"ContainerDied","Data":"73e2edb250daf825266970870f3f2a349be81739a9036d30b42153269453e43e"}
Sep 18 17:23:45.049631 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:45.049200    2569 scope.go:117] "RemoveContainer" containerID="6ace29aae66bd87574d2e5d229b0836ce4ab32dd830b8aa32681056a05b2890c"
Sep 18 17:23:45.057533 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:45.057517    2569 scope.go:117] "RemoveContainer" containerID="6ace29aae66bd87574d2e5d229b0836ce4ab32dd830b8aa32681056a05b2890c"
Sep 18 17:23:45.057810 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:23:45.057791    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ace29aae66bd87574d2e5d229b0836ce4ab32dd830b8aa32681056a05b2890c\": container with ID starting with 6ace29aae66bd87574d2e5d229b0836ce4ab32dd830b8aa32681056a05b2890c not found: ID does not exist" containerID="6ace29aae66bd87574d2e5d229b0836ce4ab32dd830b8aa32681056a05b2890c"
Sep 18 17:23:45.057883 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:45.057819    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ace29aae66bd87574d2e5d229b0836ce4ab32dd830b8aa32681056a05b2890c"} err="failed to get container status \"6ace29aae66bd87574d2e5d229b0836ce4ab32dd830b8aa32681056a05b2890c\": rpc error: code = NotFound desc = could not find container \"6ace29aae66bd87574d2e5d229b0836ce4ab32dd830b8aa32681056a05b2890c\": container with ID starting with 6ace29aae66bd87574d2e5d229b0836ce4ab32dd830b8aa32681056a05b2890c not found: ID does not exist"
Sep 18 17:23:45.072475 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:45.072447    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6srx9"]
Sep 18 17:23:45.076258 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:45.076235    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6srx9"]
Sep 18 17:23:45.764481 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:45.764449    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6554c194-ff32-4cf0-9308-86e12a32dd1b" path="/var/lib/kubelet/pods/6554c194-ff32-4cf0-9308-86e12a32dd1b/volumes"
Sep 18 17:23:45.764802 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:45.764790    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cec52271-ee94-49f4-afef-acf23c374ffe" path="/var/lib/kubelet/pods/cec52271-ee94-49f4-afef-acf23c374ffe/volumes"
Sep 18 17:23:45.765009 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:23:45.764997    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9c610f6-b3ee-4177-bf56-36f95c991ccb" path="/var/lib/kubelet/pods/e9c610f6-b3ee-4177-bf56-36f95c991ccb/volumes"
Sep 18 17:24:54.114326 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:24:54.114288    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-x2z66"]
Sep 18 17:24:54.116631 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:24:54.114745    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6554c194-ff32-4cf0-9308-86e12a32dd1b" containerName="authorino"
Sep 18 17:24:54.116631 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:24:54.114760    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6554c194-ff32-4cf0-9308-86e12a32dd1b" containerName="authorino"
Sep 18 17:24:54.116631 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:24:54.114883    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6554c194-ff32-4cf0-9308-86e12a32dd1b" containerName="authorino"
Sep 18 17:24:54.117754 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:24:54.117737    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x2z66"
Sep 18 17:24:54.120852 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:24:54.120829    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 18 17:24:54.120852 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:24:54.120844    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 18 17:24:54.121418 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:24:54.121396    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-htmgb\""
Sep 18 17:24:54.121509 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:24:54.121424    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 18 17:24:54.130651 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:24:54.130626    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-x2z66"]
Sep 18 17:24:54.236729 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:24:54.236690    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b847607-8f6e-4cd3-94b8-283fbd05fa46-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x2z66\" (UID: \"8b847607-8f6e-4cd3-94b8-283fbd05fa46\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x2z66"
Sep 18 17:24:54.236922 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:24:54.236735    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b847607-8f6e-4cd3-94b8-283fbd05fa46-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x2z66\" (UID: \"8b847607-8f6e-4cd3-94b8-283fbd05fa46\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x2z66"
Sep 18 17:24:54.236922 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:24:54.236760    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b847607-8f6e-4cd3-94b8-283fbd05fa46-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x2z66\" (UID: \"8b847607-8f6e-4cd3-94b8-283fbd05fa46\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x2z66"
Sep 18 17:24:54.236922 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:24:54.236829    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b847607-8f6e-4cd3-94b8-283fbd05fa46-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x2z66\" (UID: \"8b847607-8f6e-4cd3-94b8-283fbd05fa46\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x2z66"
Sep 18 17:24:54.236922 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:24:54.236850    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgfxs\" (UniqueName: \"kubernetes.io/projected/8b847607-8f6e-4cd3-94b8-283fbd05fa46-kube-api-access-qgfxs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x2z66\" (UID: \"8b847607-8f6e-4cd3-94b8-283fbd05fa46\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x2z66"
Sep 18 17:24:54.236922 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:24:54.236889    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b847607-8f6e-4cd3-94b8-283fbd05fa46-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x2z66\" (UID: \"8b847607-8f6e-4cd3-94b8-283fbd05fa46\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x2z66"
Sep 18 17:24:54.338025 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:24:54.337991    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b847607-8f6e-4cd3-94b8-283fbd05fa46-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x2z66\" (UID: \"8b847607-8f6e-4cd3-94b8-283fbd05fa46\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x2z66"
Sep 18 17:24:54.338154 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:24:54.338042    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b847607-8f6e-4cd3-94b8-283fbd05fa46-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x2z66\" (UID: \"8b847607-8f6e-4cd3-94b8-283fbd05fa46\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x2z66"
Sep 18 17:24:54.338154 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:24:54.338077    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b847607-8f6e-4cd3-94b8-283fbd05fa46-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x2z66\" (UID: \"8b847607-8f6e-4cd3-94b8-283fbd05fa46\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x2z66"
Sep 18 17:24:54.338154 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:24:54.338102    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b847607-8f6e-4cd3-94b8-283fbd05fa46-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x2z66\" (UID: \"8b847607-8f6e-4cd3-94b8-283fbd05fa46\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x2z66"
Sep 18 17:24:54.338154 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:24:54.338123    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b847607-8f6e-4cd3-94b8-283fbd05fa46-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x2z66\" (UID: \"8b847607-8f6e-4cd3-94b8-283fbd05fa46\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x2z66"
Sep 18 17:24:54.338154 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:24:54.338144    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgfxs\" (UniqueName: \"kubernetes.io/projected/8b847607-8f6e-4cd3-94b8-283fbd05fa46-kube-api-access-qgfxs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x2z66\" (UID: \"8b847607-8f6e-4cd3-94b8-283fbd05fa46\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x2z66"
Sep 18 17:24:54.338470 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:24:54.338446    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b847607-8f6e-4cd3-94b8-283fbd05fa46-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x2z66\" (UID: \"8b847607-8f6e-4cd3-94b8-283fbd05fa46\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x2z66"
Sep 18 17:24:54.338535 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:24:54.338488    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b847607-8f6e-4cd3-94b8-283fbd05fa46-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x2z66\" (UID: \"8b847607-8f6e-4cd3-94b8-283fbd05fa46\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x2z66"
Sep 18 17:24:54.338535 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:24:54.338514    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b847607-8f6e-4cd3-94b8-283fbd05fa46-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x2z66\" (UID: \"8b847607-8f6e-4cd3-94b8-283fbd05fa46\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x2z66"
Sep 18 17:24:54.340424 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:24:54.340401    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b847607-8f6e-4cd3-94b8-283fbd05fa46-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x2z66\" (UID: \"8b847607-8f6e-4cd3-94b8-283fbd05fa46\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x2z66"
Sep 18 17:24:54.340692 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:24:54.340673    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b847607-8f6e-4cd3-94b8-283fbd05fa46-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x2z66\" (UID: \"8b847607-8f6e-4cd3-94b8-283fbd05fa46\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x2z66"
Sep 18 17:24:54.348817 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:24:54.348797    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgfxs\" (UniqueName: \"kubernetes.io/projected/8b847607-8f6e-4cd3-94b8-283fbd05fa46-kube-api-access-qgfxs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x2z66\" (UID: \"8b847607-8f6e-4cd3-94b8-283fbd05fa46\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x2z66"
Sep 18 17:24:54.430786 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:24:54.430686    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x2z66"
Sep 18 17:24:54.718014 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:24:54.717930    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-x2z66"]
Sep 18 17:24:54.721022 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:24:54.720989    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b847607_8f6e_4cd3_94b8_283fbd05fa46.slice/crio-c5191e0db25dfe1a4888adda71e653309bbea9d9ac06d8e625fcb9a75f1dc8fe WatchSource:0}: Error finding container c5191e0db25dfe1a4888adda71e653309bbea9d9ac06d8e625fcb9a75f1dc8fe: Status 404 returned error can't find the container with id c5191e0db25dfe1a4888adda71e653309bbea9d9ac06d8e625fcb9a75f1dc8fe
Sep 18 17:24:55.358018 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:24:55.357981    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x2z66" event={"ID":"8b847607-8f6e-4cd3-94b8-283fbd05fa46","Type":"ContainerStarted","Data":"c5191e0db25dfe1a4888adda71e653309bbea9d9ac06d8e625fcb9a75f1dc8fe"}
Sep 18 17:24:56.363396 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:24:56.363353    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x2z66" event={"ID":"8b847607-8f6e-4cd3-94b8-283fbd05fa46","Type":"ContainerStarted","Data":"d3be095362638a438d0f08abe4e79f92324a67dda05f0e0b126558fc8c75703e"}
Sep 18 17:24:56.382947 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:24:56.382899    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x2z66" podStartSLOduration=1.39777559 podStartE2EDuration="2.382856928s" totalImagesPullingTime="985.081338ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:24:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:24:56.38264899 +0000 UTC m=+609.076576546" watchObservedRunningTime="2026-09-18 17:24:56.382856928 +0000 UTC m=+609.076784456"
Sep 18 17:25:04.431645 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:04.431612    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x2z66"
Sep 18 17:25:04.432060 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:04.431913    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x2z66"
Sep 18 17:25:04.444479 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:04.444450    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x2z66"
Sep 18 17:25:05.411904 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:05.411845    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x2z66"
Sep 18 17:25:10.132927 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:10.132829    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz"]
Sep 18 17:25:10.136600 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:10.136582    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz"
Sep 18 17:25:10.139264 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:10.139246    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 18 17:25:10.149804 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:10.149781    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz"]
Sep 18 17:25:10.279731 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:10.279688    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d104c24-7c53-4359-b2a4-20f7e2805a9a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz\" (UID: \"5d104c24-7c53-4359-b2a4-20f7e2805a9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz"
Sep 18 17:25:10.279963 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:10.279776    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d104c24-7c53-4359-b2a4-20f7e2805a9a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz\" (UID: \"5d104c24-7c53-4359-b2a4-20f7e2805a9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz"
Sep 18 17:25:10.279963 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:10.279820    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x624f\" (UniqueName: \"kubernetes.io/projected/5d104c24-7c53-4359-b2a4-20f7e2805a9a-kube-api-access-x624f\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz\" (UID: \"5d104c24-7c53-4359-b2a4-20f7e2805a9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz"
Sep 18 17:25:10.279963 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:10.279905    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d104c24-7c53-4359-b2a4-20f7e2805a9a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz\" (UID: \"5d104c24-7c53-4359-b2a4-20f7e2805a9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz"
Sep 18 17:25:10.279963 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:10.279937    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d104c24-7c53-4359-b2a4-20f7e2805a9a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz\" (UID: \"5d104c24-7c53-4359-b2a4-20f7e2805a9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz"
Sep 18 17:25:10.280149 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:10.280020    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d104c24-7c53-4359-b2a4-20f7e2805a9a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz\" (UID: \"5d104c24-7c53-4359-b2a4-20f7e2805a9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz"
Sep 18 17:25:10.380993 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:10.380958    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d104c24-7c53-4359-b2a4-20f7e2805a9a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz\" (UID: \"5d104c24-7c53-4359-b2a4-20f7e2805a9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz"
Sep 18 17:25:10.381112 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:10.381004    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x624f\" (UniqueName: \"kubernetes.io/projected/5d104c24-7c53-4359-b2a4-20f7e2805a9a-kube-api-access-x624f\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz\" (UID: \"5d104c24-7c53-4359-b2a4-20f7e2805a9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz"
Sep 18 17:25:10.381112 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:10.381032    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d104c24-7c53-4359-b2a4-20f7e2805a9a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz\" (UID: \"5d104c24-7c53-4359-b2a4-20f7e2805a9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz"
Sep 18 17:25:10.381112 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:10.381064    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d104c24-7c53-4359-b2a4-20f7e2805a9a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz\" (UID: \"5d104c24-7c53-4359-b2a4-20f7e2805a9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz"
Sep 18 17:25:10.381235 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:10.381131    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d104c24-7c53-4359-b2a4-20f7e2805a9a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz\" (UID: \"5d104c24-7c53-4359-b2a4-20f7e2805a9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz"
Sep 18 17:25:10.381235 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:10.381173    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d104c24-7c53-4359-b2a4-20f7e2805a9a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz\" (UID: \"5d104c24-7c53-4359-b2a4-20f7e2805a9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz"
Sep 18 17:25:10.381445 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:10.381420    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d104c24-7c53-4359-b2a4-20f7e2805a9a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz\" (UID: \"5d104c24-7c53-4359-b2a4-20f7e2805a9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz"
Sep 18 17:25:10.381530 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:10.381461    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d104c24-7c53-4359-b2a4-20f7e2805a9a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz\" (UID: \"5d104c24-7c53-4359-b2a4-20f7e2805a9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz"
Sep 18 17:25:10.381530 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:10.381512    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d104c24-7c53-4359-b2a4-20f7e2805a9a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz\" (UID: \"5d104c24-7c53-4359-b2a4-20f7e2805a9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz"
Sep 18 17:25:10.383349 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:10.383300    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d104c24-7c53-4359-b2a4-20f7e2805a9a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz\" (UID: \"5d104c24-7c53-4359-b2a4-20f7e2805a9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz"
Sep 18 17:25:10.383656 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:10.383637    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d104c24-7c53-4359-b2a4-20f7e2805a9a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz\" (UID: \"5d104c24-7c53-4359-b2a4-20f7e2805a9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz"
Sep 18 17:25:10.391073 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:10.391048    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x624f\" (UniqueName: \"kubernetes.io/projected/5d104c24-7c53-4359-b2a4-20f7e2805a9a-kube-api-access-x624f\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz\" (UID: \"5d104c24-7c53-4359-b2a4-20f7e2805a9a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz"
Sep 18 17:25:10.446547 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:10.446518    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz"
Sep 18 17:25:10.724662 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:10.724634    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz"]
Sep 18 17:25:10.726311 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:25:10.726276    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d104c24_7c53_4359_b2a4_20f7e2805a9a.slice/crio-9b09186f909a09d9eaf2ce57c687fcd2ced6f01c2729d94babc508a5a5c73c68 WatchSource:0}: Error finding container 9b09186f909a09d9eaf2ce57c687fcd2ced6f01c2729d94babc508a5a5c73c68: Status 404 returned error can't find the container with id 9b09186f909a09d9eaf2ce57c687fcd2ced6f01c2729d94babc508a5a5c73c68
Sep 18 17:25:10.727572 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:10.727551    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 17:25:11.431517 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:11.431478    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz" event={"ID":"5d104c24-7c53-4359-b2a4-20f7e2805a9a","Type":"ContainerStarted","Data":"e025d692ae78db7f192c7b550f018896a7454422d558c26579567b72938cade7"}
Sep 18 17:25:11.431919 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:11.431538    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz" event={"ID":"5d104c24-7c53-4359-b2a4-20f7e2805a9a","Type":"ContainerStarted","Data":"9b09186f909a09d9eaf2ce57c687fcd2ced6f01c2729d94babc508a5a5c73c68"}
Sep 18 17:25:11.456123 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:11.456076    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz" podStartSLOduration=1.275652627 podStartE2EDuration="1.456056517s" totalImagesPullingTime="180.40389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:25:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:25:11.45479787 +0000 UTC m=+624.148725419" watchObservedRunningTime="2026-09-18 17:25:11.456056517 +0000 UTC m=+624.149984045"
Sep 18 17:25:20.447442 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:20.447402    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz"
Sep 18 17:25:20.447945 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:20.447468    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz"
Sep 18 17:25:20.459616 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:20.459592    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz"
Sep 18 17:25:20.482100 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:20.482064    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz"
Sep 18 17:25:30.839488 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:30.839455    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-757lh"]
Sep 18 17:25:30.842906 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:30.842888    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-757lh"
Sep 18 17:25:30.846081 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:30.846062    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 18 17:25:30.848646 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:30.848624    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-757lh\" (UID: \"55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-757lh"
Sep 18 17:25:30.848764 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:30.848689    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-757lh\" (UID: \"55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-757lh"
Sep 18 17:25:30.848764 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:30.848732    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-757lh\" (UID: \"55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-757lh"
Sep 18 17:25:30.848906 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:30.848816    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snjhc\" (UniqueName: \"kubernetes.io/projected/55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23-kube-api-access-snjhc\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-757lh\" (UID: \"55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-757lh"
Sep 18 17:25:30.848906 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:30.848878    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-757lh\" (UID: \"55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-757lh"
Sep 18 17:25:30.848998 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:30.848903    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-757lh\" (UID: \"55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-757lh"
Sep 18 17:25:30.868200 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:30.868170    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-757lh"]
Sep 18 17:25:30.949912 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:30.949858    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-757lh\" (UID: \"55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-757lh"
Sep 18 17:25:30.949912 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:30.949914    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-757lh\" (UID: \"55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-757lh"
Sep 18 17:25:30.950140 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:30.949965    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snjhc\" (UniqueName: \"kubernetes.io/projected/55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23-kube-api-access-snjhc\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-757lh\" (UID: \"55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-757lh"
Sep 18 17:25:30.950140 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:30.949990    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-757lh\" (UID: \"55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-757lh"
Sep 18 17:25:30.950140 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:30.950008    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-757lh\" (UID: \"55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-757lh"
Sep 18 17:25:30.950140 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:30.950028    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-757lh\" (UID: \"55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-757lh"
Sep 18 17:25:30.950385 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:30.950361    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-757lh\" (UID: \"55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-757lh"
Sep 18 17:25:30.950447 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:30.950396    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-757lh\" (UID: \"55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-757lh"
Sep 18 17:25:30.950485 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:30.950440    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-757lh\" (UID: \"55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-757lh"
Sep 18 17:25:30.952237 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:30.952210    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-757lh\" (UID: \"55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-757lh"
Sep 18 17:25:30.952471 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:30.952455    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-757lh\" (UID: \"55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-757lh"
Sep 18 17:25:30.962355 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:30.962326    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snjhc\" (UniqueName: \"kubernetes.io/projected/55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23-kube-api-access-snjhc\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-757lh\" (UID: \"55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-757lh"
Sep 18 17:25:31.151953 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:31.151848    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-757lh"
Sep 18 17:25:31.433409 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:31.433385    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-757lh"]
Sep 18 17:25:31.434497 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:25:31.434468    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55cebd16_d2c2_4a44_ad3a_9a0c3f0daf23.slice/crio-634c411a8452f496350cb479a988d53806ba833751ecd8411a205e3b75a8bbac WatchSource:0}: Error finding container 634c411a8452f496350cb479a988d53806ba833751ecd8411a205e3b75a8bbac: Status 404 returned error can't find the container with id 634c411a8452f496350cb479a988d53806ba833751ecd8411a205e3b75a8bbac
Sep 18 17:25:31.524484 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:31.524452    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-757lh" event={"ID":"55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23","Type":"ContainerStarted","Data":"634c411a8452f496350cb479a988d53806ba833751ecd8411a205e3b75a8bbac"}
Sep 18 17:25:32.530762 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:32.530719    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-757lh" event={"ID":"55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23","Type":"ContainerStarted","Data":"66a1c750651bfb7d35b6d992d5f988d24b42ad6ee15e3cf2a9168d85db7e7256"}
Sep 18 17:25:32.563191 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:32.563146    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-757lh" podStartSLOduration=2.359826259 podStartE2EDuration="2.563130138s" totalImagesPullingTime="203.303879ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:25:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:25:32.561461314 +0000 UTC m=+645.255388854" watchObservedRunningTime="2026-09-18 17:25:32.563130138 +0000 UTC m=+645.257057666"
Sep 18 17:25:41.152165 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:41.152126    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-757lh"
Sep 18 17:25:41.152552 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:41.152185    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-757lh"
Sep 18 17:25:41.164471 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:41.164441    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-757lh"
Sep 18 17:25:41.587902 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:25:41.587858    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-757lh"
Sep 18 17:26:22.212781 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:22.212746    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-dc59d87f8-q2x86"]
Sep 18 17:26:22.216008 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:22.215986    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-dc59d87f8-q2x86"
Sep 18 17:26:22.226019 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:22.225991    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-dc59d87f8-q2x86"]
Sep 18 17:26:22.374354 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:22.374320    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f431da55-9418-45da-be88-cd303b6af64a-tls-cert\") pod \"authorino-dc59d87f8-q2x86\" (UID: \"f431da55-9418-45da-be88-cd303b6af64a\") " pod="kuadrant-system/authorino-dc59d87f8-q2x86"
Sep 18 17:26:22.374511 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:22.374383    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qh5v\" (UniqueName: \"kubernetes.io/projected/f431da55-9418-45da-be88-cd303b6af64a-kube-api-access-2qh5v\") pod \"authorino-dc59d87f8-q2x86\" (UID: \"f431da55-9418-45da-be88-cd303b6af64a\") " pod="kuadrant-system/authorino-dc59d87f8-q2x86"
Sep 18 17:26:22.475552 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:22.475479    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qh5v\" (UniqueName: \"kubernetes.io/projected/f431da55-9418-45da-be88-cd303b6af64a-kube-api-access-2qh5v\") pod \"authorino-dc59d87f8-q2x86\" (UID: \"f431da55-9418-45da-be88-cd303b6af64a\") " pod="kuadrant-system/authorino-dc59d87f8-q2x86"
Sep 18 17:26:22.475552 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:22.475547    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f431da55-9418-45da-be88-cd303b6af64a-tls-cert\") pod \"authorino-dc59d87f8-q2x86\" (UID: \"f431da55-9418-45da-be88-cd303b6af64a\") " pod="kuadrant-system/authorino-dc59d87f8-q2x86"
Sep 18 17:26:22.477986 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:22.477961    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f431da55-9418-45da-be88-cd303b6af64a-tls-cert\") pod \"authorino-dc59d87f8-q2x86\" (UID: \"f431da55-9418-45da-be88-cd303b6af64a\") " pod="kuadrant-system/authorino-dc59d87f8-q2x86"
Sep 18 17:26:22.492047 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:22.492019    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qh5v\" (UniqueName: \"kubernetes.io/projected/f431da55-9418-45da-be88-cd303b6af64a-kube-api-access-2qh5v\") pod \"authorino-dc59d87f8-q2x86\" (UID: \"f431da55-9418-45da-be88-cd303b6af64a\") " pod="kuadrant-system/authorino-dc59d87f8-q2x86"
Sep 18 17:26:22.524963 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:22.524942    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-dc59d87f8-q2x86"
Sep 18 17:26:22.795567 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:22.795531    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-dc59d87f8-q2x86"]
Sep 18 17:26:22.798422 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:26:22.798393    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf431da55_9418_45da_be88_cd303b6af64a.slice/crio-4133c5a9624cd32c2692e7614cce4ee045178ad462ae8c8bf470f05dd1aa3daa WatchSource:0}: Error finding container 4133c5a9624cd32c2692e7614cce4ee045178ad462ae8c8bf470f05dd1aa3daa: Status 404 returned error can't find the container with id 4133c5a9624cd32c2692e7614cce4ee045178ad462ae8c8bf470f05dd1aa3daa
Sep 18 17:26:23.767945 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:23.767900    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-dc59d87f8-q2x86" event={"ID":"f431da55-9418-45da-be88-cd303b6af64a","Type":"ContainerStarted","Data":"4cdfa1faa948c78eccc2a4918f5a824f4e279dfe444de4733955c5f9f820ce1d"}
Sep 18 17:26:23.767945 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:23.767952    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-dc59d87f8-q2x86" event={"ID":"f431da55-9418-45da-be88-cd303b6af64a","Type":"ContainerStarted","Data":"4133c5a9624cd32c2692e7614cce4ee045178ad462ae8c8bf470f05dd1aa3daa"}
Sep 18 17:26:23.790240 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:23.790025    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-dc59d87f8-q2x86" podStartSLOduration=1.064744552 podStartE2EDuration="1.790002906s" totalImagesPullingTime="725.258354ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:26:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:26:23.789681147 +0000 UTC m=+696.483608667" watchObservedRunningTime="2026-09-18 17:26:23.790002906 +0000 UTC m=+696.483930435"
Sep 18 17:26:23.825116 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:23.825068    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-88d9d4659-8vk4h"]
Sep 18 17:26:23.825803 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:23.825545    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-88d9d4659-8vk4h" podUID="707cc028-82b8-465b-b891-28fd16b18e04" containerName="authorino" containerID="cri-o://dafa4e1ea2593c6d57a629ef6249f429ea12c31292070e977ac7b4d8bf82fdf9" gracePeriod=30
Sep 18 17:26:24.069819 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:24.069793    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-88d9d4659-8vk4h"
Sep 18 17:26:24.192653 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:24.192605    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7njch\" (UniqueName: \"kubernetes.io/projected/707cc028-82b8-465b-b891-28fd16b18e04-kube-api-access-7njch\") pod \"707cc028-82b8-465b-b891-28fd16b18e04\" (UID: \"707cc028-82b8-465b-b891-28fd16b18e04\") "
Sep 18 17:26:24.192936 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:24.192720    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/707cc028-82b8-465b-b891-28fd16b18e04-tls-cert\") pod \"707cc028-82b8-465b-b891-28fd16b18e04\" (UID: \"707cc028-82b8-465b-b891-28fd16b18e04\") "
Sep 18 17:26:24.194620 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:24.194590    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/707cc028-82b8-465b-b891-28fd16b18e04-kube-api-access-7njch" (OuterVolumeSpecName: "kube-api-access-7njch") pod "707cc028-82b8-465b-b891-28fd16b18e04" (UID: "707cc028-82b8-465b-b891-28fd16b18e04"). InnerVolumeSpecName "kube-api-access-7njch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:26:24.202677 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:24.202645    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/707cc028-82b8-465b-b891-28fd16b18e04-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "707cc028-82b8-465b-b891-28fd16b18e04" (UID: "707cc028-82b8-465b-b891-28fd16b18e04"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:26:24.294246 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:24.294208    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7njch\" (UniqueName: \"kubernetes.io/projected/707cc028-82b8-465b-b891-28fd16b18e04-kube-api-access-7njch\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:26:24.294246 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:24.294238    2569 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/707cc028-82b8-465b-b891-28fd16b18e04-tls-cert\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:26:24.775756 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:24.775721    2569 generic.go:356] "Generic (PLEG): container finished" podID="707cc028-82b8-465b-b891-28fd16b18e04" containerID="dafa4e1ea2593c6d57a629ef6249f429ea12c31292070e977ac7b4d8bf82fdf9" exitCode=0
Sep 18 17:26:24.776208 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:24.775767    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-88d9d4659-8vk4h"
Sep 18 17:26:24.776208 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:24.775817    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-88d9d4659-8vk4h" event={"ID":"707cc028-82b8-465b-b891-28fd16b18e04","Type":"ContainerDied","Data":"dafa4e1ea2593c6d57a629ef6249f429ea12c31292070e977ac7b4d8bf82fdf9"}
Sep 18 17:26:24.776208 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:24.775880    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-88d9d4659-8vk4h" event={"ID":"707cc028-82b8-465b-b891-28fd16b18e04","Type":"ContainerDied","Data":"e7cb2d73e8027e74c4c92b0da82d2caaa4b7e62da50ac5cfc39af777a9124135"}
Sep 18 17:26:24.776208 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:24.775905    2569 scope.go:117] "RemoveContainer" containerID="dafa4e1ea2593c6d57a629ef6249f429ea12c31292070e977ac7b4d8bf82fdf9"
Sep 18 17:26:24.787305 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:24.787279    2569 scope.go:117] "RemoveContainer" containerID="dafa4e1ea2593c6d57a629ef6249f429ea12c31292070e977ac7b4d8bf82fdf9"
Sep 18 17:26:24.787828 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:26:24.787802    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dafa4e1ea2593c6d57a629ef6249f429ea12c31292070e977ac7b4d8bf82fdf9\": container with ID starting with dafa4e1ea2593c6d57a629ef6249f429ea12c31292070e977ac7b4d8bf82fdf9 not found: ID does not exist" containerID="dafa4e1ea2593c6d57a629ef6249f429ea12c31292070e977ac7b4d8bf82fdf9"
Sep 18 17:26:24.787960 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:24.787835    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dafa4e1ea2593c6d57a629ef6249f429ea12c31292070e977ac7b4d8bf82fdf9"} err="failed to get container status \"dafa4e1ea2593c6d57a629ef6249f429ea12c31292070e977ac7b4d8bf82fdf9\": rpc error: code = NotFound desc = could not find container \"dafa4e1ea2593c6d57a629ef6249f429ea12c31292070e977ac7b4d8bf82fdf9\": container with ID starting with dafa4e1ea2593c6d57a629ef6249f429ea12c31292070e977ac7b4d8bf82fdf9 not found: ID does not exist"
Sep 18 17:26:24.810960 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:24.810931    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-88d9d4659-8vk4h"]
Sep 18 17:26:24.821361 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:24.821336    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-88d9d4659-8vk4h"]
Sep 18 17:26:25.769975 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:25.769942    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="707cc028-82b8-465b-b891-28fd16b18e04" path="/var/lib/kubelet/pods/707cc028-82b8-465b-b891-28fd16b18e04/volumes"
Sep 18 17:26:46.212255 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:46.212178    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7735-cbdc71"]
Sep 18 17:26:46.212655 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:46.212484    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="707cc028-82b8-465b-b891-28fd16b18e04" containerName="authorino"
Sep 18 17:26:46.212655 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:46.212495    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="707cc028-82b8-465b-b891-28fd16b18e04" containerName="authorino"
Sep 18 17:26:46.212655 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:46.212564    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="707cc028-82b8-465b-b891-28fd16b18e04" containerName="authorino"
Sep 18 17:26:46.215319 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:46.215301    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-cbdc71"
Sep 18 17:26:46.218399 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:46.218374    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mjflw\""
Sep 18 17:26:46.227112 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:46.227090    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7735-cbdc71"]
Sep 18 17:26:46.266411 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:46.266385    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgsdm\" (UniqueName: \"kubernetes.io/projected/eab8b59e-36e3-4eeb-a3ad-697528968238-kube-api-access-cgsdm\") pod \"test-curl-7735-cbdc71\" (UID: \"eab8b59e-36e3-4eeb-a3ad-697528968238\") " pod="opendatahub/test-curl-7735-cbdc71"
Sep 18 17:26:46.366909 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:46.366854    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgsdm\" (UniqueName: \"kubernetes.io/projected/eab8b59e-36e3-4eeb-a3ad-697528968238-kube-api-access-cgsdm\") pod \"test-curl-7735-cbdc71\" (UID: \"eab8b59e-36e3-4eeb-a3ad-697528968238\") " pod="opendatahub/test-curl-7735-cbdc71"
Sep 18 17:26:46.377070 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:46.377038    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgsdm\" (UniqueName: \"kubernetes.io/projected/eab8b59e-36e3-4eeb-a3ad-697528968238-kube-api-access-cgsdm\") pod \"test-curl-7735-cbdc71\" (UID: \"eab8b59e-36e3-4eeb-a3ad-697528968238\") " pod="opendatahub/test-curl-7735-cbdc71"
Sep 18 17:26:46.524210 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:46.524170    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-cbdc71"
Sep 18 17:26:46.805986 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:46.805962    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7735-cbdc71"]
Sep 18 17:26:46.808530 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:26:46.808501    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeab8b59e_36e3_4eeb_a3ad_697528968238.slice/crio-d2bafaf10167e069be21b1add8f97c458f1ea0373735f64bcd2c7158c1236443 WatchSource:0}: Error finding container d2bafaf10167e069be21b1add8f97c458f1ea0373735f64bcd2c7158c1236443: Status 404 returned error can't find the container with id d2bafaf10167e069be21b1add8f97c458f1ea0373735f64bcd2c7158c1236443
Sep 18 17:26:46.872986 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:46.872957    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7735-cbdc71" event={"ID":"eab8b59e-36e3-4eeb-a3ad-697528968238","Type":"ContainerStarted","Data":"d2bafaf10167e069be21b1add8f97c458f1ea0373735f64bcd2c7158c1236443"}
Sep 18 17:26:49.890493 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:49.890454    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7735-cbdc71" event={"ID":"eab8b59e-36e3-4eeb-a3ad-697528968238","Type":"ContainerStarted","Data":"291e394d7d5c66c483e92bf40e4cd6dc0faa9b038731ee1be1f37efcf2385d02"}
Sep 18 17:26:49.907953 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:49.907905    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7735-cbdc71" podStartSLOduration=1.612710217 podStartE2EDuration="3.907891503s" totalImagesPullingTime="2.295181286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:26:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:26:49.90636187 +0000 UTC m=+722.600289397" watchObservedRunningTime="2026-09-18 17:26:49.907891503 +0000 UTC m=+722.601819028"
Sep 18 17:26:50.515423 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:50.515387    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7735-cbdc71"]
Sep 18 17:26:50.520477 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:50.520452    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7735-cbdc71"]
Sep 18 17:26:50.644832 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:50.644744    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7735-813782"]
Sep 18 17:26:50.645153 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:50.645134    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eab8b59e-36e3-4eeb-a3ad-697528968238" containerName="test-curl-7735-cbdc71"
Sep 18 17:26:50.645248 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:50.645156    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="eab8b59e-36e3-4eeb-a3ad-697528968238" containerName="test-curl-7735-cbdc71"
Sep 18 17:26:50.645248 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:50.645243    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="eab8b59e-36e3-4eeb-a3ad-697528968238" containerName="test-curl-7735-cbdc71"
Sep 18 17:26:50.648229 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:50.648209    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-813782"
Sep 18 17:26:50.655206 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:50.654930    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7735-813782"]
Sep 18 17:26:50.707671 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:50.707630    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pplgf\" (UniqueName: \"kubernetes.io/projected/c27e84e1-d8ce-4add-baab-92a3fd759c31-kube-api-access-pplgf\") pod \"test-curl-7735-813782\" (UID: \"c27e84e1-d8ce-4add-baab-92a3fd759c31\") " pod="opendatahub/test-curl-7735-813782"
Sep 18 17:26:50.808282 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:50.808242    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pplgf\" (UniqueName: \"kubernetes.io/projected/c27e84e1-d8ce-4add-baab-92a3fd759c31-kube-api-access-pplgf\") pod \"test-curl-7735-813782\" (UID: \"c27e84e1-d8ce-4add-baab-92a3fd759c31\") " pod="opendatahub/test-curl-7735-813782"
Sep 18 17:26:50.821389 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:50.821359    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pplgf\" (UniqueName: \"kubernetes.io/projected/c27e84e1-d8ce-4add-baab-92a3fd759c31-kube-api-access-pplgf\") pod \"test-curl-7735-813782\" (UID: \"c27e84e1-d8ce-4add-baab-92a3fd759c31\") " pod="opendatahub/test-curl-7735-813782"
Sep 18 17:26:50.895179 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:26:50.895109    2569 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7735-cbdc71" container="test-curl-7735-cbdc71"
Sep 18 17:26:50.958234 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:50.958199    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-813782"
Sep 18 17:26:50.990741 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:50.990699    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-636b80-94f555b66-ws742"]
Sep 18 17:26:50.994120 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:50.994099    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636b80-94f555b66-ws742"
Sep 18 17:26:50.996693 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:50.996670    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-636b80\""
Sep 18 17:26:51.003294 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:51.003273    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-636b80-94f555b66-ws742"]
Sep 18 17:26:51.015111 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:51.015087    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-636b80-f85c7966c-vp6zj"]
Sep 18 17:26:51.018281 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:51.018263    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-636b80-f85c7966c-vp6zj"
Sep 18 17:26:51.021513 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:51.021465    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-636b80-dockercfg-5xdk6\""
Sep 18 17:26:51.037092 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:51.037069    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-636b80-f85c7966c-vp6zj"]
Sep 18 17:26:51.112031 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:51.111998    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz4cr\" (UniqueName: \"kubernetes.io/projected/1b4e16b6-d717-4a39-a879-d34d21f70c67-kube-api-access-pz4cr\") pod \"payload-pre-processing-e2e-inf-a-636b80-94f555b66-ws742\" (UID: \"1b4e16b6-d717-4a39-a879-d34d21f70c67\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636b80-94f555b66-ws742"
Sep 18 17:26:51.112189 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:51.112064    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b4e16b6-d717-4a39-a879-d34d21f70c67-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-636b80-94f555b66-ws742\" (UID: \"1b4e16b6-d717-4a39-a879-d34d21f70c67\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636b80-94f555b66-ws742"
Sep 18 17:26:51.112189 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:51.112151    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18c90c69-6f7a-4738-a2d3-01ad0cc83d38-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-636b80-f85c7966c-vp6zj\" (UID: \"18c90c69-6f7a-4738-a2d3-01ad0cc83d38\") " pod="openshift-ingress/payload-processing-e2e-inf-a-636b80-f85c7966c-vp6zj"
Sep 18 17:26:51.112259 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:51.112201    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnp24\" (UniqueName: \"kubernetes.io/projected/18c90c69-6f7a-4738-a2d3-01ad0cc83d38-kube-api-access-lnp24\") pod \"payload-processing-e2e-inf-a-636b80-f85c7966c-vp6zj\" (UID: \"18c90c69-6f7a-4738-a2d3-01ad0cc83d38\") " pod="openshift-ingress/payload-processing-e2e-inf-a-636b80-f85c7966c-vp6zj"
Sep 18 17:26:51.213167 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:51.213094    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b4e16b6-d717-4a39-a879-d34d21f70c67-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-636b80-94f555b66-ws742\" (UID: \"1b4e16b6-d717-4a39-a879-d34d21f70c67\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636b80-94f555b66-ws742"
Sep 18 17:26:51.213167 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:51.213139    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18c90c69-6f7a-4738-a2d3-01ad0cc83d38-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-636b80-f85c7966c-vp6zj\" (UID: \"18c90c69-6f7a-4738-a2d3-01ad0cc83d38\") " pod="openshift-ingress/payload-processing-e2e-inf-a-636b80-f85c7966c-vp6zj"
Sep 18 17:26:51.213167 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:51.213163    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnp24\" (UniqueName: \"kubernetes.io/projected/18c90c69-6f7a-4738-a2d3-01ad0cc83d38-kube-api-access-lnp24\") pod \"payload-processing-e2e-inf-a-636b80-f85c7966c-vp6zj\" (UID: \"18c90c69-6f7a-4738-a2d3-01ad0cc83d38\") " pod="openshift-ingress/payload-processing-e2e-inf-a-636b80-f85c7966c-vp6zj"
Sep 18 17:26:51.213406 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:51.213211    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz4cr\" (UniqueName: \"kubernetes.io/projected/1b4e16b6-d717-4a39-a879-d34d21f70c67-kube-api-access-pz4cr\") pod \"payload-pre-processing-e2e-inf-a-636b80-94f555b66-ws742\" (UID: \"1b4e16b6-d717-4a39-a879-d34d21f70c67\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636b80-94f555b66-ws742"
Sep 18 17:26:51.213951 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:51.213925    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18c90c69-6f7a-4738-a2d3-01ad0cc83d38-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-636b80-f85c7966c-vp6zj\" (UID: \"18c90c69-6f7a-4738-a2d3-01ad0cc83d38\") " pod="openshift-ingress/payload-processing-e2e-inf-a-636b80-f85c7966c-vp6zj"
Sep 18 17:26:51.214084 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:51.213960    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b4e16b6-d717-4a39-a879-d34d21f70c67-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-636b80-94f555b66-ws742\" (UID: \"1b4e16b6-d717-4a39-a879-d34d21f70c67\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636b80-94f555b66-ws742"
Sep 18 17:26:51.226917 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:51.226894    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz4cr\" (UniqueName: \"kubernetes.io/projected/1b4e16b6-d717-4a39-a879-d34d21f70c67-kube-api-access-pz4cr\") pod \"payload-pre-processing-e2e-inf-a-636b80-94f555b66-ws742\" (UID: \"1b4e16b6-d717-4a39-a879-d34d21f70c67\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636b80-94f555b66-ws742"
Sep 18 17:26:51.248503 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:51.248483    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnp24\" (UniqueName: \"kubernetes.io/projected/18c90c69-6f7a-4738-a2d3-01ad0cc83d38-kube-api-access-lnp24\") pod \"payload-processing-e2e-inf-a-636b80-f85c7966c-vp6zj\" (UID: \"18c90c69-6f7a-4738-a2d3-01ad0cc83d38\") " pod="openshift-ingress/payload-processing-e2e-inf-a-636b80-f85c7966c-vp6zj"
Sep 18 17:26:51.248590 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:51.248492    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7735-813782"]
Sep 18 17:26:51.250629 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:26:51.250599    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc27e84e1_d8ce_4add_baab_92a3fd759c31.slice/crio-30706e1dd75d908306274cd14680e85eb720937737d5c0a39ebc873f5f3933a7 WatchSource:0}: Error finding container 30706e1dd75d908306274cd14680e85eb720937737d5c0a39ebc873f5f3933a7: Status 404 returned error can't find the container with id 30706e1dd75d908306274cd14680e85eb720937737d5c0a39ebc873f5f3933a7
Sep 18 17:26:51.305559 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:51.305537    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636b80-94f555b66-ws742"
Sep 18 17:26:51.331808 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:51.331782    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-636b80-f85c7966c-vp6zj"
Sep 18 17:26:51.582896 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:51.580749    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-636b80-94f555b66-ws742"]
Sep 18 17:26:51.618379 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:51.618355    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-636b80-f85c7966c-vp6zj"]
Sep 18 17:26:51.620272 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:26:51.620244    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18c90c69_6f7a_4738_a2d3_01ad0cc83d38.slice/crio-24082803b477e6e571872d0e43c59264ac0a55dbc604277fde674c79b92226eb WatchSource:0}: Error finding container 24082803b477e6e571872d0e43c59264ac0a55dbc604277fde674c79b92226eb: Status 404 returned error can't find the container with id 24082803b477e6e571872d0e43c59264ac0a55dbc604277fde674c79b92226eb
Sep 18 17:26:51.900773 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:51.900683    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-636b80-f85c7966c-vp6zj" event={"ID":"18c90c69-6f7a-4738-a2d3-01ad0cc83d38","Type":"ContainerStarted","Data":"24082803b477e6e571872d0e43c59264ac0a55dbc604277fde674c79b92226eb"}
Sep 18 17:26:51.901777 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:51.901749    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636b80-94f555b66-ws742" event={"ID":"1b4e16b6-d717-4a39-a879-d34d21f70c67","Type":"ContainerStarted","Data":"a1fbccc4b764a53b502b21c834eef517696a927893d56389314a93ac3322195a"}
Sep 18 17:26:51.903173 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:51.903147    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7735-813782" event={"ID":"c27e84e1-d8ce-4add-baab-92a3fd759c31","Type":"ContainerStarted","Data":"ac951e0f2a1ae38922ce72d35b4c58286f2f74fb0875bf3fa5bdf6af8870d6a9"}
Sep 18 17:26:51.903267 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:51.903191    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7735-813782" event={"ID":"c27e84e1-d8ce-4add-baab-92a3fd759c31","Type":"ContainerStarted","Data":"30706e1dd75d908306274cd14680e85eb720937737d5c0a39ebc873f5f3933a7"}
Sep 18 17:26:51.903267 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:51.903233    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7735-cbdc71" podUID="eab8b59e-36e3-4eeb-a3ad-697528968238" containerName="test-curl-7735-cbdc71" containerID="cri-o://291e394d7d5c66c483e92bf40e4cd6dc0faa9b038731ee1be1f37efcf2385d02" gracePeriod=2
Sep 18 17:26:51.919566 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:51.919524    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7735-813782" podStartSLOduration=1.9195120110000001 podStartE2EDuration="1.919512011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:26:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:26:51.91777829 +0000 UTC m=+724.611705831" watchObservedRunningTime="2026-09-18 17:26:51.919512011 +0000 UTC m=+724.613439539"
Sep 18 17:26:52.382709 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:52.382661    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0a2e13-5bdc574c94-r2lmr"]
Sep 18 17:26:52.386471 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:52.386444    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a2e13-5bdc574c94-r2lmr"
Sep 18 17:26:52.389502 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:52.389387    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-0a2e13-dockercfg-7wj8w\""
Sep 18 17:26:52.389642 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:52.389591    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-0a2e13\""
Sep 18 17:26:52.420407 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:52.420280    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0a2e13-5bdc574c94-r2lmr"]
Sep 18 17:26:52.423249 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:52.423218    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72m2p\" (UniqueName: \"kubernetes.io/projected/fa4e3db2-2e1e-42e0-9e05-107039d7ed3e-kube-api-access-72m2p\") pod \"payload-pre-processing-e2e-worker-w0-0a2e13-5bdc574c94-r2lmr\" (UID: \"fa4e3db2-2e1e-42e0-9e05-107039d7ed3e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a2e13-5bdc574c94-r2lmr"
Sep 18 17:26:52.423392 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:52.423294    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4e3db2-2e1e-42e0-9e05-107039d7ed3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-0a2e13-5bdc574c94-r2lmr\" (UID: \"fa4e3db2-2e1e-42e0-9e05-107039d7ed3e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a2e13-5bdc574c94-r2lmr"
Sep 18 17:26:52.524755 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:52.524715    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72m2p\" (UniqueName: \"kubernetes.io/projected/fa4e3db2-2e1e-42e0-9e05-107039d7ed3e-kube-api-access-72m2p\") pod \"payload-pre-processing-e2e-worker-w0-0a2e13-5bdc574c94-r2lmr\" (UID: \"fa4e3db2-2e1e-42e0-9e05-107039d7ed3e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a2e13-5bdc574c94-r2lmr"
Sep 18 17:26:52.524954 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:52.524798    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4e3db2-2e1e-42e0-9e05-107039d7ed3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-0a2e13-5bdc574c94-r2lmr\" (UID: \"fa4e3db2-2e1e-42e0-9e05-107039d7ed3e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a2e13-5bdc574c94-r2lmr"
Sep 18 17:26:52.525679 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:52.525649    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4e3db2-2e1e-42e0-9e05-107039d7ed3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-0a2e13-5bdc574c94-r2lmr\" (UID: \"fa4e3db2-2e1e-42e0-9e05-107039d7ed3e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a2e13-5bdc574c94-r2lmr"
Sep 18 17:26:52.534379 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:52.534353    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72m2p\" (UniqueName: \"kubernetes.io/projected/fa4e3db2-2e1e-42e0-9e05-107039d7ed3e-kube-api-access-72m2p\") pod \"payload-pre-processing-e2e-worker-w0-0a2e13-5bdc574c94-r2lmr\" (UID: \"fa4e3db2-2e1e-42e0-9e05-107039d7ed3e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a2e13-5bdc574c94-r2lmr"
Sep 18 17:26:52.700873 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:52.700778    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a2e13-5bdc574c94-r2lmr"
Sep 18 17:26:52.717638 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:52.717610    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7735-813782"]
Sep 18 17:26:52.726446 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:52.726419    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7735-813782"]
Sep 18 17:26:52.908849 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:26:52.908815    2569 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7735-813782" container="test-curl-7735-813782"
Sep 18 17:26:53.245146 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:53.244955    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0a2e13-5bdc574c94-r2lmr"]
Sep 18 17:26:53.248678 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:26:53.248636    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa4e3db2_2e1e_42e0_9e05_107039d7ed3e.slice/crio-a82e572254e629c6be5a95f0bb7b89087bc3870748ff752e4695102c4e2936f2 WatchSource:0}: Error finding container a82e572254e629c6be5a95f0bb7b89087bc3870748ff752e4695102c4e2936f2: Status 404 returned error can't find the container with id a82e572254e629c6be5a95f0bb7b89087bc3870748ff752e4695102c4e2936f2
Sep 18 17:26:53.915646 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:53.915612    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a2e13-5bdc574c94-r2lmr" event={"ID":"fa4e3db2-2e1e-42e0-9e05-107039d7ed3e","Type":"ContainerStarted","Data":"911af9e2b551a5008fa5e881bd7b60c056affb9c7d17120242b5d04bc38b2d71"}
Sep 18 17:26:53.916012 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:53.915676    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a2e13-5bdc574c94-r2lmr" event={"ID":"fa4e3db2-2e1e-42e0-9e05-107039d7ed3e","Type":"ContainerStarted","Data":"a82e572254e629c6be5a95f0bb7b89087bc3870748ff752e4695102c4e2936f2"}
Sep 18 17:26:53.916012 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:53.915705    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a2e13-5bdc574c94-r2lmr"
Sep 18 17:26:53.917310 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:53.917255    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-636b80-f85c7966c-vp6zj" event={"ID":"18c90c69-6f7a-4738-a2d3-01ad0cc83d38","Type":"ContainerStarted","Data":"7216a6fb8daea23c8c148f195cdb2f02ba2ddaca25b948cf5f333063025539e9"}
Sep 18 17:26:53.917415 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:53.917338    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-636b80-f85c7966c-vp6zj"
Sep 18 17:26:53.918982 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:53.918960    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636b80-94f555b66-ws742" event={"ID":"1b4e16b6-d717-4a39-a879-d34d21f70c67","Type":"ContainerStarted","Data":"f4a5fa741521cdf238deea41b0ab44ccf3e210e5499bc3a58e3e40baf3191519"}
Sep 18 17:26:53.919088 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:53.919040    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636b80-94f555b66-ws742"
Sep 18 17:26:53.919088 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:53.919058    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7735-813782" podUID="c27e84e1-d8ce-4add-baab-92a3fd759c31" containerName="test-curl-7735-813782" containerID="cri-o://ac951e0f2a1ae38922ce72d35b4c58286f2f74fb0875bf3fa5bdf6af8870d6a9" gracePeriod=2
Sep 18 17:26:53.955935 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:53.955849    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a2e13-5bdc574c94-r2lmr" podStartSLOduration=1.955829281 podStartE2EDuration="1.955829281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:26:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:26:53.947972091 +0000 UTC m=+726.641899647" watchObservedRunningTime="2026-09-18 17:26:53.955829281 +0000 UTC m=+726.649756808"
Sep 18 17:26:53.996350 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:53.996293    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636b80-94f555b66-ws742" podStartSLOduration=2.5269159180000003 podStartE2EDuration="3.99627322s" totalImagesPullingTime="1.469357302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:26:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:26:53.993233773 +0000 UTC m=+726.687161300" watchObservedRunningTime="2026-09-18 17:26:53.99627322 +0000 UTC m=+726.690200748"
Sep 18 17:26:54.025615 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:54.025563    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-636b80-f85c7966c-vp6zj" podStartSLOduration=2.597004787 podStartE2EDuration="4.025545436s" totalImagesPullingTime="1.428540649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:26:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:26:54.025155147 +0000 UTC m=+726.719082700" watchObservedRunningTime="2026-09-18 17:26:54.025545436 +0000 UTC m=+726.719472964"
Sep 18 17:26:54.072931 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:54.072903    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-cbdc71"
Sep 18 17:26:54.139904 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:54.139847    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgsdm\" (UniqueName: \"kubernetes.io/projected/eab8b59e-36e3-4eeb-a3ad-697528968238-kube-api-access-cgsdm\") pod \"eab8b59e-36e3-4eeb-a3ad-697528968238\" (UID: \"eab8b59e-36e3-4eeb-a3ad-697528968238\") "
Sep 18 17:26:54.142114 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:54.142082    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eab8b59e-36e3-4eeb-a3ad-697528968238-kube-api-access-cgsdm" (OuterVolumeSpecName: "kube-api-access-cgsdm") pod "eab8b59e-36e3-4eeb-a3ad-697528968238" (UID: "eab8b59e-36e3-4eeb-a3ad-697528968238"). InnerVolumeSpecName "kube-api-access-cgsdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:26:54.241314 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:54.241287    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgsdm\" (UniqueName: \"kubernetes.io/projected/eab8b59e-36e3-4eeb-a3ad-697528968238-kube-api-access-cgsdm\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:26:54.925771 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:54.925735    2569 generic.go:356] "Generic (PLEG): container finished" podID="eab8b59e-36e3-4eeb-a3ad-697528968238" containerID="291e394d7d5c66c483e92bf40e4cd6dc0faa9b038731ee1be1f37efcf2385d02" exitCode=137
Sep 18 17:26:54.926274 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:54.925776    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-cbdc71"
Sep 18 17:26:54.926274 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:54.925833    2569 scope.go:117] "RemoveContainer" containerID="291e394d7d5c66c483e92bf40e4cd6dc0faa9b038731ee1be1f37efcf2385d02"
Sep 18 17:26:54.935411 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:54.935396    2569 scope.go:117] "RemoveContainer" containerID="291e394d7d5c66c483e92bf40e4cd6dc0faa9b038731ee1be1f37efcf2385d02"
Sep 18 17:26:54.935669 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:26:54.935651    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"291e394d7d5c66c483e92bf40e4cd6dc0faa9b038731ee1be1f37efcf2385d02\": container with ID starting with 291e394d7d5c66c483e92bf40e4cd6dc0faa9b038731ee1be1f37efcf2385d02 not found: ID does not exist" containerID="291e394d7d5c66c483e92bf40e4cd6dc0faa9b038731ee1be1f37efcf2385d02"
Sep 18 17:26:54.935757 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:54.935677    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"291e394d7d5c66c483e92bf40e4cd6dc0faa9b038731ee1be1f37efcf2385d02"} err="failed to get container status \"291e394d7d5c66c483e92bf40e4cd6dc0faa9b038731ee1be1f37efcf2385d02\": rpc error: code = NotFound desc = could not find container \"291e394d7d5c66c483e92bf40e4cd6dc0faa9b038731ee1be1f37efcf2385d02\": container with ID starting with 291e394d7d5c66c483e92bf40e4cd6dc0faa9b038731ee1be1f37efcf2385d02 not found: ID does not exist"
Sep 18 17:26:54.938265 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:26:54.938243    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7735-cbdc71\" is forbidden: User \"system:node:ip-10-0-139-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-122.ec2.internal' and this object" podUID="eab8b59e-36e3-4eeb-a3ad-697528968238" pod="opendatahub/test-curl-7735-cbdc71"
Sep 18 17:26:55.766104 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:55.766069    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eab8b59e-36e3-4eeb-a3ad-697528968238" path="/var/lib/kubelet/pods/eab8b59e-36e3-4eeb-a3ad-697528968238/volumes"
Sep 18 17:26:56.108893 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:56.108851    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-813782"
Sep 18 17:26:56.162129 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:56.162098    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pplgf\" (UniqueName: \"kubernetes.io/projected/c27e84e1-d8ce-4add-baab-92a3fd759c31-kube-api-access-pplgf\") pod \"c27e84e1-d8ce-4add-baab-92a3fd759c31\" (UID: \"c27e84e1-d8ce-4add-baab-92a3fd759c31\") "
Sep 18 17:26:56.164149 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:56.164120    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c27e84e1-d8ce-4add-baab-92a3fd759c31-kube-api-access-pplgf" (OuterVolumeSpecName: "kube-api-access-pplgf") pod "c27e84e1-d8ce-4add-baab-92a3fd759c31" (UID: "c27e84e1-d8ce-4add-baab-92a3fd759c31"). InnerVolumeSpecName "kube-api-access-pplgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:26:56.262957 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:56.262925    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pplgf\" (UniqueName: \"kubernetes.io/projected/c27e84e1-d8ce-4add-baab-92a3fd759c31-kube-api-access-pplgf\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:26:56.941643 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:56.941610    2569 generic.go:356] "Generic (PLEG): container finished" podID="c27e84e1-d8ce-4add-baab-92a3fd759c31" containerID="ac951e0f2a1ae38922ce72d35b4c58286f2f74fb0875bf3fa5bdf6af8870d6a9" exitCode=137
Sep 18 17:26:56.941829 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:56.941651    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-813782"
Sep 18 17:26:56.941829 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:56.941687    2569 scope.go:117] "RemoveContainer" containerID="ac951e0f2a1ae38922ce72d35b4c58286f2f74fb0875bf3fa5bdf6af8870d6a9"
Sep 18 17:26:56.950420 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:56.950401    2569 scope.go:117] "RemoveContainer" containerID="ac951e0f2a1ae38922ce72d35b4c58286f2f74fb0875bf3fa5bdf6af8870d6a9"
Sep 18 17:26:56.950708 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:26:56.950689    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac951e0f2a1ae38922ce72d35b4c58286f2f74fb0875bf3fa5bdf6af8870d6a9\": container with ID starting with ac951e0f2a1ae38922ce72d35b4c58286f2f74fb0875bf3fa5bdf6af8870d6a9 not found: ID does not exist" containerID="ac951e0f2a1ae38922ce72d35b4c58286f2f74fb0875bf3fa5bdf6af8870d6a9"
Sep 18 17:26:56.950782 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:56.950724    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac951e0f2a1ae38922ce72d35b4c58286f2f74fb0875bf3fa5bdf6af8870d6a9"} err="failed to get container status \"ac951e0f2a1ae38922ce72d35b4c58286f2f74fb0875bf3fa5bdf6af8870d6a9\": rpc error: code = NotFound desc = could not find container \"ac951e0f2a1ae38922ce72d35b4c58286f2f74fb0875bf3fa5bdf6af8870d6a9\": container with ID starting with ac951e0f2a1ae38922ce72d35b4c58286f2f74fb0875bf3fa5bdf6af8870d6a9 not found: ID does not exist"
Sep 18 17:26:56.952231 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:26:56.952210    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7735-813782\" is forbidden: User \"system:node:ip-10-0-139-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-122.ec2.internal' and this object" podUID="c27e84e1-d8ce-4add-baab-92a3fd759c31" pod="opendatahub/test-curl-7735-813782"
Sep 18 17:26:57.762845 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:26:57.762811    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7735-813782\" is forbidden: User \"system:node:ip-10-0-139-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-122.ec2.internal' and this object" podUID="c27e84e1-d8ce-4add-baab-92a3fd759c31" pod="opendatahub/test-curl-7735-813782"
Sep 18 17:26:57.764054 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:26:57.764034    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c27e84e1-d8ce-4add-baab-92a3fd759c31" path="/var/lib/kubelet/pods/c27e84e1-d8ce-4add-baab-92a3fd759c31/volumes"
Sep 18 17:27:01.026049 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.026018    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh"]
Sep 18 17:27:01.026420 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.026365    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c27e84e1-d8ce-4add-baab-92a3fd759c31" containerName="test-curl-7735-813782"
Sep 18 17:27:01.026420 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.026378    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c27e84e1-d8ce-4add-baab-92a3fd759c31" containerName="test-curl-7735-813782"
Sep 18 17:27:01.026488 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.026453    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c27e84e1-d8ce-4add-baab-92a3fd759c31" containerName="test-curl-7735-813782"
Sep 18 17:27:01.030403 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.030386    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh"
Sep 18 17:27:01.033383 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.033357    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 18 17:27:01.033504 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.033423    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sjtq4\""
Sep 18 17:27:01.033611 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.033584    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-535fd3-serving-cert\""
Sep 18 17:27:01.033747 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.033723    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 17:27:01.034038 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.034017    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 17:27:01.042805 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.042782    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh"]
Sep 18 17:27:01.109355 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.109325    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrxsf\" (UniqueName: \"kubernetes.io/projected/801c76f4-6045-42d3-8cf1-a464f147e1b4-kube-api-access-zrxsf\") pod \"maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh\" (UID: \"801c76f4-6045-42d3-8cf1-a464f147e1b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh"
Sep 18 17:27:01.109516 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.109370    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/801c76f4-6045-42d3-8cf1-a464f147e1b4-metrics-certs\") pod \"maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh\" (UID: \"801c76f4-6045-42d3-8cf1-a464f147e1b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh"
Sep 18 17:27:01.109516 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.109400    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/801c76f4-6045-42d3-8cf1-a464f147e1b4-maas-api-tls\") pod \"maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh\" (UID: \"801c76f4-6045-42d3-8cf1-a464f147e1b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh"
Sep 18 17:27:01.109516 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.109489    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/801c76f4-6045-42d3-8cf1-a464f147e1b4-openshift-service-ca\") pod \"maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh\" (UID: \"801c76f4-6045-42d3-8cf1-a464f147e1b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh"
Sep 18 17:27:01.210870 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.210822    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/801c76f4-6045-42d3-8cf1-a464f147e1b4-maas-api-tls\") pod \"maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh\" (UID: \"801c76f4-6045-42d3-8cf1-a464f147e1b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh"
Sep 18 17:27:01.211027 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.210889    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/801c76f4-6045-42d3-8cf1-a464f147e1b4-openshift-service-ca\") pod \"maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh\" (UID: \"801c76f4-6045-42d3-8cf1-a464f147e1b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh"
Sep 18 17:27:01.211027 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.210938    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrxsf\" (UniqueName: \"kubernetes.io/projected/801c76f4-6045-42d3-8cf1-a464f147e1b4-kube-api-access-zrxsf\") pod \"maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh\" (UID: \"801c76f4-6045-42d3-8cf1-a464f147e1b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh"
Sep 18 17:27:01.211027 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.210965    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/801c76f4-6045-42d3-8cf1-a464f147e1b4-metrics-certs\") pod \"maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh\" (UID: \"801c76f4-6045-42d3-8cf1-a464f147e1b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh"
Sep 18 17:27:01.211027 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:27:01.210997    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-535fd3-serving-cert: secret "maas-api-e2e-autores-535fd3-serving-cert" not found
Sep 18 17:27:01.211261 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:27:01.211082    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/801c76f4-6045-42d3-8cf1-a464f147e1b4-maas-api-tls podName:801c76f4-6045-42d3-8cf1-a464f147e1b4 nodeName:}" failed. No retries permitted until 2026-09-18 17:27:01.711059714 +0000 UTC m=+734.404987222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/801c76f4-6045-42d3-8cf1-a464f147e1b4-maas-api-tls") pod "maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh" (UID: "801c76f4-6045-42d3-8cf1-a464f147e1b4") : secret "maas-api-e2e-autores-535fd3-serving-cert" not found
Sep 18 17:27:01.211590 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.211567    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/801c76f4-6045-42d3-8cf1-a464f147e1b4-openshift-service-ca\") pod \"maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh\" (UID: \"801c76f4-6045-42d3-8cf1-a464f147e1b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh"
Sep 18 17:27:01.213230 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.213212    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/801c76f4-6045-42d3-8cf1-a464f147e1b4-metrics-certs\") pod \"maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh\" (UID: \"801c76f4-6045-42d3-8cf1-a464f147e1b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh"
Sep 18 17:27:01.222813 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.222787    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrxsf\" (UniqueName: \"kubernetes.io/projected/801c76f4-6045-42d3-8cf1-a464f147e1b4-kube-api-access-zrxsf\") pod \"maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh\" (UID: \"801c76f4-6045-42d3-8cf1-a464f147e1b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh"
Sep 18 17:27:01.715736 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.715702    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/801c76f4-6045-42d3-8cf1-a464f147e1b4-maas-api-tls\") pod \"maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh\" (UID: \"801c76f4-6045-42d3-8cf1-a464f147e1b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh"
Sep 18 17:27:01.717954 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.717934    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/801c76f4-6045-42d3-8cf1-a464f147e1b4-maas-api-tls\") pod \"maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh\" (UID: \"801c76f4-6045-42d3-8cf1-a464f147e1b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh"
Sep 18 17:27:01.749917 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.749879    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-535fd3-6ff4c66bf8-4c5nk"]
Sep 18 17:27:01.753616 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.753592    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-535fd3-6ff4c66bf8-4c5nk"
Sep 18 17:27:01.756604 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.756585    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-535fd3-dockercfg-9lmk5\""
Sep 18 17:27:01.756699 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.756616    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-535fd3\""
Sep 18 17:27:01.766377 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.766353    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-535fd3-6ff4c66bf8-4c5nk"]
Sep 18 17:27:01.816202 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.816169    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rgkd\" (UniqueName: \"kubernetes.io/projected/bb37e50a-97c7-4c6d-9721-b9eefead8bc5-kube-api-access-5rgkd\") pod \"payload-pre-processing-e2e-autores-535fd3-6ff4c66bf8-4c5nk\" (UID: \"bb37e50a-97c7-4c6d-9721-b9eefead8bc5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-535fd3-6ff4c66bf8-4c5nk"
Sep 18 17:27:01.816346 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.816231    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb37e50a-97c7-4c6d-9721-b9eefead8bc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-535fd3-6ff4c66bf8-4c5nk\" (UID: \"bb37e50a-97c7-4c6d-9721-b9eefead8bc5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-535fd3-6ff4c66bf8-4c5nk"
Sep 18 17:27:01.917102 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.917065    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rgkd\" (UniqueName: \"kubernetes.io/projected/bb37e50a-97c7-4c6d-9721-b9eefead8bc5-kube-api-access-5rgkd\") pod \"payload-pre-processing-e2e-autores-535fd3-6ff4c66bf8-4c5nk\" (UID: \"bb37e50a-97c7-4c6d-9721-b9eefead8bc5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-535fd3-6ff4c66bf8-4c5nk"
Sep 18 17:27:01.917273 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.917117    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb37e50a-97c7-4c6d-9721-b9eefead8bc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-535fd3-6ff4c66bf8-4c5nk\" (UID: \"bb37e50a-97c7-4c6d-9721-b9eefead8bc5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-535fd3-6ff4c66bf8-4c5nk"
Sep 18 17:27:01.917746 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.917726    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb37e50a-97c7-4c6d-9721-b9eefead8bc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-535fd3-6ff4c66bf8-4c5nk\" (UID: \"bb37e50a-97c7-4c6d-9721-b9eefead8bc5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-535fd3-6ff4c66bf8-4c5nk"
Sep 18 17:27:01.929610 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.929581    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rgkd\" (UniqueName: \"kubernetes.io/projected/bb37e50a-97c7-4c6d-9721-b9eefead8bc5-kube-api-access-5rgkd\") pod \"payload-pre-processing-e2e-autores-535fd3-6ff4c66bf8-4c5nk\" (UID: \"bb37e50a-97c7-4c6d-9721-b9eefead8bc5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-535fd3-6ff4c66bf8-4c5nk"
Sep 18 17:27:01.940424 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:01.940406    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh"
Sep 18 17:27:02.065888 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:02.065839    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-535fd3-6ff4c66bf8-4c5nk"
Sep 18 17:27:02.220064 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:02.220041    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh"]
Sep 18 17:27:02.222411 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:27:02.222382    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod801c76f4_6045_42d3_8cf1_a464f147e1b4.slice/crio-f9d328cd10b0fcea1b5c993e1dee7233c602fe4036bb6295afe2aa63d57825d9 WatchSource:0}: Error finding container f9d328cd10b0fcea1b5c993e1dee7233c602fe4036bb6295afe2aa63d57825d9: Status 404 returned error can't find the container with id f9d328cd10b0fcea1b5c993e1dee7233c602fe4036bb6295afe2aa63d57825d9
Sep 18 17:27:02.349214 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:02.349189    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-535fd3-6ff4c66bf8-4c5nk"]
Sep 18 17:27:02.350910 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:27:02.350885    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb37e50a_97c7_4c6d_9721_b9eefead8bc5.slice/crio-f4dc13f42b5f2d6cc5acf802a65c4d6de9a87183310bc9c9d9ff672e33062fff WatchSource:0}: Error finding container f4dc13f42b5f2d6cc5acf802a65c4d6de9a87183310bc9c9d9ff672e33062fff: Status 404 returned error can't find the container with id f4dc13f42b5f2d6cc5acf802a65c4d6de9a87183310bc9c9d9ff672e33062fff
Sep 18 17:27:02.976549 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:02.976507    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-535fd3-6ff4c66bf8-4c5nk" event={"ID":"bb37e50a-97c7-4c6d-9721-b9eefead8bc5","Type":"ContainerStarted","Data":"91ac6fd4b814ceecc159faf834e8454697e42d89c0c2c59cf314654051ea3117"}
Sep 18 17:27:02.976721 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:02.976581    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-535fd3-6ff4c66bf8-4c5nk" event={"ID":"bb37e50a-97c7-4c6d-9721-b9eefead8bc5","Type":"ContainerStarted","Data":"f4dc13f42b5f2d6cc5acf802a65c4d6de9a87183310bc9c9d9ff672e33062fff"}
Sep 18 17:27:02.977339 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:02.977316    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-535fd3-6ff4c66bf8-4c5nk"
Sep 18 17:27:02.979831 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:02.979796    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh" event={"ID":"801c76f4-6045-42d3-8cf1-a464f147e1b4","Type":"ContainerStarted","Data":"f9d328cd10b0fcea1b5c993e1dee7233c602fe4036bb6295afe2aa63d57825d9"}
Sep 18 17:27:03.045902 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:03.041880    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-535fd3-6ff4c66bf8-4c5nk" podStartSLOduration=2.041839498 podStartE2EDuration="2.041839498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:27:03.035694909 +0000 UTC m=+735.729622454" watchObservedRunningTime="2026-09-18 17:27:03.041839498 +0000 UTC m=+735.735767027"
Sep 18 17:27:03.415493 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:03.415455    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5cfd4b5c-777bfcf5f8-7mgk9"]
Sep 18 17:27:03.419569 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:03.419544    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5cfd4b5c-777bfcf5f8-7mgk9"
Sep 18 17:27:03.424138 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:03.424113    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5cfd4b5c-dockercfg-th4dc\""
Sep 18 17:27:03.431368 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:03.431339    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5cfd4b5c\""
Sep 18 17:27:03.452944 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:03.452875    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5cfd4b5c-777bfcf5f8-7mgk9"]
Sep 18 17:27:03.532729 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:03.532608    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrtzr\" (UniqueName: \"kubernetes.io/projected/bd05d5fb-06a7-43b8-922b-002788f44477-kube-api-access-zrtzr\") pod \"payload-pre-processing-e2e-ait-5cfd4b5c-777bfcf5f8-7mgk9\" (UID: \"bd05d5fb-06a7-43b8-922b-002788f44477\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5cfd4b5c-777bfcf5f8-7mgk9"
Sep 18 17:27:03.532948 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:03.532771    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd05d5fb-06a7-43b8-922b-002788f44477-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5cfd4b5c-777bfcf5f8-7mgk9\" (UID: \"bd05d5fb-06a7-43b8-922b-002788f44477\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5cfd4b5c-777bfcf5f8-7mgk9"
Sep 18 17:27:03.535402 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:03.535376    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5cfd4b5c-b6475559-gj4rq"]
Sep 18 17:27:03.538824 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:03.538798    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5cfd4b5c-b6475559-gj4rq"
Sep 18 17:27:03.586621 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:03.586587    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5cfd4b5c-b6475559-gj4rq"]
Sep 18 17:27:03.633523 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:03.633490    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8544c492-9431-4223-ae6e-dc0680751721-plugins-config-volume\") pod \"payload-processing-e2e-ait-5cfd4b5c-b6475559-gj4rq\" (UID: \"8544c492-9431-4223-ae6e-dc0680751721\") " pod="openshift-ingress/payload-processing-e2e-ait-5cfd4b5c-b6475559-gj4rq"
Sep 18 17:27:03.633686 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:03.633543    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd05d5fb-06a7-43b8-922b-002788f44477-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5cfd4b5c-777bfcf5f8-7mgk9\" (UID: \"bd05d5fb-06a7-43b8-922b-002788f44477\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5cfd4b5c-777bfcf5f8-7mgk9"
Sep 18 17:27:03.633686 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:03.633631    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrtzr\" (UniqueName: \"kubernetes.io/projected/bd05d5fb-06a7-43b8-922b-002788f44477-kube-api-access-zrtzr\") pod \"payload-pre-processing-e2e-ait-5cfd4b5c-777bfcf5f8-7mgk9\" (UID: \"bd05d5fb-06a7-43b8-922b-002788f44477\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5cfd4b5c-777bfcf5f8-7mgk9"
Sep 18 17:27:03.633807 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:03.633722    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr6wq\" (UniqueName: \"kubernetes.io/projected/8544c492-9431-4223-ae6e-dc0680751721-kube-api-access-cr6wq\") pod \"payload-processing-e2e-ait-5cfd4b5c-b6475559-gj4rq\" (UID: \"8544c492-9431-4223-ae6e-dc0680751721\") " pod="openshift-ingress/payload-processing-e2e-ait-5cfd4b5c-b6475559-gj4rq"
Sep 18 17:27:03.634299 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:03.634280    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd05d5fb-06a7-43b8-922b-002788f44477-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5cfd4b5c-777bfcf5f8-7mgk9\" (UID: \"bd05d5fb-06a7-43b8-922b-002788f44477\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5cfd4b5c-777bfcf5f8-7mgk9"
Sep 18 17:27:03.646465 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:03.646440    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrtzr\" (UniqueName: \"kubernetes.io/projected/bd05d5fb-06a7-43b8-922b-002788f44477-kube-api-access-zrtzr\") pod \"payload-pre-processing-e2e-ait-5cfd4b5c-777bfcf5f8-7mgk9\" (UID: \"bd05d5fb-06a7-43b8-922b-002788f44477\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5cfd4b5c-777bfcf5f8-7mgk9"
Sep 18 17:27:03.733899 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:03.733851    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5cfd4b5c-777bfcf5f8-7mgk9"
Sep 18 17:27:03.734229 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:03.734198    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr6wq\" (UniqueName: \"kubernetes.io/projected/8544c492-9431-4223-ae6e-dc0680751721-kube-api-access-cr6wq\") pod \"payload-processing-e2e-ait-5cfd4b5c-b6475559-gj4rq\" (UID: \"8544c492-9431-4223-ae6e-dc0680751721\") " pod="openshift-ingress/payload-processing-e2e-ait-5cfd4b5c-b6475559-gj4rq"
Sep 18 17:27:03.734358 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:03.734259    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8544c492-9431-4223-ae6e-dc0680751721-plugins-config-volume\") pod \"payload-processing-e2e-ait-5cfd4b5c-b6475559-gj4rq\" (UID: \"8544c492-9431-4223-ae6e-dc0680751721\") " pod="openshift-ingress/payload-processing-e2e-ait-5cfd4b5c-b6475559-gj4rq"
Sep 18 17:27:03.735300 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:03.735276    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8544c492-9431-4223-ae6e-dc0680751721-plugins-config-volume\") pod \"payload-processing-e2e-ait-5cfd4b5c-b6475559-gj4rq\" (UID: \"8544c492-9431-4223-ae6e-dc0680751721\") " pod="openshift-ingress/payload-processing-e2e-ait-5cfd4b5c-b6475559-gj4rq"
Sep 18 17:27:03.748586 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:03.748560    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr6wq\" (UniqueName: \"kubernetes.io/projected/8544c492-9431-4223-ae6e-dc0680751721-kube-api-access-cr6wq\") pod \"payload-processing-e2e-ait-5cfd4b5c-b6475559-gj4rq\" (UID: \"8544c492-9431-4223-ae6e-dc0680751721\") " pod="openshift-ingress/payload-processing-e2e-ait-5cfd4b5c-b6475559-gj4rq"
Sep 18 17:27:03.848851 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:03.848812    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5cfd4b5c-b6475559-gj4rq"
Sep 18 17:27:04.915204 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:04.915118    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5cfd4b5c-777bfcf5f8-7mgk9"]
Sep 18 17:27:04.916800 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:04.916769    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5cfd4b5c-b6475559-gj4rq"]
Sep 18 17:27:04.921165 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:27:04.921138    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8544c492_9431_4223_ae6e_dc0680751721.slice/crio-5019f4a934d1c6462ec232671f36eba27b1fb2586cf227daf4b1bedff2ac5469 WatchSource:0}: Error finding container 5019f4a934d1c6462ec232671f36eba27b1fb2586cf227daf4b1bedff2ac5469: Status 404 returned error can't find the container with id 5019f4a934d1c6462ec232671f36eba27b1fb2586cf227daf4b1bedff2ac5469
Sep 18 17:27:04.921891 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:27:04.921852    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd05d5fb_06a7_43b8_922b_002788f44477.slice/crio-c809247ebdc97ab93757f4ad9fc13ed15a7807fc2aa59bf1cffaa8194fb0c82b WatchSource:0}: Error finding container c809247ebdc97ab93757f4ad9fc13ed15a7807fc2aa59bf1cffaa8194fb0c82b: Status 404 returned error can't find the container with id c809247ebdc97ab93757f4ad9fc13ed15a7807fc2aa59bf1cffaa8194fb0c82b
Sep 18 17:27:04.928024 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:04.928001    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636b80-94f555b66-ws742"
Sep 18 17:27:04.928121 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:04.928067    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a2e13-5bdc574c94-r2lmr"
Sep 18 17:27:04.928217 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:04.928199    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-636b80-f85c7966c-vp6zj"
Sep 18 17:27:04.995145 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:04.995113    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5cfd4b5c-777bfcf5f8-7mgk9" event={"ID":"bd05d5fb-06a7-43b8-922b-002788f44477","Type":"ContainerStarted","Data":"c809247ebdc97ab93757f4ad9fc13ed15a7807fc2aa59bf1cffaa8194fb0c82b"}
Sep 18 17:27:04.997250 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:04.997219    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh" event={"ID":"801c76f4-6045-42d3-8cf1-a464f147e1b4","Type":"ContainerStarted","Data":"05cf6d944da56f93931352d8227ae14e02b121f1620d66bd12fb6b9541687dc6"}
Sep 18 17:27:04.997365 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:04.997302    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh"
Sep 18 17:27:04.999358 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:04.999323    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5cfd4b5c-b6475559-gj4rq" event={"ID":"8544c492-9431-4223-ae6e-dc0680751721","Type":"ContainerStarted","Data":"5019f4a934d1c6462ec232671f36eba27b1fb2586cf227daf4b1bedff2ac5469"}
Sep 18 17:27:05.041884 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:05.041804    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh" podStartSLOduration=2.560467278 podStartE2EDuration="5.041784897s" totalImagesPullingTime="2.481317619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:27:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:27:05.039748172 +0000 UTC m=+737.733675726" watchObservedRunningTime="2026-09-18 17:27:05.041784897 +0000 UTC m=+737.735712427"
Sep 18 17:27:06.006647 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:06.006609    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5cfd4b5c-b6475559-gj4rq" event={"ID":"8544c492-9431-4223-ae6e-dc0680751721","Type":"ContainerStarted","Data":"5a54e4fa9b9475e60e739815c7cb30c8e107dd29da2901ccb04607f090618940"}
Sep 18 17:27:06.007118 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:06.006680    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-5cfd4b5c-b6475559-gj4rq"
Sep 18 17:27:06.008339 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:06.008314    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5cfd4b5c-777bfcf5f8-7mgk9" event={"ID":"bd05d5fb-06a7-43b8-922b-002788f44477","Type":"ContainerStarted","Data":"8a78239418d583e7bdba059b0134ec6c12504ffd588c9666fe81d2ab6b1f036d"}
Sep 18 17:27:06.008450 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:06.008387    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5cfd4b5c-777bfcf5f8-7mgk9"
Sep 18 17:27:06.027679 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:06.027623    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-5cfd4b5c-b6475559-gj4rq" podStartSLOduration=3.027609258 podStartE2EDuration="3.027609258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:27:06.025822506 +0000 UTC m=+738.719750033" watchObservedRunningTime="2026-09-18 17:27:06.027609258 +0000 UTC m=+738.721536786"
Sep 18 17:27:06.046187 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:06.046143    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-5cfd4b5c-777bfcf5f8-7mgk9" podStartSLOduration=3.046129068 podStartE2EDuration="3.046129068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:27:06.045048945 +0000 UTC m=+738.738976478" watchObservedRunningTime="2026-09-18 17:27:06.046129068 +0000 UTC m=+738.740056596"
Sep 18 17:27:11.014679 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:11.014646    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh"
Sep 18 17:27:14.122734 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:14.122696    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2aa65a-59c986568b-bt4sb"]
Sep 18 17:27:14.125854 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:14.125835    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aa65a-59c986568b-bt4sb"
Sep 18 17:27:14.129960 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:14.129939    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-2aa65a-dockercfg-q4qs5\""
Sep 18 17:27:14.130075 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:14.129973    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2aa65a\""
Sep 18 17:27:14.145369 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:14.145340    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2aa65a-59c986568b-bt4sb"]
Sep 18 17:27:14.234361 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:14.234327    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hv5w\" (UniqueName: \"kubernetes.io/projected/087178e2-6dda-463a-89f8-66c246d90692-kube-api-access-2hv5w\") pod \"payload-pre-processing-e2e-inf-b-2aa65a-59c986568b-bt4sb\" (UID: \"087178e2-6dda-463a-89f8-66c246d90692\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aa65a-59c986568b-bt4sb"
Sep 18 17:27:14.234518 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:14.234381    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/087178e2-6dda-463a-89f8-66c246d90692-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2aa65a-59c986568b-bt4sb\" (UID: \"087178e2-6dda-463a-89f8-66c246d90692\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aa65a-59c986568b-bt4sb"
Sep 18 17:27:14.335649 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:14.335604    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hv5w\" (UniqueName: \"kubernetes.io/projected/087178e2-6dda-463a-89f8-66c246d90692-kube-api-access-2hv5w\") pod \"payload-pre-processing-e2e-inf-b-2aa65a-59c986568b-bt4sb\" (UID: \"087178e2-6dda-463a-89f8-66c246d90692\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aa65a-59c986568b-bt4sb"
Sep 18 17:27:14.335826 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:14.335682    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/087178e2-6dda-463a-89f8-66c246d90692-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2aa65a-59c986568b-bt4sb\" (UID: \"087178e2-6dda-463a-89f8-66c246d90692\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aa65a-59c986568b-bt4sb"
Sep 18 17:27:14.336442 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:14.336420    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/087178e2-6dda-463a-89f8-66c246d90692-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2aa65a-59c986568b-bt4sb\" (UID: \"087178e2-6dda-463a-89f8-66c246d90692\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aa65a-59c986568b-bt4sb"
Sep 18 17:27:14.348924 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:14.348895    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hv5w\" (UniqueName: \"kubernetes.io/projected/087178e2-6dda-463a-89f8-66c246d90692-kube-api-access-2hv5w\") pod \"payload-pre-processing-e2e-inf-b-2aa65a-59c986568b-bt4sb\" (UID: \"087178e2-6dda-463a-89f8-66c246d90692\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aa65a-59c986568b-bt4sb"
Sep 18 17:27:14.436638 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:14.436549    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aa65a-59c986568b-bt4sb"
Sep 18 17:27:14.507981 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:14.507951    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cfd4b5c-xqh54"]
Sep 18 17:27:14.511575 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:14.511554    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cfd4b5c-xqh54"
Sep 18 17:27:14.516784 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:14.516758    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qt978\""
Sep 18 17:27:14.525348 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:14.525325    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cfd4b5c-xqh54"]
Sep 18 17:27:14.638977 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:14.638941    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee5b67d4-b2f9-4e03-8dba-e6654e8552e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5cfd4b5c-xqh54\" (UID: \"ee5b67d4-b2f9-4e03-8dba-e6654e8552e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cfd4b5c-xqh54"
Sep 18 17:27:14.726930 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:14.726904    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2aa65a-59c986568b-bt4sb"]
Sep 18 17:27:14.728559 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:27:14.728521    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087178e2_6dda_463a_89f8_66c246d90692.slice/crio-0202a8b0417eff249fa8e790517d8c3f842e2be94164ef9951a354786afed7b3 WatchSource:0}: Error finding container 0202a8b0417eff249fa8e790517d8c3f842e2be94164ef9951a354786afed7b3: Status 404 returned error can't find the container with id 0202a8b0417eff249fa8e790517d8c3f842e2be94164ef9951a354786afed7b3
Sep 18 17:27:14.739766 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:14.739737    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee5b67d4-b2f9-4e03-8dba-e6654e8552e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5cfd4b5c-xqh54\" (UID: \"ee5b67d4-b2f9-4e03-8dba-e6654e8552e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cfd4b5c-xqh54"
Sep 18 17:27:14.740524 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:14.740447    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee5b67d4-b2f9-4e03-8dba-e6654e8552e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5cfd4b5c-xqh54\" (UID: \"ee5b67d4-b2f9-4e03-8dba-e6654e8552e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cfd4b5c-xqh54"
Sep 18 17:27:14.822826 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:14.822792    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cfd4b5c-xqh54"
Sep 18 17:27:15.001673 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:15.001642    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-535fd3-6ff4c66bf8-4c5nk"
Sep 18 17:27:15.058181 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:15.058049    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aa65a-59c986568b-bt4sb" event={"ID":"087178e2-6dda-463a-89f8-66c246d90692","Type":"ContainerStarted","Data":"612ff215393dbb06c297ceeb2f254bdfa9cf2e7daf78275f0e698320d1c5c641"}
Sep 18 17:27:15.058181 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:15.058116    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aa65a-59c986568b-bt4sb"
Sep 18 17:27:15.058181 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:15.058142    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aa65a-59c986568b-bt4sb" event={"ID":"087178e2-6dda-463a-89f8-66c246d90692","Type":"ContainerStarted","Data":"0202a8b0417eff249fa8e790517d8c3f842e2be94164ef9951a354786afed7b3"}
Sep 18 17:27:15.102659 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:15.102605    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aa65a-59c986568b-bt4sb" podStartSLOduration=1.10258688 podStartE2EDuration="1.10258688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:27:15.101428753 +0000 UTC m=+747.795356307" watchObservedRunningTime="2026-09-18 17:27:15.10258688 +0000 UTC m=+747.796514408"
Sep 18 17:27:15.121043 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:15.121017    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cfd4b5c-xqh54"]
Sep 18 17:27:15.123600 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:27:15.123574    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee5b67d4_b2f9_4e03_8dba_e6654e8552e6.slice/crio-f3fc80d226081916f2d4b23dca9f8d67d7ba3c66574b3baecd8070418ba331e4 WatchSource:0}: Error finding container f3fc80d226081916f2d4b23dca9f8d67d7ba3c66574b3baecd8070418ba331e4: Status 404 returned error can't find the container with id f3fc80d226081916f2d4b23dca9f8d67d7ba3c66574b3baecd8070418ba331e4
Sep 18 17:27:16.067616 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:16.067577    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cfd4b5c-xqh54" event={"ID":"ee5b67d4-b2f9-4e03-8dba-e6654e8552e6","Type":"ContainerStarted","Data":"f3fc80d226081916f2d4b23dca9f8d67d7ba3c66574b3baecd8070418ba331e4"}
Sep 18 17:27:17.016994 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:17.016956    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-5cfd4b5c-b6475559-gj4rq"
Sep 18 17:27:17.017531 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:17.017047    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5cfd4b5c-777bfcf5f8-7mgk9"
Sep 18 17:27:18.080328 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:18.080280    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cfd4b5c-xqh54" event={"ID":"ee5b67d4-b2f9-4e03-8dba-e6654e8552e6","Type":"ContainerStarted","Data":"075eda9ebcc4a20464c3057a4303d70138078855e10b31b077084c250a90c170"}
Sep 18 17:27:18.117320 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:18.117268    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cfd4b5c-xqh54" podStartSLOduration=1.8380344229999999 podStartE2EDuration="4.117253194s" totalImagesPullingTime="2.279218771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:27:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:27:18.116564783 +0000 UTC m=+750.810492330" watchObservedRunningTime="2026-09-18 17:27:18.117253194 +0000 UTC m=+750.811180777"
Sep 18 17:27:19.086763 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:19.086716    2569 generic.go:356] "Generic (PLEG): container finished" podID="ee5b67d4-b2f9-4e03-8dba-e6654e8552e6" containerID="075eda9ebcc4a20464c3057a4303d70138078855e10b31b077084c250a90c170" exitCode=0
Sep 18 17:27:19.087146 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:19.086809    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cfd4b5c-xqh54" event={"ID":"ee5b67d4-b2f9-4e03-8dba-e6654e8552e6","Type":"ContainerDied","Data":"075eda9ebcc4a20464c3057a4303d70138078855e10b31b077084c250a90c170"}
Sep 18 17:27:20.235203 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:20.235175    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cfd4b5c-xqh54"
Sep 18 17:27:20.396213 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:20.396120    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee5b67d4-b2f9-4e03-8dba-e6654e8552e6-maas-api-service-ca\") pod \"ee5b67d4-b2f9-4e03-8dba-e6654e8552e6\" (UID: \"ee5b67d4-b2f9-4e03-8dba-e6654e8552e6\") "
Sep 18 17:27:20.396496 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:20.396471    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5b67d4-b2f9-4e03-8dba-e6654e8552e6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ee5b67d4-b2f9-4e03-8dba-e6654e8552e6" (UID: "ee5b67d4-b2f9-4e03-8dba-e6654e8552e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:27:20.497618 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:20.497582    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee5b67d4-b2f9-4e03-8dba-e6654e8552e6-maas-api-service-ca\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:27:21.097493 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:21.097457    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cfd4b5c-xqh54"
Sep 18 17:27:21.097660 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:21.097492    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cfd4b5c-xqh54" event={"ID":"ee5b67d4-b2f9-4e03-8dba-e6654e8552e6","Type":"ContainerDied","Data":"f3fc80d226081916f2d4b23dca9f8d67d7ba3c66574b3baecd8070418ba331e4"}
Sep 18 17:27:21.097660 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:21.097541    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3fc80d226081916f2d4b23dca9f8d67d7ba3c66574b3baecd8070418ba331e4"
Sep 18 17:27:26.070543 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:26.070510    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aa65a-59c986568b-bt4sb"
Sep 18 17:27:28.110041 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:28.110008    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5cfd4b5c-b6475559-gj4rq"]
Sep 18 17:27:28.110474 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:28.110226    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-5cfd4b5c-b6475559-gj4rq" podUID="8544c492-9431-4223-ae6e-dc0680751721" containerName="payload-processing" containerID="cri-o://5a54e4fa9b9475e60e739815c7cb30c8e107dd29da2901ccb04607f090618940" gracePeriod=30
Sep 18 17:27:28.124738 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:28.124708    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5cfd4b5c-777bfcf5f8-7mgk9"]
Sep 18 17:27:28.125000 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:28.124959    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-5cfd4b5c-777bfcf5f8-7mgk9" podUID="bd05d5fb-06a7-43b8-922b-002788f44477" containerName="payload-pre-processing" containerID="cri-o://8a78239418d583e7bdba059b0134ec6c12504ffd588c9666fe81d2ab6b1f036d" gracePeriod=30
Sep 18 17:27:28.379563 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:28.379537    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5cfd4b5c-b6475559-gj4rq"
Sep 18 17:27:28.399320 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:28.399298    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5cfd4b5c-777bfcf5f8-7mgk9"
Sep 18 17:27:28.457879 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:28.457827    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cr6wq\" (UniqueName: \"kubernetes.io/projected/8544c492-9431-4223-ae6e-dc0680751721-kube-api-access-cr6wq\") pod \"8544c492-9431-4223-ae6e-dc0680751721\" (UID: \"8544c492-9431-4223-ae6e-dc0680751721\") "
Sep 18 17:27:28.458069 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:28.457888    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd05d5fb-06a7-43b8-922b-002788f44477-plugins-config-volume\") pod \"bd05d5fb-06a7-43b8-922b-002788f44477\" (UID: \"bd05d5fb-06a7-43b8-922b-002788f44477\") "
Sep 18 17:27:28.458069 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:28.457942    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrtzr\" (UniqueName: \"kubernetes.io/projected/bd05d5fb-06a7-43b8-922b-002788f44477-kube-api-access-zrtzr\") pod \"bd05d5fb-06a7-43b8-922b-002788f44477\" (UID: \"bd05d5fb-06a7-43b8-922b-002788f44477\") "
Sep 18 17:27:28.458069 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:28.457971    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8544c492-9431-4223-ae6e-dc0680751721-plugins-config-volume\") pod \"8544c492-9431-4223-ae6e-dc0680751721\" (UID: \"8544c492-9431-4223-ae6e-dc0680751721\") "
Sep 18 17:27:28.458348 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:28.458321    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd05d5fb-06a7-43b8-922b-002788f44477-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bd05d5fb-06a7-43b8-922b-002788f44477" (UID: "bd05d5fb-06a7-43b8-922b-002788f44477"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:27:28.458403 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:28.458352    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8544c492-9431-4223-ae6e-dc0680751721-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8544c492-9431-4223-ae6e-dc0680751721" (UID: "8544c492-9431-4223-ae6e-dc0680751721"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:27:28.459997 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:28.459976    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8544c492-9431-4223-ae6e-dc0680751721-kube-api-access-cr6wq" (OuterVolumeSpecName: "kube-api-access-cr6wq") pod "8544c492-9431-4223-ae6e-dc0680751721" (UID: "8544c492-9431-4223-ae6e-dc0680751721"). InnerVolumeSpecName "kube-api-access-cr6wq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:27:28.459997 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:28.459989    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd05d5fb-06a7-43b8-922b-002788f44477-kube-api-access-zrtzr" (OuterVolumeSpecName: "kube-api-access-zrtzr") pod "bd05d5fb-06a7-43b8-922b-002788f44477" (UID: "bd05d5fb-06a7-43b8-922b-002788f44477"). InnerVolumeSpecName "kube-api-access-zrtzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:27:28.558751 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:28.558709    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrtzr\" (UniqueName: \"kubernetes.io/projected/bd05d5fb-06a7-43b8-922b-002788f44477-kube-api-access-zrtzr\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:27:28.558751 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:28.558746    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8544c492-9431-4223-ae6e-dc0680751721-plugins-config-volume\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:27:28.558751 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:28.558756    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr6wq\" (UniqueName: \"kubernetes.io/projected/8544c492-9431-4223-ae6e-dc0680751721-kube-api-access-cr6wq\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:27:28.558994 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:28.558765    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd05d5fb-06a7-43b8-922b-002788f44477-plugins-config-volume\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:27:28.634708 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:28.634627    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cfd4b5c-xqh54"]
Sep 18 17:27:28.643053 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:28.643024    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cfd4b5c-xqh54"]
Sep 18 17:27:29.139186 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:29.139150    2569 generic.go:356] "Generic (PLEG): container finished" podID="8544c492-9431-4223-ae6e-dc0680751721" containerID="5a54e4fa9b9475e60e739815c7cb30c8e107dd29da2901ccb04607f090618940" exitCode=0
Sep 18 17:27:29.139609 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:29.139198    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5cfd4b5c-b6475559-gj4rq"
Sep 18 17:27:29.139609 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:29.139232    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5cfd4b5c-b6475559-gj4rq" event={"ID":"8544c492-9431-4223-ae6e-dc0680751721","Type":"ContainerDied","Data":"5a54e4fa9b9475e60e739815c7cb30c8e107dd29da2901ccb04607f090618940"}
Sep 18 17:27:29.139609 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:29.139280    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5cfd4b5c-b6475559-gj4rq" event={"ID":"8544c492-9431-4223-ae6e-dc0680751721","Type":"ContainerDied","Data":"5019f4a934d1c6462ec232671f36eba27b1fb2586cf227daf4b1bedff2ac5469"}
Sep 18 17:27:29.139609 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:29.139310    2569 scope.go:117] "RemoveContainer" containerID="5a54e4fa9b9475e60e739815c7cb30c8e107dd29da2901ccb04607f090618940"
Sep 18 17:27:29.140681 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:29.140618    2569 generic.go:356] "Generic (PLEG): container finished" podID="bd05d5fb-06a7-43b8-922b-002788f44477" containerID="8a78239418d583e7bdba059b0134ec6c12504ffd588c9666fe81d2ab6b1f036d" exitCode=0
Sep 18 17:27:29.140681 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:29.140652    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5cfd4b5c-777bfcf5f8-7mgk9" event={"ID":"bd05d5fb-06a7-43b8-922b-002788f44477","Type":"ContainerDied","Data":"8a78239418d583e7bdba059b0134ec6c12504ffd588c9666fe81d2ab6b1f036d"}
Sep 18 17:27:29.140681 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:29.140662    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5cfd4b5c-777bfcf5f8-7mgk9"
Sep 18 17:27:29.140857 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:29.140693    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5cfd4b5c-777bfcf5f8-7mgk9" event={"ID":"bd05d5fb-06a7-43b8-922b-002788f44477","Type":"ContainerDied","Data":"c809247ebdc97ab93757f4ad9fc13ed15a7807fc2aa59bf1cffaa8194fb0c82b"}
Sep 18 17:27:29.148537 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:29.148425    2569 scope.go:117] "RemoveContainer" containerID="5a54e4fa9b9475e60e739815c7cb30c8e107dd29da2901ccb04607f090618940"
Sep 18 17:27:29.148846 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:27:29.148819    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a54e4fa9b9475e60e739815c7cb30c8e107dd29da2901ccb04607f090618940\": container with ID starting with 5a54e4fa9b9475e60e739815c7cb30c8e107dd29da2901ccb04607f090618940 not found: ID does not exist" containerID="5a54e4fa9b9475e60e739815c7cb30c8e107dd29da2901ccb04607f090618940"
Sep 18 17:27:29.148932 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:29.148875    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a54e4fa9b9475e60e739815c7cb30c8e107dd29da2901ccb04607f090618940"} err="failed to get container status \"5a54e4fa9b9475e60e739815c7cb30c8e107dd29da2901ccb04607f090618940\": rpc error: code = NotFound desc = could not find container \"5a54e4fa9b9475e60e739815c7cb30c8e107dd29da2901ccb04607f090618940\": container with ID starting with 5a54e4fa9b9475e60e739815c7cb30c8e107dd29da2901ccb04607f090618940 not found: ID does not exist"
Sep 18 17:27:29.148932 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:29.148898    2569 scope.go:117] "RemoveContainer" containerID="8a78239418d583e7bdba059b0134ec6c12504ffd588c9666fe81d2ab6b1f036d"
Sep 18 17:27:29.159688 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:29.159666    2569 scope.go:117] "RemoveContainer" containerID="8a78239418d583e7bdba059b0134ec6c12504ffd588c9666fe81d2ab6b1f036d"
Sep 18 17:27:29.160008 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:27:29.159983    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a78239418d583e7bdba059b0134ec6c12504ffd588c9666fe81d2ab6b1f036d\": container with ID starting with 8a78239418d583e7bdba059b0134ec6c12504ffd588c9666fe81d2ab6b1f036d not found: ID does not exist" containerID="8a78239418d583e7bdba059b0134ec6c12504ffd588c9666fe81d2ab6b1f036d"
Sep 18 17:27:29.160084 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:29.160018    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a78239418d583e7bdba059b0134ec6c12504ffd588c9666fe81d2ab6b1f036d"} err="failed to get container status \"8a78239418d583e7bdba059b0134ec6c12504ffd588c9666fe81d2ab6b1f036d\": rpc error: code = NotFound desc = could not find container \"8a78239418d583e7bdba059b0134ec6c12504ffd588c9666fe81d2ab6b1f036d\": container with ID starting with 8a78239418d583e7bdba059b0134ec6c12504ffd588c9666fe81d2ab6b1f036d not found: ID does not exist"
Sep 18 17:27:29.169626 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:29.169601    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5cfd4b5c-777bfcf5f8-7mgk9"]
Sep 18 17:27:29.177127 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:29.177104    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5cfd4b5c-777bfcf5f8-7mgk9"]
Sep 18 17:27:29.189317 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:29.189293    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5cfd4b5c-b6475559-gj4rq"]
Sep 18 17:27:29.191000 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:29.190978    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5cfd4b5c-b6475559-gj4rq"]
Sep 18 17:27:29.764600 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:29.764566    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8544c492-9431-4223-ae6e-dc0680751721" path="/var/lib/kubelet/pods/8544c492-9431-4223-ae6e-dc0680751721/volumes"
Sep 18 17:27:29.764971 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:29.764957    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd05d5fb-06a7-43b8-922b-002788f44477" path="/var/lib/kubelet/pods/bd05d5fb-06a7-43b8-922b-002788f44477/volumes"
Sep 18 17:27:29.765257 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:29.765246    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee5b67d4-b2f9-4e03-8dba-e6654e8552e6" path="/var/lib/kubelet/pods/ee5b67d4-b2f9-4e03-8dba-e6654e8552e6/volumes"
Sep 18 17:27:35.620570 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:35.620534    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7"]
Sep 18 17:27:35.620976 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:35.620905    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee5b67d4-b2f9-4e03-8dba-e6654e8552e6" containerName="revoke-keys"
Sep 18 17:27:35.620976 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:35.620917    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee5b67d4-b2f9-4e03-8dba-e6654e8552e6" containerName="revoke-keys"
Sep 18 17:27:35.620976 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:35.620928    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8544c492-9431-4223-ae6e-dc0680751721" containerName="payload-processing"
Sep 18 17:27:35.620976 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:35.620934    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8544c492-9431-4223-ae6e-dc0680751721" containerName="payload-processing"
Sep 18 17:27:35.620976 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:35.620945    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd05d5fb-06a7-43b8-922b-002788f44477" containerName="payload-pre-processing"
Sep 18 17:27:35.620976 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:35.620951    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd05d5fb-06a7-43b8-922b-002788f44477" containerName="payload-pre-processing"
Sep 18 17:27:35.621304 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:35.621006    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee5b67d4-b2f9-4e03-8dba-e6654e8552e6" containerName="revoke-keys"
Sep 18 17:27:35.621304 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:35.621015    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8544c492-9431-4223-ae6e-dc0680751721" containerName="payload-processing"
Sep 18 17:27:35.621304 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:35.621022    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd05d5fb-06a7-43b8-922b-002788f44477" containerName="payload-pre-processing"
Sep 18 17:27:35.630212 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:35.630186    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7"
Sep 18 17:27:35.632837 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:35.632809    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-fea15b54-serving-cert\""
Sep 18 17:27:35.640361 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:35.640335    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7"]
Sep 18 17:27:35.719160 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:35.719122    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf2f4\" (UniqueName: \"kubernetes.io/projected/c2e2c5f9-b95a-40bf-a149-56b6e98625e9-kube-api-access-kf2f4\") pod \"maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7\" (UID: \"c2e2c5f9-b95a-40bf-a149-56b6e98625e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7"
Sep 18 17:27:35.719343 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:35.719169    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e2c5f9-b95a-40bf-a149-56b6e98625e9-maas-api-tls\") pod \"maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7\" (UID: \"c2e2c5f9-b95a-40bf-a149-56b6e98625e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7"
Sep 18 17:27:35.719343 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:35.719241    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2e2c5f9-b95a-40bf-a149-56b6e98625e9-metrics-certs\") pod \"maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7\" (UID: \"c2e2c5f9-b95a-40bf-a149-56b6e98625e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7"
Sep 18 17:27:35.719343 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:35.719262    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e2c5f9-b95a-40bf-a149-56b6e98625e9-openshift-service-ca\") pod \"maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7\" (UID: \"c2e2c5f9-b95a-40bf-a149-56b6e98625e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7"
Sep 18 17:27:35.820435 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:35.820405    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2e2c5f9-b95a-40bf-a149-56b6e98625e9-metrics-certs\") pod \"maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7\" (UID: \"c2e2c5f9-b95a-40bf-a149-56b6e98625e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7"
Sep 18 17:27:35.820596 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:35.820440    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e2c5f9-b95a-40bf-a149-56b6e98625e9-openshift-service-ca\") pod \"maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7\" (UID: \"c2e2c5f9-b95a-40bf-a149-56b6e98625e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7"
Sep 18 17:27:35.820596 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:35.820498    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf2f4\" (UniqueName: \"kubernetes.io/projected/c2e2c5f9-b95a-40bf-a149-56b6e98625e9-kube-api-access-kf2f4\") pod \"maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7\" (UID: \"c2e2c5f9-b95a-40bf-a149-56b6e98625e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7"
Sep 18 17:27:35.820596 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:35.820520    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e2c5f9-b95a-40bf-a149-56b6e98625e9-maas-api-tls\") pod \"maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7\" (UID: \"c2e2c5f9-b95a-40bf-a149-56b6e98625e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7"
Sep 18 17:27:35.820810 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:27:35.820673    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-fea15b54-serving-cert: secret "maas-api-e2e-migrate-fea15b54-serving-cert" not found
Sep 18 17:27:35.820810 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:27:35.820730    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2e2c5f9-b95a-40bf-a149-56b6e98625e9-maas-api-tls podName:c2e2c5f9-b95a-40bf-a149-56b6e98625e9 nodeName:}" failed. No retries permitted until 2026-09-18 17:27:36.320713421 +0000 UTC m=+769.014640927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c2e2c5f9-b95a-40bf-a149-56b6e98625e9-maas-api-tls") pod "maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7" (UID: "c2e2c5f9-b95a-40bf-a149-56b6e98625e9") : secret "maas-api-e2e-migrate-fea15b54-serving-cert" not found
Sep 18 17:27:35.821146 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:35.821118    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e2c5f9-b95a-40bf-a149-56b6e98625e9-openshift-service-ca\") pod \"maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7\" (UID: \"c2e2c5f9-b95a-40bf-a149-56b6e98625e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7"
Sep 18 17:27:35.823025 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:35.823002    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2e2c5f9-b95a-40bf-a149-56b6e98625e9-metrics-certs\") pod \"maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7\" (UID: \"c2e2c5f9-b95a-40bf-a149-56b6e98625e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7"
Sep 18 17:27:35.831649 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:35.831626    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf2f4\" (UniqueName: \"kubernetes.io/projected/c2e2c5f9-b95a-40bf-a149-56b6e98625e9-kube-api-access-kf2f4\") pod \"maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7\" (UID: \"c2e2c5f9-b95a-40bf-a149-56b6e98625e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7"
Sep 18 17:27:36.284341 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:36.284306    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fea15b54-84449d857d-xvvrh"]
Sep 18 17:27:36.288131 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:36.288113    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fea15b54-84449d857d-xvvrh"
Sep 18 17:27:36.292302 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:36.292280    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-fea15b54\""
Sep 18 17:27:36.316363 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:36.316339    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fea15b54-84449d857d-xvvrh"]
Sep 18 17:27:36.325380 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:36.325359    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aadebca-94a0-4274-9e59-1e8d844acd55-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fea15b54-84449d857d-xvvrh\" (UID: \"2aadebca-94a0-4274-9e59-1e8d844acd55\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fea15b54-84449d857d-xvvrh"
Sep 18 17:27:36.325483 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:36.325397    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc2hg\" (UniqueName: \"kubernetes.io/projected/2aadebca-94a0-4274-9e59-1e8d844acd55-kube-api-access-hc2hg\") pod \"payload-pre-processing-e2e-migrate-fea15b54-84449d857d-xvvrh\" (UID: \"2aadebca-94a0-4274-9e59-1e8d844acd55\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fea15b54-84449d857d-xvvrh"
Sep 18 17:27:36.325483 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:36.325436    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e2c5f9-b95a-40bf-a149-56b6e98625e9-maas-api-tls\") pod \"maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7\" (UID: \"c2e2c5f9-b95a-40bf-a149-56b6e98625e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7"
Sep 18 17:27:36.327681 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:36.327658    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e2c5f9-b95a-40bf-a149-56b6e98625e9-maas-api-tls\") pod \"maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7\" (UID: \"c2e2c5f9-b95a-40bf-a149-56b6e98625e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7"
Sep 18 17:27:36.360240 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:36.360209    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fea15b54-6c8b5c75db-5srkn"]
Sep 18 17:27:36.363256 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:36.363239    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fea15b54-6c8b5c75db-5srkn"
Sep 18 17:27:36.367488 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:36.367467    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-fea15b54-dockercfg-g95f4\""
Sep 18 17:27:36.397311 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:36.397282    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fea15b54-6c8b5c75db-5srkn"]
Sep 18 17:27:36.426398 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:36.426371    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aadebca-94a0-4274-9e59-1e8d844acd55-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fea15b54-84449d857d-xvvrh\" (UID: \"2aadebca-94a0-4274-9e59-1e8d844acd55\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fea15b54-84449d857d-xvvrh"
Sep 18 17:27:36.426541 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:36.426414    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc2hg\" (UniqueName: \"kubernetes.io/projected/2aadebca-94a0-4274-9e59-1e8d844acd55-kube-api-access-hc2hg\") pod \"payload-pre-processing-e2e-migrate-fea15b54-84449d857d-xvvrh\" (UID: \"2aadebca-94a0-4274-9e59-1e8d844acd55\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fea15b54-84449d857d-xvvrh"
Sep 18 17:27:36.426541 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:36.426439    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r74ln\" (UniqueName: \"kubernetes.io/projected/532404e6-1144-47c1-86e0-9e5f33573717-kube-api-access-r74ln\") pod \"payload-processing-e2e-migrate-fea15b54-6c8b5c75db-5srkn\" (UID: \"532404e6-1144-47c1-86e0-9e5f33573717\") " pod="openshift-ingress/payload-processing-e2e-migrate-fea15b54-6c8b5c75db-5srkn"
Sep 18 17:27:36.426541 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:36.426495    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/532404e6-1144-47c1-86e0-9e5f33573717-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fea15b54-6c8b5c75db-5srkn\" (UID: \"532404e6-1144-47c1-86e0-9e5f33573717\") " pod="openshift-ingress/payload-processing-e2e-migrate-fea15b54-6c8b5c75db-5srkn"
Sep 18 17:27:36.427055 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:36.427035    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aadebca-94a0-4274-9e59-1e8d844acd55-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fea15b54-84449d857d-xvvrh\" (UID: \"2aadebca-94a0-4274-9e59-1e8d844acd55\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fea15b54-84449d857d-xvvrh"
Sep 18 17:27:36.447535 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:36.447510    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc2hg\" (UniqueName: \"kubernetes.io/projected/2aadebca-94a0-4274-9e59-1e8d844acd55-kube-api-access-hc2hg\") pod \"payload-pre-processing-e2e-migrate-fea15b54-84449d857d-xvvrh\" (UID: \"2aadebca-94a0-4274-9e59-1e8d844acd55\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fea15b54-84449d857d-xvvrh"
Sep 18 17:27:36.527349 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:36.527315    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r74ln\" (UniqueName: \"kubernetes.io/projected/532404e6-1144-47c1-86e0-9e5f33573717-kube-api-access-r74ln\") pod \"payload-processing-e2e-migrate-fea15b54-6c8b5c75db-5srkn\" (UID: \"532404e6-1144-47c1-86e0-9e5f33573717\") " pod="openshift-ingress/payload-processing-e2e-migrate-fea15b54-6c8b5c75db-5srkn"
Sep 18 17:27:36.527488 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:36.527367    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/532404e6-1144-47c1-86e0-9e5f33573717-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fea15b54-6c8b5c75db-5srkn\" (UID: \"532404e6-1144-47c1-86e0-9e5f33573717\") " pod="openshift-ingress/payload-processing-e2e-migrate-fea15b54-6c8b5c75db-5srkn"
Sep 18 17:27:36.528019 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:36.528001    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/532404e6-1144-47c1-86e0-9e5f33573717-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fea15b54-6c8b5c75db-5srkn\" (UID: \"532404e6-1144-47c1-86e0-9e5f33573717\") " pod="openshift-ingress/payload-processing-e2e-migrate-fea15b54-6c8b5c75db-5srkn"
Sep 18 17:27:36.539431 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:36.539362    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r74ln\" (UniqueName: \"kubernetes.io/projected/532404e6-1144-47c1-86e0-9e5f33573717-kube-api-access-r74ln\") pod \"payload-processing-e2e-migrate-fea15b54-6c8b5c75db-5srkn\" (UID: \"532404e6-1144-47c1-86e0-9e5f33573717\") " pod="openshift-ingress/payload-processing-e2e-migrate-fea15b54-6c8b5c75db-5srkn"
Sep 18 17:27:36.541061 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:36.541033    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7"
Sep 18 17:27:36.598524 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:36.598305    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fea15b54-84449d857d-xvvrh"
Sep 18 17:27:36.673247 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:36.673219    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fea15b54-6c8b5c75db-5srkn"
Sep 18 17:27:36.813389 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:36.813365    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7"]
Sep 18 17:27:36.815453 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:27:36.815424    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2e2c5f9_b95a_40bf_a149_56b6e98625e9.slice/crio-cd5847bbd9d694d0277fc576eb29808518c67b3088ef20884b6101cb24a2c06b WatchSource:0}: Error finding container cd5847bbd9d694d0277fc576eb29808518c67b3088ef20884b6101cb24a2c06b: Status 404 returned error can't find the container with id cd5847bbd9d694d0277fc576eb29808518c67b3088ef20884b6101cb24a2c06b
Sep 18 17:27:36.874027 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:36.873984    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fea15b54-84449d857d-xvvrh"]
Sep 18 17:27:36.876554 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:27:36.876527    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aadebca_94a0_4274_9e59_1e8d844acd55.slice/crio-11c820528d67ea43db0a5aa281acfa0e65a6b6331201403483edbd1bd4c1e8f2 WatchSource:0}: Error finding container 11c820528d67ea43db0a5aa281acfa0e65a6b6331201403483edbd1bd4c1e8f2: Status 404 returned error can't find the container with id 11c820528d67ea43db0a5aa281acfa0e65a6b6331201403483edbd1bd4c1e8f2
Sep 18 17:27:36.949182 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:36.949156    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fea15b54-6c8b5c75db-5srkn"]
Sep 18 17:27:36.950930 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:27:36.950902    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod532404e6_1144_47c1_86e0_9e5f33573717.slice/crio-0c660bbb6c0458c306558f6e76fa30a62c5c1796c037d094fd8429b38635a4bb WatchSource:0}: Error finding container 0c660bbb6c0458c306558f6e76fa30a62c5c1796c037d094fd8429b38635a4bb: Status 404 returned error can't find the container with id 0c660bbb6c0458c306558f6e76fa30a62c5c1796c037d094fd8429b38635a4bb
Sep 18 17:27:37.184132 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:37.184028    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fea15b54-84449d857d-xvvrh" event={"ID":"2aadebca-94a0-4274-9e59-1e8d844acd55","Type":"ContainerStarted","Data":"099af70d2a9267997b387d74381fb64f6dbf640e042af3360a53140ed839fb3d"}
Sep 18 17:27:37.184132 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:37.184092    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fea15b54-84449d857d-xvvrh" event={"ID":"2aadebca-94a0-4274-9e59-1e8d844acd55","Type":"ContainerStarted","Data":"11c820528d67ea43db0a5aa281acfa0e65a6b6331201403483edbd1bd4c1e8f2"}
Sep 18 17:27:37.184371 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:37.184147    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fea15b54-84449d857d-xvvrh"
Sep 18 17:27:37.185550 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:37.185516    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7" event={"ID":"c2e2c5f9-b95a-40bf-a149-56b6e98625e9","Type":"ContainerStarted","Data":"cd5847bbd9d694d0277fc576eb29808518c67b3088ef20884b6101cb24a2c06b"}
Sep 18 17:27:37.187119 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:37.187094    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fea15b54-6c8b5c75db-5srkn" event={"ID":"532404e6-1144-47c1-86e0-9e5f33573717","Type":"ContainerStarted","Data":"2f83a4d5aa5304f5438afe3e741fa62f7fc2cf90033a7af21206864dc2c1fc11"}
Sep 18 17:27:37.187210 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:37.187141    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fea15b54-6c8b5c75db-5srkn" event={"ID":"532404e6-1144-47c1-86e0-9e5f33573717","Type":"ContainerStarted","Data":"0c660bbb6c0458c306558f6e76fa30a62c5c1796c037d094fd8429b38635a4bb"}
Sep 18 17:27:37.187210 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:37.187166    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-fea15b54-6c8b5c75db-5srkn"
Sep 18 17:27:37.207234 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:37.207190    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fea15b54-84449d857d-xvvrh" podStartSLOduration=1.207178114 podStartE2EDuration="1.207178114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:27:37.206652142 +0000 UTC m=+769.900579694" watchObservedRunningTime="2026-09-18 17:27:37.207178114 +0000 UTC m=+769.901105642"
Sep 18 17:27:37.245800 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:37.245735    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-fea15b54-6c8b5c75db-5srkn" podStartSLOduration=1.245715563 podStartE2EDuration="1.245715563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:27:37.241969204 +0000 UTC m=+769.935896751" watchObservedRunningTime="2026-09-18 17:27:37.245715563 +0000 UTC m=+769.939643093"
Sep 18 17:27:38.193919 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:38.193878    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7" event={"ID":"c2e2c5f9-b95a-40bf-a149-56b6e98625e9","Type":"ContainerStarted","Data":"2de903bbb0b81a0adeab1ec97e7fff60987cc296f607e72231a793a5fe46afc4"}
Sep 18 17:27:38.194388 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:38.194105    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7"
Sep 18 17:27:38.216585 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:38.216542    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7" podStartSLOduration=2.324917873 podStartE2EDuration="3.216525779s" totalImagesPullingTime="891.607906ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:27:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:27:38.213716973 +0000 UTC m=+770.907644512" watchObservedRunningTime="2026-09-18 17:27:38.216525779 +0000 UTC m=+770.910453307"
Sep 18 17:27:44.205450 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:44.205420    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7"
Sep 18 17:27:48.195928 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:48.195832    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-fea15b54-6c8b5c75db-5srkn"
Sep 18 17:27:48.196366 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:27:48.196075    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fea15b54-84449d857d-xvvrh"
Sep 18 17:28:02.558562 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:02.558528    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fea15b54-8x8nd"]
Sep 18 17:28:02.566581 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:02.566563    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fea15b54-8x8nd"
Sep 18 17:28:02.586338 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:02.586312    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qt978\""
Sep 18 17:28:02.594752 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:02.594710    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fea15b54-8x8nd"]
Sep 18 17:28:02.665049 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:02.665022    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/066c44c3-c28e-4c8a-8397-7a6ee7965535-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fea15b54-8x8nd\" (UID: \"066c44c3-c28e-4c8a-8397-7a6ee7965535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fea15b54-8x8nd"
Sep 18 17:28:02.765669 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:02.765633    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/066c44c3-c28e-4c8a-8397-7a6ee7965535-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fea15b54-8x8nd\" (UID: \"066c44c3-c28e-4c8a-8397-7a6ee7965535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fea15b54-8x8nd"
Sep 18 17:28:02.766246 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:02.766226    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/066c44c3-c28e-4c8a-8397-7a6ee7965535-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fea15b54-8x8nd\" (UID: \"066c44c3-c28e-4c8a-8397-7a6ee7965535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fea15b54-8x8nd"
Sep 18 17:28:02.876567 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:02.876489    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fea15b54-8x8nd"
Sep 18 17:28:03.152710 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:03.152682    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fea15b54-8x8nd"]
Sep 18 17:28:03.154666 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:28:03.154636    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod066c44c3_c28e_4c8a_8397_7a6ee7965535.slice/crio-6b4cdbd91717633905b89788fc8cdd83a8fe6d79ab39485cb71887d0015fa82b WatchSource:0}: Error finding container 6b4cdbd91717633905b89788fc8cdd83a8fe6d79ab39485cb71887d0015fa82b: Status 404 returned error can't find the container with id 6b4cdbd91717633905b89788fc8cdd83a8fe6d79ab39485cb71887d0015fa82b
Sep 18 17:28:03.332265 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:03.332221    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fea15b54-8x8nd" event={"ID":"066c44c3-c28e-4c8a-8397-7a6ee7965535","Type":"ContainerStarted","Data":"6b4cdbd91717633905b89788fc8cdd83a8fe6d79ab39485cb71887d0015fa82b"}
Sep 18 17:28:04.338020 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:04.337987    2569 generic.go:356] "Generic (PLEG): container finished" podID="066c44c3-c28e-4c8a-8397-7a6ee7965535" containerID="27591c77892acce9c82a63ce6933ef7749920ca7fdd693c1ccda782c689b94e7" exitCode=0
Sep 18 17:28:04.338398 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:04.338072    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fea15b54-8x8nd" event={"ID":"066c44c3-c28e-4c8a-8397-7a6ee7965535","Type":"ContainerDied","Data":"27591c77892acce9c82a63ce6933ef7749920ca7fdd693c1ccda782c689b94e7"}
Sep 18 17:28:05.494168 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:05.494143    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fea15b54-8x8nd"
Sep 18 17:28:05.592061 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:05.592021    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/066c44c3-c28e-4c8a-8397-7a6ee7965535-maas-api-service-ca\") pod \"066c44c3-c28e-4c8a-8397-7a6ee7965535\" (UID: \"066c44c3-c28e-4c8a-8397-7a6ee7965535\") "
Sep 18 17:28:05.592366 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:05.592341    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/066c44c3-c28e-4c8a-8397-7a6ee7965535-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "066c44c3-c28e-4c8a-8397-7a6ee7965535" (UID: "066c44c3-c28e-4c8a-8397-7a6ee7965535"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:28:05.693256 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:05.693179    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/066c44c3-c28e-4c8a-8397-7a6ee7965535-maas-api-service-ca\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:28:06.349549 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:06.349518    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fea15b54-8x8nd"
Sep 18 17:28:06.349735 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:06.349561    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fea15b54-8x8nd" event={"ID":"066c44c3-c28e-4c8a-8397-7a6ee7965535","Type":"ContainerDied","Data":"6b4cdbd91717633905b89788fc8cdd83a8fe6d79ab39485cb71887d0015fa82b"}
Sep 18 17:28:06.349735 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:06.349618    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b4cdbd91717633905b89788fc8cdd83a8fe6d79ab39485cb71887d0015fa82b"
Sep 18 17:28:14.005427 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.005335    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7"]
Sep 18 17:28:14.005971 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.005564    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7" podUID="c2e2c5f9-b95a-40bf-a149-56b6e98625e9" containerName="maas-api" containerID="cri-o://2de903bbb0b81a0adeab1ec97e7fff60987cc296f607e72231a793a5fe46afc4" gracePeriod=30
Sep 18 17:28:14.062656 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.062603    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fea15b54-6c8b5c75db-5srkn"]
Sep 18 17:28:14.063043 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.063008    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-fea15b54-6c8b5c75db-5srkn" podUID="532404e6-1144-47c1-86e0-9e5f33573717" containerName="payload-processing" containerID="cri-o://2f83a4d5aa5304f5438afe3e741fa62f7fc2cf90033a7af21206864dc2c1fc11" gracePeriod=30
Sep 18 17:28:14.065056 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.065029    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fea15b54-84449d857d-xvvrh"]
Sep 18 17:28:14.065265 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.065228    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fea15b54-84449d857d-xvvrh" podUID="2aadebca-94a0-4274-9e59-1e8d844acd55" containerName="payload-pre-processing" containerID="cri-o://099af70d2a9267997b387d74381fb64f6dbf640e042af3360a53140ed839fb3d" gracePeriod=30
Sep 18 17:28:14.195165 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.195104    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fea15b54-8x8nd"]
Sep 18 17:28:14.197975 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.197920    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fea15b54-8x8nd"]
Sep 18 17:28:14.200952 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.200912    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7" podUID="c2e2c5f9-b95a-40bf-a149-56b6e98625e9" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.50:8443/health\": dial tcp 10.133.0.50:8443: connect: connection refused"
Sep 18 17:28:14.326307 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.326275    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7"
Sep 18 17:28:14.357995 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.357968    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fea15b54-6c8b5c75db-5srkn"
Sep 18 17:28:14.362565 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.362538    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fea15b54-84449d857d-xvvrh"
Sep 18 17:28:14.367028 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.367006    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kf2f4\" (UniqueName: \"kubernetes.io/projected/c2e2c5f9-b95a-40bf-a149-56b6e98625e9-kube-api-access-kf2f4\") pod \"c2e2c5f9-b95a-40bf-a149-56b6e98625e9\" (UID: \"c2e2c5f9-b95a-40bf-a149-56b6e98625e9\") "
Sep 18 17:28:14.367175 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.367038    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2e2c5f9-b95a-40bf-a149-56b6e98625e9-metrics-certs\") pod \"c2e2c5f9-b95a-40bf-a149-56b6e98625e9\" (UID: \"c2e2c5f9-b95a-40bf-a149-56b6e98625e9\") "
Sep 18 17:28:14.367175 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.367096    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r74ln\" (UniqueName: \"kubernetes.io/projected/532404e6-1144-47c1-86e0-9e5f33573717-kube-api-access-r74ln\") pod \"532404e6-1144-47c1-86e0-9e5f33573717\" (UID: \"532404e6-1144-47c1-86e0-9e5f33573717\") "
Sep 18 17:28:14.367175 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.367140    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e2c5f9-b95a-40bf-a149-56b6e98625e9-maas-api-tls\") pod \"c2e2c5f9-b95a-40bf-a149-56b6e98625e9\" (UID: \"c2e2c5f9-b95a-40bf-a149-56b6e98625e9\") "
Sep 18 17:28:14.367175 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.367156    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/532404e6-1144-47c1-86e0-9e5f33573717-plugins-config-volume\") pod \"532404e6-1144-47c1-86e0-9e5f33573717\" (UID: \"532404e6-1144-47c1-86e0-9e5f33573717\") "
Sep 18 17:28:14.367408 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.367198    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e2c5f9-b95a-40bf-a149-56b6e98625e9-openshift-service-ca\") pod \"c2e2c5f9-b95a-40bf-a149-56b6e98625e9\" (UID: \"c2e2c5f9-b95a-40bf-a149-56b6e98625e9\") "
Sep 18 17:28:14.367761 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.367721    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2e2c5f9-b95a-40bf-a149-56b6e98625e9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c2e2c5f9-b95a-40bf-a149-56b6e98625e9" (UID: "c2e2c5f9-b95a-40bf-a149-56b6e98625e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:28:14.367998 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.367942    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/532404e6-1144-47c1-86e0-9e5f33573717-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "532404e6-1144-47c1-86e0-9e5f33573717" (UID: "532404e6-1144-47c1-86e0-9e5f33573717"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:28:14.369537 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.369513    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/532404e6-1144-47c1-86e0-9e5f33573717-kube-api-access-r74ln" (OuterVolumeSpecName: "kube-api-access-r74ln") pod "532404e6-1144-47c1-86e0-9e5f33573717" (UID: "532404e6-1144-47c1-86e0-9e5f33573717"). InnerVolumeSpecName "kube-api-access-r74ln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:28:14.369836 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.369813    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2e2c5f9-b95a-40bf-a149-56b6e98625e9-kube-api-access-kf2f4" (OuterVolumeSpecName: "kube-api-access-kf2f4") pod "c2e2c5f9-b95a-40bf-a149-56b6e98625e9" (UID: "c2e2c5f9-b95a-40bf-a149-56b6e98625e9"). InnerVolumeSpecName "kube-api-access-kf2f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:28:14.369836 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.369824    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2e2c5f9-b95a-40bf-a149-56b6e98625e9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c2e2c5f9-b95a-40bf-a149-56b6e98625e9" (UID: "c2e2c5f9-b95a-40bf-a149-56b6e98625e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:28:14.370081 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.369987    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2e2c5f9-b95a-40bf-a149-56b6e98625e9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c2e2c5f9-b95a-40bf-a149-56b6e98625e9" (UID: "c2e2c5f9-b95a-40bf-a149-56b6e98625e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:28:14.392044 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.392008    2569 generic.go:356] "Generic (PLEG): container finished" podID="532404e6-1144-47c1-86e0-9e5f33573717" containerID="2f83a4d5aa5304f5438afe3e741fa62f7fc2cf90033a7af21206864dc2c1fc11" exitCode=0
Sep 18 17:28:14.392194 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.392046    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fea15b54-6c8b5c75db-5srkn"
Sep 18 17:28:14.392194 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.392086    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fea15b54-6c8b5c75db-5srkn" event={"ID":"532404e6-1144-47c1-86e0-9e5f33573717","Type":"ContainerDied","Data":"2f83a4d5aa5304f5438afe3e741fa62f7fc2cf90033a7af21206864dc2c1fc11"}
Sep 18 17:28:14.392194 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.392170    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fea15b54-6c8b5c75db-5srkn" event={"ID":"532404e6-1144-47c1-86e0-9e5f33573717","Type":"ContainerDied","Data":"0c660bbb6c0458c306558f6e76fa30a62c5c1796c037d094fd8429b38635a4bb"}
Sep 18 17:28:14.392355 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.392224    2569 scope.go:117] "RemoveContainer" containerID="2f83a4d5aa5304f5438afe3e741fa62f7fc2cf90033a7af21206864dc2c1fc11"
Sep 18 17:28:14.393674 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.393648    2569 generic.go:356] "Generic (PLEG): container finished" podID="2aadebca-94a0-4274-9e59-1e8d844acd55" containerID="099af70d2a9267997b387d74381fb64f6dbf640e042af3360a53140ed839fb3d" exitCode=0
Sep 18 17:28:14.393779 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.393694    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fea15b54-84449d857d-xvvrh"
Sep 18 17:28:14.393779 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.393725    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fea15b54-84449d857d-xvvrh" event={"ID":"2aadebca-94a0-4274-9e59-1e8d844acd55","Type":"ContainerDied","Data":"099af70d2a9267997b387d74381fb64f6dbf640e042af3360a53140ed839fb3d"}
Sep 18 17:28:14.393926 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.393770    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fea15b54-84449d857d-xvvrh" event={"ID":"2aadebca-94a0-4274-9e59-1e8d844acd55","Type":"ContainerDied","Data":"11c820528d67ea43db0a5aa281acfa0e65a6b6331201403483edbd1bd4c1e8f2"}
Sep 18 17:28:14.395317 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.395293    2569 generic.go:356] "Generic (PLEG): container finished" podID="c2e2c5f9-b95a-40bf-a149-56b6e98625e9" containerID="2de903bbb0b81a0adeab1ec97e7fff60987cc296f607e72231a793a5fe46afc4" exitCode=0
Sep 18 17:28:14.395437 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.395336    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7" event={"ID":"c2e2c5f9-b95a-40bf-a149-56b6e98625e9","Type":"ContainerDied","Data":"2de903bbb0b81a0adeab1ec97e7fff60987cc296f607e72231a793a5fe46afc4"}
Sep 18 17:28:14.395437 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.395374    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7" event={"ID":"c2e2c5f9-b95a-40bf-a149-56b6e98625e9","Type":"ContainerDied","Data":"cd5847bbd9d694d0277fc576eb29808518c67b3088ef20884b6101cb24a2c06b"}
Sep 18 17:28:14.395437 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.395339    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7"
Sep 18 17:28:14.402302 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.402278    2569 scope.go:117] "RemoveContainer" containerID="2f83a4d5aa5304f5438afe3e741fa62f7fc2cf90033a7af21206864dc2c1fc11"
Sep 18 17:28:14.402543 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:28:14.402525    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f83a4d5aa5304f5438afe3e741fa62f7fc2cf90033a7af21206864dc2c1fc11\": container with ID starting with 2f83a4d5aa5304f5438afe3e741fa62f7fc2cf90033a7af21206864dc2c1fc11 not found: ID does not exist" containerID="2f83a4d5aa5304f5438afe3e741fa62f7fc2cf90033a7af21206864dc2c1fc11"
Sep 18 17:28:14.402642 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.402549    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f83a4d5aa5304f5438afe3e741fa62f7fc2cf90033a7af21206864dc2c1fc11"} err="failed to get container status \"2f83a4d5aa5304f5438afe3e741fa62f7fc2cf90033a7af21206864dc2c1fc11\": rpc error: code = NotFound desc = could not find container \"2f83a4d5aa5304f5438afe3e741fa62f7fc2cf90033a7af21206864dc2c1fc11\": container with ID starting with 2f83a4d5aa5304f5438afe3e741fa62f7fc2cf90033a7af21206864dc2c1fc11 not found: ID does not exist"
Sep 18 17:28:14.402642 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.402569    2569 scope.go:117] "RemoveContainer" containerID="099af70d2a9267997b387d74381fb64f6dbf640e042af3360a53140ed839fb3d"
Sep 18 17:28:14.417053 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.417029    2569 scope.go:117] "RemoveContainer" containerID="099af70d2a9267997b387d74381fb64f6dbf640e042af3360a53140ed839fb3d"
Sep 18 17:28:14.417620 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:28:14.417555    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"099af70d2a9267997b387d74381fb64f6dbf640e042af3360a53140ed839fb3d\": container with ID starting with 099af70d2a9267997b387d74381fb64f6dbf640e042af3360a53140ed839fb3d not found: ID does not exist" containerID="099af70d2a9267997b387d74381fb64f6dbf640e042af3360a53140ed839fb3d"
Sep 18 17:28:14.417620 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.417603    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"099af70d2a9267997b387d74381fb64f6dbf640e042af3360a53140ed839fb3d"} err="failed to get container status \"099af70d2a9267997b387d74381fb64f6dbf640e042af3360a53140ed839fb3d\": rpc error: code = NotFound desc = could not find container \"099af70d2a9267997b387d74381fb64f6dbf640e042af3360a53140ed839fb3d\": container with ID starting with 099af70d2a9267997b387d74381fb64f6dbf640e042af3360a53140ed839fb3d not found: ID does not exist"
Sep 18 17:28:14.417998 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.417623    2569 scope.go:117] "RemoveContainer" containerID="2de903bbb0b81a0adeab1ec97e7fff60987cc296f607e72231a793a5fe46afc4"
Sep 18 17:28:14.419201 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.419179    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fea15b54-6c8b5c75db-5srkn"]
Sep 18 17:28:14.421048 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.421026    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fea15b54-6c8b5c75db-5srkn"]
Sep 18 17:28:14.426421 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.426407    2569 scope.go:117] "RemoveContainer" containerID="2de903bbb0b81a0adeab1ec97e7fff60987cc296f607e72231a793a5fe46afc4"
Sep 18 17:28:14.426696 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:28:14.426678    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2de903bbb0b81a0adeab1ec97e7fff60987cc296f607e72231a793a5fe46afc4\": container with ID starting with 2de903bbb0b81a0adeab1ec97e7fff60987cc296f607e72231a793a5fe46afc4 not found: ID does not exist" containerID="2de903bbb0b81a0adeab1ec97e7fff60987cc296f607e72231a793a5fe46afc4"
Sep 18 17:28:14.426765 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.426703    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2de903bbb0b81a0adeab1ec97e7fff60987cc296f607e72231a793a5fe46afc4"} err="failed to get container status \"2de903bbb0b81a0adeab1ec97e7fff60987cc296f607e72231a793a5fe46afc4\": rpc error: code = NotFound desc = could not find container \"2de903bbb0b81a0adeab1ec97e7fff60987cc296f607e72231a793a5fe46afc4\": container with ID starting with 2de903bbb0b81a0adeab1ec97e7fff60987cc296f607e72231a793a5fe46afc4 not found: ID does not exist"
Sep 18 17:28:14.436649 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.436623    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7"]
Sep 18 17:28:14.440350 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.440327    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fea15b54-58b69b97c8-h59z7"]
Sep 18 17:28:14.467814 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.467788    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hc2hg\" (UniqueName: \"kubernetes.io/projected/2aadebca-94a0-4274-9e59-1e8d844acd55-kube-api-access-hc2hg\") pod \"2aadebca-94a0-4274-9e59-1e8d844acd55\" (UID: \"2aadebca-94a0-4274-9e59-1e8d844acd55\") "
Sep 18 17:28:14.467978 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.467826    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aadebca-94a0-4274-9e59-1e8d844acd55-plugins-config-volume\") pod \"2aadebca-94a0-4274-9e59-1e8d844acd55\" (UID: \"2aadebca-94a0-4274-9e59-1e8d844acd55\") "
Sep 18 17:28:14.468080 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.468067    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2e2c5f9-b95a-40bf-a149-56b6e98625e9-maas-api-tls\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:28:14.468137 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.468082    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/532404e6-1144-47c1-86e0-9e5f33573717-plugins-config-volume\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:28:14.468137 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.468092    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e2c5f9-b95a-40bf-a149-56b6e98625e9-openshift-service-ca\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:28:14.468137 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.468100    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf2f4\" (UniqueName: \"kubernetes.io/projected/c2e2c5f9-b95a-40bf-a149-56b6e98625e9-kube-api-access-kf2f4\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:28:14.468137 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.468108    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2e2c5f9-b95a-40bf-a149-56b6e98625e9-metrics-certs\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:28:14.468137 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.468115    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r74ln\" (UniqueName: \"kubernetes.io/projected/532404e6-1144-47c1-86e0-9e5f33573717-kube-api-access-r74ln\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:28:14.468382 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.468297    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aadebca-94a0-4274-9e59-1e8d844acd55-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2aadebca-94a0-4274-9e59-1e8d844acd55" (UID: "2aadebca-94a0-4274-9e59-1e8d844acd55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:28:14.469812 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.469791    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aadebca-94a0-4274-9e59-1e8d844acd55-kube-api-access-hc2hg" (OuterVolumeSpecName: "kube-api-access-hc2hg") pod "2aadebca-94a0-4274-9e59-1e8d844acd55" (UID: "2aadebca-94a0-4274-9e59-1e8d844acd55"). InnerVolumeSpecName "kube-api-access-hc2hg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:28:14.569273 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.569193    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc2hg\" (UniqueName: \"kubernetes.io/projected/2aadebca-94a0-4274-9e59-1e8d844acd55-kube-api-access-hc2hg\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:28:14.569273 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.569220    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aadebca-94a0-4274-9e59-1e8d844acd55-plugins-config-volume\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:28:14.719098 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.719068    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fea15b54-84449d857d-xvvrh"]
Sep 18 17:28:14.728496 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:14.728462    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fea15b54-84449d857d-xvvrh"]
Sep 18 17:28:15.765240 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:15.765209    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="066c44c3-c28e-4c8a-8397-7a6ee7965535" path="/var/lib/kubelet/pods/066c44c3-c28e-4c8a-8397-7a6ee7965535/volumes"
Sep 18 17:28:15.765744 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:15.765722    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2aadebca-94a0-4274-9e59-1e8d844acd55" path="/var/lib/kubelet/pods/2aadebca-94a0-4274-9e59-1e8d844acd55/volumes"
Sep 18 17:28:15.766134 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:15.766115    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="532404e6-1144-47c1-86e0-9e5f33573717" path="/var/lib/kubelet/pods/532404e6-1144-47c1-86e0-9e5f33573717/volumes"
Sep 18 17:28:15.766516 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:15.766499    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2e2c5f9-b95a-40bf-a149-56b6e98625e9" path="/var/lib/kubelet/pods/c2e2c5f9-b95a-40bf-a149-56b6e98625e9/volumes"
Sep 18 17:28:18.948590 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:18.948556    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd"]
Sep 18 17:28:18.949059 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:18.948930    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aadebca-94a0-4274-9e59-1e8d844acd55" containerName="payload-pre-processing"
Sep 18 17:28:18.949059 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:18.948941    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aadebca-94a0-4274-9e59-1e8d844acd55" containerName="payload-pre-processing"
Sep 18 17:28:18.949059 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:18.948949    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="532404e6-1144-47c1-86e0-9e5f33573717" containerName="payload-processing"
Sep 18 17:28:18.949059 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:18.948955    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="532404e6-1144-47c1-86e0-9e5f33573717" containerName="payload-processing"
Sep 18 17:28:18.949059 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:18.948963    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2e2c5f9-b95a-40bf-a149-56b6e98625e9" containerName="maas-api"
Sep 18 17:28:18.949059 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:18.948969    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2e2c5f9-b95a-40bf-a149-56b6e98625e9" containerName="maas-api"
Sep 18 17:28:18.949059 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:18.948994    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="066c44c3-c28e-4c8a-8397-7a6ee7965535" containerName="revoke-keys"
Sep 18 17:28:18.949059 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:18.949000    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="066c44c3-c28e-4c8a-8397-7a6ee7965535" containerName="revoke-keys"
Sep 18 17:28:18.949318 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:18.949066    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="532404e6-1144-47c1-86e0-9e5f33573717" containerName="payload-processing"
Sep 18 17:28:18.949318 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:18.949076    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2aadebca-94a0-4274-9e59-1e8d844acd55" containerName="payload-pre-processing"
Sep 18 17:28:18.949318 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:18.949083    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="066c44c3-c28e-4c8a-8397-7a6ee7965535" containerName="revoke-keys"
Sep 18 17:28:18.949318 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:18.949090    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2e2c5f9-b95a-40bf-a149-56b6e98625e9" containerName="maas-api"
Sep 18 17:28:18.953688 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:18.953671    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd"
Sep 18 17:28:18.957558 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:18.957529    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-5b6d69\"/\"openshift-service-ca.crt\""
Sep 18 17:28:18.958400 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:18.958382    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-5b6d69\"/\"default-dockercfg-r86nm\""
Sep 18 17:28:18.958511 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:18.958419    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-5b6d69\"/\"kube-root-ca.crt\""
Sep 18 17:28:18.958511 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:18.958420    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-5b6d69\"/\"simulator-w4-5b6d69-kserve-self-signed-certs\""
Sep 18 17:28:18.968804 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:18.968782    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd"]
Sep 18 17:28:19.007645 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:19.007612    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93943d04-0dd3-4f9d-828e-f41af19c5d86-home\") pod \"simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd\" (UID: \"93943d04-0dd3-4f9d-828e-f41af19c5d86\") " pod="e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd"
Sep 18 17:28:19.007815 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:19.007670    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93943d04-0dd3-4f9d-828e-f41af19c5d86-dshm\") pod \"simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd\" (UID: \"93943d04-0dd3-4f9d-828e-f41af19c5d86\") " pod="e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd"
Sep 18 17:28:19.007815 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:19.007697    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93943d04-0dd3-4f9d-828e-f41af19c5d86-tls-certs\") pod \"simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd\" (UID: \"93943d04-0dd3-4f9d-828e-f41af19c5d86\") " pod="e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd"
Sep 18 17:28:19.007815 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:19.007721    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93943d04-0dd3-4f9d-828e-f41af19c5d86-tmp-dir\") pod \"simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd\" (UID: \"93943d04-0dd3-4f9d-828e-f41af19c5d86\") " pod="e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd"
Sep 18 17:28:19.007815 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:19.007761    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93943d04-0dd3-4f9d-828e-f41af19c5d86-model-cache\") pod \"simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd\" (UID: \"93943d04-0dd3-4f9d-828e-f41af19c5d86\") " pod="e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd"
Sep 18 17:28:19.008000 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:19.007902    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4fht\" (UniqueName: \"kubernetes.io/projected/93943d04-0dd3-4f9d-828e-f41af19c5d86-kube-api-access-j4fht\") pod \"simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd\" (UID: \"93943d04-0dd3-4f9d-828e-f41af19c5d86\") " pod="e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd"
Sep 18 17:28:19.109217 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:19.109180    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93943d04-0dd3-4f9d-828e-f41af19c5d86-model-cache\") pod \"simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd\" (UID: \"93943d04-0dd3-4f9d-828e-f41af19c5d86\") " pod="e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd"
Sep 18 17:28:19.109385 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:19.109238    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4fht\" (UniqueName: \"kubernetes.io/projected/93943d04-0dd3-4f9d-828e-f41af19c5d86-kube-api-access-j4fht\") pod \"simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd\" (UID: \"93943d04-0dd3-4f9d-828e-f41af19c5d86\") " pod="e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd"
Sep 18 17:28:19.109385 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:19.109270    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93943d04-0dd3-4f9d-828e-f41af19c5d86-home\") pod \"simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd\" (UID: \"93943d04-0dd3-4f9d-828e-f41af19c5d86\") " pod="e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd"
Sep 18 17:28:19.109385 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:19.109301    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93943d04-0dd3-4f9d-828e-f41af19c5d86-dshm\") pod \"simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd\" (UID: \"93943d04-0dd3-4f9d-828e-f41af19c5d86\") " pod="e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd"
Sep 18 17:28:19.109385 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:19.109328    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93943d04-0dd3-4f9d-828e-f41af19c5d86-tls-certs\") pod \"simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd\" (UID: \"93943d04-0dd3-4f9d-828e-f41af19c5d86\") " pod="e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd"
Sep 18 17:28:19.109385 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:19.109359    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93943d04-0dd3-4f9d-828e-f41af19c5d86-tmp-dir\") pod \"simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd\" (UID: \"93943d04-0dd3-4f9d-828e-f41af19c5d86\") " pod="e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd"
Sep 18 17:28:19.109657 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:19.109634    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93943d04-0dd3-4f9d-828e-f41af19c5d86-model-cache\") pod \"simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd\" (UID: \"93943d04-0dd3-4f9d-828e-f41af19c5d86\") " pod="e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd"
Sep 18 17:28:19.109698 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:19.109672    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93943d04-0dd3-4f9d-828e-f41af19c5d86-home\") pod \"simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd\" (UID: \"93943d04-0dd3-4f9d-828e-f41af19c5d86\") " pod="e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd"
Sep 18 17:28:19.109743 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:19.109728    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93943d04-0dd3-4f9d-828e-f41af19c5d86-tmp-dir\") pod \"simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd\" (UID: \"93943d04-0dd3-4f9d-828e-f41af19c5d86\") " pod="e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd"
Sep 18 17:28:19.111507 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:19.111488    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93943d04-0dd3-4f9d-828e-f41af19c5d86-dshm\") pod \"simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd\" (UID: \"93943d04-0dd3-4f9d-828e-f41af19c5d86\") " pod="e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd"
Sep 18 17:28:19.111783 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:19.111765    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93943d04-0dd3-4f9d-828e-f41af19c5d86-tls-certs\") pod \"simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd\" (UID: \"93943d04-0dd3-4f9d-828e-f41af19c5d86\") " pod="e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd"
Sep 18 17:28:19.120073 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:19.120052    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4fht\" (UniqueName: \"kubernetes.io/projected/93943d04-0dd3-4f9d-828e-f41af19c5d86-kube-api-access-j4fht\") pod \"simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd\" (UID: \"93943d04-0dd3-4f9d-828e-f41af19c5d86\") " pod="e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd"
Sep 18 17:28:19.263031 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:19.262997    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd"
Sep 18 17:28:19.588953 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:19.588926    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd"]
Sep 18 17:28:19.591922 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:28:19.591894    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93943d04_0dd3_4f9d_828e_f41af19c5d86.slice/crio-9d49094189826f1095ea3f5cf54ff8e3878ac5de61655b0e66cba93597f0dee6 WatchSource:0}: Error finding container 9d49094189826f1095ea3f5cf54ff8e3878ac5de61655b0e66cba93597f0dee6: Status 404 returned error can't find the container with id 9d49094189826f1095ea3f5cf54ff8e3878ac5de61655b0e66cba93597f0dee6
Sep 18 17:28:20.428458 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:20.428421    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd" event={"ID":"93943d04-0dd3-4f9d-828e-f41af19c5d86","Type":"ContainerStarted","Data":"c2e648bfa94039f48fa7274dfcdab9e21a5c8075b534002ed9211d4389177982"}
Sep 18 17:28:20.428849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:20.428468    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd" event={"ID":"93943d04-0dd3-4f9d-828e-f41af19c5d86","Type":"ContainerStarted","Data":"9d49094189826f1095ea3f5cf54ff8e3878ac5de61655b0e66cba93597f0dee6"}
Sep 18 17:28:20.457424 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:20.457368    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd" podStartSLOduration=2.457352426 podStartE2EDuration="2.457352426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:28:20.454941177 +0000 UTC m=+813.148868724" watchObservedRunningTime="2026-09-18 17:28:20.457352426 +0000 UTC m=+813.151279955"
Sep 18 17:28:22.086313 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:22.086277    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt"]
Sep 18 17:28:22.089565 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:22.089545    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt"
Sep 18 17:28:22.098029 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:22.098012    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8bd53078-serving-cert\""
Sep 18 17:28:22.136323 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:22.136289    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2d35705-eefa-4ad4-be40-7bf3f4fcd392-maas-api-tls\") pod \"maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt\" (UID: \"d2d35705-eefa-4ad4-be40-7bf3f4fcd392\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt"
Sep 18 17:28:22.136470 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:22.136365    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2d35705-eefa-4ad4-be40-7bf3f4fcd392-openshift-service-ca\") pod \"maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt\" (UID: \"d2d35705-eefa-4ad4-be40-7bf3f4fcd392\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt"
Sep 18 17:28:22.136470 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:22.136400    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d35705-eefa-4ad4-be40-7bf3f4fcd392-metrics-certs\") pod \"maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt\" (UID: \"d2d35705-eefa-4ad4-be40-7bf3f4fcd392\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt"
Sep 18 17:28:22.136470 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:22.136424    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txgkm\" (UniqueName: \"kubernetes.io/projected/d2d35705-eefa-4ad4-be40-7bf3f4fcd392-kube-api-access-txgkm\") pod \"maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt\" (UID: \"d2d35705-eefa-4ad4-be40-7bf3f4fcd392\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt"
Sep 18 17:28:22.160997 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:22.160959    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt"]
Sep 18 17:28:22.237433 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:22.237396    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d35705-eefa-4ad4-be40-7bf3f4fcd392-metrics-certs\") pod \"maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt\" (UID: \"d2d35705-eefa-4ad4-be40-7bf3f4fcd392\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt"
Sep 18 17:28:22.237593 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:22.237440    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txgkm\" (UniqueName: \"kubernetes.io/projected/d2d35705-eefa-4ad4-be40-7bf3f4fcd392-kube-api-access-txgkm\") pod \"maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt\" (UID: \"d2d35705-eefa-4ad4-be40-7bf3f4fcd392\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt"
Sep 18 17:28:22.237593 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:22.237564    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2d35705-eefa-4ad4-be40-7bf3f4fcd392-maas-api-tls\") pod \"maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt\" (UID: \"d2d35705-eefa-4ad4-be40-7bf3f4fcd392\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt"
Sep 18 17:28:22.237690 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:22.237675    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2d35705-eefa-4ad4-be40-7bf3f4fcd392-openshift-service-ca\") pod \"maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt\" (UID: \"d2d35705-eefa-4ad4-be40-7bf3f4fcd392\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt"
Sep 18 17:28:22.237728 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:28:22.237694    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-8bd53078-serving-cert: secret "maas-api-e2e-ait-8bd53078-serving-cert" not found
Sep 18 17:28:22.237798 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:28:22.237784    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2d35705-eefa-4ad4-be40-7bf3f4fcd392-maas-api-tls podName:d2d35705-eefa-4ad4-be40-7bf3f4fcd392 nodeName:}" failed. No retries permitted until 2026-09-18 17:28:22.73776038 +0000 UTC m=+815.431687887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d2d35705-eefa-4ad4-be40-7bf3f4fcd392-maas-api-tls") pod "maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt" (UID: "d2d35705-eefa-4ad4-be40-7bf3f4fcd392") : secret "maas-api-e2e-ait-8bd53078-serving-cert" not found
Sep 18 17:28:22.238345 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:22.238320    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2d35705-eefa-4ad4-be40-7bf3f4fcd392-openshift-service-ca\") pod \"maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt\" (UID: \"d2d35705-eefa-4ad4-be40-7bf3f4fcd392\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt"
Sep 18 17:28:22.239873 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:22.239844    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d35705-eefa-4ad4-be40-7bf3f4fcd392-metrics-certs\") pod \"maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt\" (UID: \"d2d35705-eefa-4ad4-be40-7bf3f4fcd392\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt"
Sep 18 17:28:22.246175 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:22.246151    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txgkm\" (UniqueName: \"kubernetes.io/projected/d2d35705-eefa-4ad4-be40-7bf3f4fcd392-kube-api-access-txgkm\") pod \"maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt\" (UID: \"d2d35705-eefa-4ad4-be40-7bf3f4fcd392\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt"
Sep 18 17:28:22.742284 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:22.742255    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2d35705-eefa-4ad4-be40-7bf3f4fcd392-maas-api-tls\") pod \"maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt\" (UID: \"d2d35705-eefa-4ad4-be40-7bf3f4fcd392\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt"
Sep 18 17:28:22.744504 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:22.744476    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2d35705-eefa-4ad4-be40-7bf3f4fcd392-maas-api-tls\") pod \"maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt\" (UID: \"d2d35705-eefa-4ad4-be40-7bf3f4fcd392\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt"
Sep 18 17:28:22.895159 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:22.895122    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8bd53078-c5d887794-w8bdc"]
Sep 18 17:28:22.898803 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:22.898777    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8bd53078-c5d887794-w8bdc"
Sep 18 17:28:22.907704 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:22.907681    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8bd53078\""
Sep 18 17:28:22.907818 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:22.907681    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8bd53078-dockercfg-5gtv4\""
Sep 18 17:28:22.917201 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:22.917176    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8bd53078-7c9597dc57-4cmxh"]
Sep 18 17:28:22.920855 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:22.920833    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8bd53078-c5d887794-w8bdc"]
Sep 18 17:28:22.921018 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:22.920944    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8bd53078-7c9597dc57-4cmxh"
Sep 18 17:28:22.944538 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:22.944514    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ace3bc-6017-4e70-962e-9f4cd30fc2bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8bd53078-c5d887794-w8bdc\" (UID: \"38ace3bc-6017-4e70-962e-9f4cd30fc2bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8bd53078-c5d887794-w8bdc"
Sep 18 17:28:22.944652 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:22.944574    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx5vt\" (UniqueName: \"kubernetes.io/projected/38ace3bc-6017-4e70-962e-9f4cd30fc2bb-kube-api-access-qx5vt\") pod \"payload-pre-processing-e2e-ait-8bd53078-c5d887794-w8bdc\" (UID: \"38ace3bc-6017-4e70-962e-9f4cd30fc2bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8bd53078-c5d887794-w8bdc"
Sep 18 17:28:22.949772 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:22.949748    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8bd53078-7c9597dc57-4cmxh"]
Sep 18 17:28:22.999956 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:22.999872    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt"
Sep 18 17:28:23.045750 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:23.045358    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b94ea912-c166-4fa4-9381-c4c4e0428f51-plugins-config-volume\") pod \"payload-processing-e2e-ait-8bd53078-7c9597dc57-4cmxh\" (UID: \"b94ea912-c166-4fa4-9381-c4c4e0428f51\") " pod="openshift-ingress/payload-processing-e2e-ait-8bd53078-7c9597dc57-4cmxh"
Sep 18 17:28:23.045750 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:23.045413    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx5vt\" (UniqueName: \"kubernetes.io/projected/38ace3bc-6017-4e70-962e-9f4cd30fc2bb-kube-api-access-qx5vt\") pod \"payload-pre-processing-e2e-ait-8bd53078-c5d887794-w8bdc\" (UID: \"38ace3bc-6017-4e70-962e-9f4cd30fc2bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8bd53078-c5d887794-w8bdc"
Sep 18 17:28:23.045750 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:23.045527    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sw25\" (UniqueName: \"kubernetes.io/projected/b94ea912-c166-4fa4-9381-c4c4e0428f51-kube-api-access-5sw25\") pod \"payload-processing-e2e-ait-8bd53078-7c9597dc57-4cmxh\" (UID: \"b94ea912-c166-4fa4-9381-c4c4e0428f51\") " pod="openshift-ingress/payload-processing-e2e-ait-8bd53078-7c9597dc57-4cmxh"
Sep 18 17:28:23.045750 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:23.045583    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ace3bc-6017-4e70-962e-9f4cd30fc2bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8bd53078-c5d887794-w8bdc\" (UID: \"38ace3bc-6017-4e70-962e-9f4cd30fc2bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8bd53078-c5d887794-w8bdc"
Sep 18 17:28:23.046551 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:23.046525    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ace3bc-6017-4e70-962e-9f4cd30fc2bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8bd53078-c5d887794-w8bdc\" (UID: \"38ace3bc-6017-4e70-962e-9f4cd30fc2bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8bd53078-c5d887794-w8bdc"
Sep 18 17:28:23.069902 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:23.069853    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx5vt\" (UniqueName: \"kubernetes.io/projected/38ace3bc-6017-4e70-962e-9f4cd30fc2bb-kube-api-access-qx5vt\") pod \"payload-pre-processing-e2e-ait-8bd53078-c5d887794-w8bdc\" (UID: \"38ace3bc-6017-4e70-962e-9f4cd30fc2bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8bd53078-c5d887794-w8bdc"
Sep 18 17:28:23.146583 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:23.146546    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b94ea912-c166-4fa4-9381-c4c4e0428f51-plugins-config-volume\") pod \"payload-processing-e2e-ait-8bd53078-7c9597dc57-4cmxh\" (UID: \"b94ea912-c166-4fa4-9381-c4c4e0428f51\") " pod="openshift-ingress/payload-processing-e2e-ait-8bd53078-7c9597dc57-4cmxh"
Sep 18 17:28:23.147070 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:23.146639    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sw25\" (UniqueName: \"kubernetes.io/projected/b94ea912-c166-4fa4-9381-c4c4e0428f51-kube-api-access-5sw25\") pod \"payload-processing-e2e-ait-8bd53078-7c9597dc57-4cmxh\" (UID: \"b94ea912-c166-4fa4-9381-c4c4e0428f51\") " pod="openshift-ingress/payload-processing-e2e-ait-8bd53078-7c9597dc57-4cmxh"
Sep 18 17:28:23.147255 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:23.147235    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b94ea912-c166-4fa4-9381-c4c4e0428f51-plugins-config-volume\") pod \"payload-processing-e2e-ait-8bd53078-7c9597dc57-4cmxh\" (UID: \"b94ea912-c166-4fa4-9381-c4c4e0428f51\") " pod="openshift-ingress/payload-processing-e2e-ait-8bd53078-7c9597dc57-4cmxh"
Sep 18 17:28:23.156896 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:23.156851    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sw25\" (UniqueName: \"kubernetes.io/projected/b94ea912-c166-4fa4-9381-c4c4e0428f51-kube-api-access-5sw25\") pod \"payload-processing-e2e-ait-8bd53078-7c9597dc57-4cmxh\" (UID: \"b94ea912-c166-4fa4-9381-c4c4e0428f51\") " pod="openshift-ingress/payload-processing-e2e-ait-8bd53078-7c9597dc57-4cmxh"
Sep 18 17:28:23.209767 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:23.209727    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8bd53078-c5d887794-w8bdc"
Sep 18 17:28:23.231331 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:23.231298    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8bd53078-7c9597dc57-4cmxh"
Sep 18 17:28:23.300503 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:23.300469    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt"]
Sep 18 17:28:23.303338 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:28:23.303297    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d35705_eefa_4ad4_be40_7bf3f4fcd392.slice/crio-022ed332ef11f1d1b56fe3b9c1727df1b19e68fcfb2dbcc5dea8485b480bebec WatchSource:0}: Error finding container 022ed332ef11f1d1b56fe3b9c1727df1b19e68fcfb2dbcc5dea8485b480bebec: Status 404 returned error can't find the container with id 022ed332ef11f1d1b56fe3b9c1727df1b19e68fcfb2dbcc5dea8485b480bebec
Sep 18 17:28:23.444023 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:23.443991    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt" event={"ID":"d2d35705-eefa-4ad4-be40-7bf3f4fcd392","Type":"ContainerStarted","Data":"022ed332ef11f1d1b56fe3b9c1727df1b19e68fcfb2dbcc5dea8485b480bebec"}
Sep 18 17:28:23.515235 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:23.515156    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8bd53078-c5d887794-w8bdc"]
Sep 18 17:28:23.518879 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:28:23.518834    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ace3bc_6017_4e70_962e_9f4cd30fc2bb.slice/crio-966c45084209384d69e4b9b8fdb1dcf8f9d8f4dbdcb7b25142d0dd0c2f37c69e WatchSource:0}: Error finding container 966c45084209384d69e4b9b8fdb1dcf8f9d8f4dbdcb7b25142d0dd0c2f37c69e: Status 404 returned error can't find the container with id 966c45084209384d69e4b9b8fdb1dcf8f9d8f4dbdcb7b25142d0dd0c2f37c69e
Sep 18 17:28:23.543905 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:23.543838    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8bd53078-7c9597dc57-4cmxh"]
Sep 18 17:28:23.563893 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:28:23.563842    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb94ea912_c166_4fa4_9381_c4c4e0428f51.slice/crio-a9a5a0ba09855f0cb8ff6b066106bbb0b46e06adcd909dfdef097f234786f092 WatchSource:0}: Error finding container a9a5a0ba09855f0cb8ff6b066106bbb0b46e06adcd909dfdef097f234786f092: Status 404 returned error can't find the container with id a9a5a0ba09855f0cb8ff6b066106bbb0b46e06adcd909dfdef097f234786f092
Sep 18 17:28:24.451649 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:24.451611    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8bd53078-7c9597dc57-4cmxh" event={"ID":"b94ea912-c166-4fa4-9381-c4c4e0428f51","Type":"ContainerStarted","Data":"2d302386d9741cf70565c459be4b1e723c66cbdb41f18ea2781d4255e53dee0a"}
Sep 18 17:28:24.452149 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:24.451676    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8bd53078-7c9597dc57-4cmxh" event={"ID":"b94ea912-c166-4fa4-9381-c4c4e0428f51","Type":"ContainerStarted","Data":"a9a5a0ba09855f0cb8ff6b066106bbb0b46e06adcd909dfdef097f234786f092"}
Sep 18 17:28:24.452149 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:24.451706    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8bd53078-7c9597dc57-4cmxh"
Sep 18 17:28:24.453366 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:24.453329    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt" event={"ID":"d2d35705-eefa-4ad4-be40-7bf3f4fcd392","Type":"ContainerStarted","Data":"938d7db3c009ae0b55625450c04f92528f71b72715347e954e971fee198109e1"}
Sep 18 17:28:24.453492 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:24.453394    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt"
Sep 18 17:28:24.454889 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:24.454848    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8bd53078-c5d887794-w8bdc" event={"ID":"38ace3bc-6017-4e70-962e-9f4cd30fc2bb","Type":"ContainerStarted","Data":"12fccf712fb4d7167f3df0f214bedae8937d6b372d7cb4b134b9a68d120d0bb1"}
Sep 18 17:28:24.455070 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:24.454911    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8bd53078-c5d887794-w8bdc" event={"ID":"38ace3bc-6017-4e70-962e-9f4cd30fc2bb","Type":"ContainerStarted","Data":"966c45084209384d69e4b9b8fdb1dcf8f9d8f4dbdcb7b25142d0dd0c2f37c69e"}
Sep 18 17:28:24.455070 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:24.454938    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8bd53078-c5d887794-w8bdc"
Sep 18 17:28:24.470982 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:24.470932    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8bd53078-7c9597dc57-4cmxh" podStartSLOduration=2.470915104 podStartE2EDuration="2.470915104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:28:24.469390536 +0000 UTC m=+817.163318081" watchObservedRunningTime="2026-09-18 17:28:24.470915104 +0000 UTC m=+817.164842634"
Sep 18 17:28:24.488593 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:24.488548    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt" podStartSLOduration=2.136562351 podStartE2EDuration="2.488535262s" totalImagesPullingTime="351.972911ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:28:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:28:24.486797211 +0000 UTC m=+817.180724762" watchObservedRunningTime="2026-09-18 17:28:24.488535262 +0000 UTC m=+817.182462789"
Sep 18 17:28:24.503994 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:24.503950    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8bd53078-c5d887794-w8bdc" podStartSLOduration=2.503935484 podStartE2EDuration="2.503935484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:28:24.502750473 +0000 UTC m=+817.196678022" watchObservedRunningTime="2026-09-18 17:28:24.503935484 +0000 UTC m=+817.197863012"
Sep 18 17:28:29.263448 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:29.263400    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd"
Sep 18 17:28:29.263887 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:29.263476    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd"
Sep 18 17:28:29.276107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:29.276080    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd"
Sep 18 17:28:29.495222 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:29.495190    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd"
Sep 18 17:28:30.464538 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:30.464504    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt"
Sep 18 17:28:35.460993 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:35.460961    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8bd53078-c5d887794-w8bdc"
Sep 18 17:28:35.461416 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:35.461035    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8bd53078-7c9597dc57-4cmxh"
Sep 18 17:28:40.407255 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:40.407219    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj"]
Sep 18 17:28:40.411577 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:40.411557    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj"
Sep 18 17:28:40.415243 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:40.415211    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-0a2e13\"/\"openshift-service-ca.crt\""
Sep 18 17:28:40.415243 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:40.415236    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0a2e13\"/\"simulator-w0-0a2e13-kserve-self-signed-certs\""
Sep 18 17:28:40.415430 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:40.415281    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0a2e13\"/\"default-dockercfg-c4wdl\""
Sep 18 17:28:40.415430 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:40.415296    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-0a2e13\"/\"kube-root-ca.crt\""
Sep 18 17:28:40.421974 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:40.421952    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj"]
Sep 18 17:28:40.507291 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:40.507256    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/399fd9db-af7b-4e20-9c6b-0d34c703dead-model-cache\") pod \"simulator-w0-0a2e13-kserve-6d85849f99-446rj\" (UID: \"399fd9db-af7b-4e20-9c6b-0d34c703dead\") " pod="e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj"
Sep 18 17:28:40.507291 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:40.507291    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/399fd9db-af7b-4e20-9c6b-0d34c703dead-dshm\") pod \"simulator-w0-0a2e13-kserve-6d85849f99-446rj\" (UID: \"399fd9db-af7b-4e20-9c6b-0d34c703dead\") " pod="e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj"
Sep 18 17:28:40.507462 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:40.507330    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/399fd9db-af7b-4e20-9c6b-0d34c703dead-home\") pod \"simulator-w0-0a2e13-kserve-6d85849f99-446rj\" (UID: \"399fd9db-af7b-4e20-9c6b-0d34c703dead\") " pod="e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj"
Sep 18 17:28:40.507462 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:40.507371    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9scct\" (UniqueName: \"kubernetes.io/projected/399fd9db-af7b-4e20-9c6b-0d34c703dead-kube-api-access-9scct\") pod \"simulator-w0-0a2e13-kserve-6d85849f99-446rj\" (UID: \"399fd9db-af7b-4e20-9c6b-0d34c703dead\") " pod="e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj"
Sep 18 17:28:40.507462 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:40.507449    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/399fd9db-af7b-4e20-9c6b-0d34c703dead-tmp-dir\") pod \"simulator-w0-0a2e13-kserve-6d85849f99-446rj\" (UID: \"399fd9db-af7b-4e20-9c6b-0d34c703dead\") " pod="e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj"
Sep 18 17:28:40.507576 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:40.507522    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/399fd9db-af7b-4e20-9c6b-0d34c703dead-tls-certs\") pod \"simulator-w0-0a2e13-kserve-6d85849f99-446rj\" (UID: \"399fd9db-af7b-4e20-9c6b-0d34c703dead\") " pod="e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj"
Sep 18 17:28:40.608681 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:40.608640    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/399fd9db-af7b-4e20-9c6b-0d34c703dead-model-cache\") pod \"simulator-w0-0a2e13-kserve-6d85849f99-446rj\" (UID: \"399fd9db-af7b-4e20-9c6b-0d34c703dead\") " pod="e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj"
Sep 18 17:28:40.608681 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:40.608680    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/399fd9db-af7b-4e20-9c6b-0d34c703dead-dshm\") pod \"simulator-w0-0a2e13-kserve-6d85849f99-446rj\" (UID: \"399fd9db-af7b-4e20-9c6b-0d34c703dead\") " pod="e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj"
Sep 18 17:28:40.608979 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:40.608718    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/399fd9db-af7b-4e20-9c6b-0d34c703dead-home\") pod \"simulator-w0-0a2e13-kserve-6d85849f99-446rj\" (UID: \"399fd9db-af7b-4e20-9c6b-0d34c703dead\") " pod="e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj"
Sep 18 17:28:40.608979 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:40.608736    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9scct\" (UniqueName: \"kubernetes.io/projected/399fd9db-af7b-4e20-9c6b-0d34c703dead-kube-api-access-9scct\") pod \"simulator-w0-0a2e13-kserve-6d85849f99-446rj\" (UID: \"399fd9db-af7b-4e20-9c6b-0d34c703dead\") " pod="e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj"
Sep 18 17:28:40.608979 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:40.608757    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/399fd9db-af7b-4e20-9c6b-0d34c703dead-tmp-dir\") pod \"simulator-w0-0a2e13-kserve-6d85849f99-446rj\" (UID: \"399fd9db-af7b-4e20-9c6b-0d34c703dead\") " pod="e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj"
Sep 18 17:28:40.608979 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:40.608803    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/399fd9db-af7b-4e20-9c6b-0d34c703dead-tls-certs\") pod \"simulator-w0-0a2e13-kserve-6d85849f99-446rj\" (UID: \"399fd9db-af7b-4e20-9c6b-0d34c703dead\") " pod="e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj"
Sep 18 17:28:40.609253 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:40.609228    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/399fd9db-af7b-4e20-9c6b-0d34c703dead-model-cache\") pod \"simulator-w0-0a2e13-kserve-6d85849f99-446rj\" (UID: \"399fd9db-af7b-4e20-9c6b-0d34c703dead\") " pod="e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj"
Sep 18 17:28:40.609310 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:40.609251    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/399fd9db-af7b-4e20-9c6b-0d34c703dead-home\") pod \"simulator-w0-0a2e13-kserve-6d85849f99-446rj\" (UID: \"399fd9db-af7b-4e20-9c6b-0d34c703dead\") " pod="e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj"
Sep 18 17:28:40.609310 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:40.609290    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/399fd9db-af7b-4e20-9c6b-0d34c703dead-tmp-dir\") pod \"simulator-w0-0a2e13-kserve-6d85849f99-446rj\" (UID: \"399fd9db-af7b-4e20-9c6b-0d34c703dead\") " pod="e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj"
Sep 18 17:28:40.611131 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:40.611107    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/399fd9db-af7b-4e20-9c6b-0d34c703dead-dshm\") pod \"simulator-w0-0a2e13-kserve-6d85849f99-446rj\" (UID: \"399fd9db-af7b-4e20-9c6b-0d34c703dead\") " pod="e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj"
Sep 18 17:28:40.611282 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:40.611264    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/399fd9db-af7b-4e20-9c6b-0d34c703dead-tls-certs\") pod \"simulator-w0-0a2e13-kserve-6d85849f99-446rj\" (UID: \"399fd9db-af7b-4e20-9c6b-0d34c703dead\") " pod="e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj"
Sep 18 17:28:40.618314 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:40.618294    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9scct\" (UniqueName: \"kubernetes.io/projected/399fd9db-af7b-4e20-9c6b-0d34c703dead-kube-api-access-9scct\") pod \"simulator-w0-0a2e13-kserve-6d85849f99-446rj\" (UID: \"399fd9db-af7b-4e20-9c6b-0d34c703dead\") " pod="e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj"
Sep 18 17:28:40.721194 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:40.721104    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj"
Sep 18 17:28:40.994668 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:40.994644    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj"]
Sep 18 17:28:40.996857 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:28:40.996827    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod399fd9db_af7b_4e20_9c6b_0d34c703dead.slice/crio-7c75f7552b0daef57a4fa0d35ff4f1556880cbb4d2deb3e427b44d875fec8990 WatchSource:0}: Error finding container 7c75f7552b0daef57a4fa0d35ff4f1556880cbb4d2deb3e427b44d875fec8990: Status 404 returned error can't find the container with id 7c75f7552b0daef57a4fa0d35ff4f1556880cbb4d2deb3e427b44d875fec8990
Sep 18 17:28:41.550953 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:41.550916    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj" event={"ID":"399fd9db-af7b-4e20-9c6b-0d34c703dead","Type":"ContainerStarted","Data":"0c6875848524d9efd329a1e7b6b6c12dd4d414f78c5734b2c88c749b8dab7c4f"}
Sep 18 17:28:41.551342 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:41.550961    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj" event={"ID":"399fd9db-af7b-4e20-9c6b-0d34c703dead","Type":"ContainerStarted","Data":"7c75f7552b0daef57a4fa0d35ff4f1556880cbb4d2deb3e427b44d875fec8990"}
Sep 18 17:28:45.010055 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:45.009990    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj" podStartSLOduration=5.009966986 podStartE2EDuration="5.009966986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:28:41.596926889 +0000 UTC m=+834.290854436" watchObservedRunningTime="2026-09-18 17:28:45.009966986 +0000 UTC m=+837.703894516"
Sep 18 17:28:45.013549 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:45.013523    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff"]
Sep 18 17:28:45.017145 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:45.017123    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff"
Sep 18 17:28:45.020591 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:45.020570    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-535fd3\"/\"autores-model-535fd3-kserve-self-signed-certs\""
Sep 18 17:28:45.021433 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:45.021414    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-535fd3\"/\"openshift-service-ca.crt\""
Sep 18 17:28:45.021531 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:45.021455    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-535fd3\"/\"kube-root-ca.crt\""
Sep 18 17:28:45.021750 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:45.021733    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-535fd3\"/\"default-dockercfg-7dj86\""
Sep 18 17:28:45.032422 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:45.032401    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff"]
Sep 18 17:28:45.152574 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:45.152540    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5948f\" (UniqueName: \"kubernetes.io/projected/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-kube-api-access-5948f\") pod \"autores-model-535fd3-kserve-76b546457f-27zff\" (UID: \"5e72e7b6-5c6f-4754-9ade-f027ccbc98ef\") " pod="ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff"
Sep 18 17:28:45.152744 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:45.152590    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-model-cache\") pod \"autores-model-535fd3-kserve-76b546457f-27zff\" (UID: \"5e72e7b6-5c6f-4754-9ade-f027ccbc98ef\") " pod="ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff"
Sep 18 17:28:45.152744 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:45.152619    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-tls-certs\") pod \"autores-model-535fd3-kserve-76b546457f-27zff\" (UID: \"5e72e7b6-5c6f-4754-9ade-f027ccbc98ef\") " pod="ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff"
Sep 18 17:28:45.152744 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:45.152653    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-tmp-dir\") pod \"autores-model-535fd3-kserve-76b546457f-27zff\" (UID: \"5e72e7b6-5c6f-4754-9ade-f027ccbc98ef\") " pod="ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff"
Sep 18 17:28:45.152744 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:45.152710    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-home\") pod \"autores-model-535fd3-kserve-76b546457f-27zff\" (UID: \"5e72e7b6-5c6f-4754-9ade-f027ccbc98ef\") " pod="ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff"
Sep 18 17:28:45.152744 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:45.152735    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-dshm\") pod \"autores-model-535fd3-kserve-76b546457f-27zff\" (UID: \"5e72e7b6-5c6f-4754-9ade-f027ccbc98ef\") " pod="ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff"
Sep 18 17:28:45.253940 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:45.253904    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5948f\" (UniqueName: \"kubernetes.io/projected/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-kube-api-access-5948f\") pod \"autores-model-535fd3-kserve-76b546457f-27zff\" (UID: \"5e72e7b6-5c6f-4754-9ade-f027ccbc98ef\") " pod="ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff"
Sep 18 17:28:45.254111 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:45.253960    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-model-cache\") pod \"autores-model-535fd3-kserve-76b546457f-27zff\" (UID: \"5e72e7b6-5c6f-4754-9ade-f027ccbc98ef\") " pod="ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff"
Sep 18 17:28:45.254111 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:45.253982    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-tls-certs\") pod \"autores-model-535fd3-kserve-76b546457f-27zff\" (UID: \"5e72e7b6-5c6f-4754-9ade-f027ccbc98ef\") " pod="ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff"
Sep 18 17:28:45.254111 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:45.254016    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-tmp-dir\") pod \"autores-model-535fd3-kserve-76b546457f-27zff\" (UID: \"5e72e7b6-5c6f-4754-9ade-f027ccbc98ef\") " pod="ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff"
Sep 18 17:28:45.254111 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:45.254052    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-home\") pod \"autores-model-535fd3-kserve-76b546457f-27zff\" (UID: \"5e72e7b6-5c6f-4754-9ade-f027ccbc98ef\") " pod="ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff"
Sep 18 17:28:45.254309 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:45.254194    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-dshm\") pod \"autores-model-535fd3-kserve-76b546457f-27zff\" (UID: \"5e72e7b6-5c6f-4754-9ade-f027ccbc98ef\") " pod="ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff"
Sep 18 17:28:45.254381 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:45.254361    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-model-cache\") pod \"autores-model-535fd3-kserve-76b546457f-27zff\" (UID: \"5e72e7b6-5c6f-4754-9ade-f027ccbc98ef\") " pod="ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff"
Sep 18 17:28:45.254439 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:45.254420    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-home\") pod \"autores-model-535fd3-kserve-76b546457f-27zff\" (UID: \"5e72e7b6-5c6f-4754-9ade-f027ccbc98ef\") " pod="ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff"
Sep 18 17:28:45.254477 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:45.254446    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-tmp-dir\") pod \"autores-model-535fd3-kserve-76b546457f-27zff\" (UID: \"5e72e7b6-5c6f-4754-9ade-f027ccbc98ef\") " pod="ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff"
Sep 18 17:28:45.256286 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:45.256255    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-dshm\") pod \"autores-model-535fd3-kserve-76b546457f-27zff\" (UID: \"5e72e7b6-5c6f-4754-9ade-f027ccbc98ef\") " pod="ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff"
Sep 18 17:28:45.256441 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:45.256423    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-tls-certs\") pod \"autores-model-535fd3-kserve-76b546457f-27zff\" (UID: \"5e72e7b6-5c6f-4754-9ade-f027ccbc98ef\") " pod="ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff"
Sep 18 17:28:45.277979 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:45.277919    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5948f\" (UniqueName: \"kubernetes.io/projected/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-kube-api-access-5948f\") pod \"autores-model-535fd3-kserve-76b546457f-27zff\" (UID: \"5e72e7b6-5c6f-4754-9ade-f027ccbc98ef\") " pod="ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff"
Sep 18 17:28:45.327840 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:45.327805    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff"
Sep 18 17:28:45.602165 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:45.601404    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff"]
Sep 18 17:28:45.604336 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:28:45.604301    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e72e7b6_5c6f_4754_9ade_f027ccbc98ef.slice/crio-ab28f18a2c720b37d6066058ccab6dd587db142ffd6a0dd3d4cd7dfd26bdd903 WatchSource:0}: Error finding container ab28f18a2c720b37d6066058ccab6dd587db142ffd6a0dd3d4cd7dfd26bdd903: Status 404 returned error can't find the container with id ab28f18a2c720b37d6066058ccab6dd587db142ffd6a0dd3d4cd7dfd26bdd903
Sep 18 17:28:46.578123 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:46.578088    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff" event={"ID":"5e72e7b6-5c6f-4754-9ade-f027ccbc98ef","Type":"ContainerStarted","Data":"f3bed3db5778ff5bd54fc137af381e5e572fb01d475e213ee78ae13947d613de"}
Sep 18 17:28:46.578123 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:46.578137    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff" event={"ID":"5e72e7b6-5c6f-4754-9ade-f027ccbc98ef","Type":"ContainerStarted","Data":"ab28f18a2c720b37d6066058ccab6dd587db142ffd6a0dd3d4cd7dfd26bdd903"}
Sep 18 17:28:46.597348 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:46.597295    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff" podStartSLOduration=2.597281481 podStartE2EDuration="2.597281481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:28:46.594488829 +0000 UTC m=+839.288416374" watchObservedRunningTime="2026-09-18 17:28:46.597281481 +0000 UTC m=+839.291209066"
Sep 18 17:28:50.721967 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:50.721928    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj"
Sep 18 17:28:50.722477 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:50.722019    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj"
Sep 18 17:28:50.734748 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:50.734712    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj"
Sep 18 17:28:51.616710 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:51.616681    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj"
Sep 18 17:28:55.327956 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:55.327914    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff"
Sep 18 17:28:55.328386 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:55.327988    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff"
Sep 18 17:28:55.340663 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:55.340637    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff"
Sep 18 17:28:55.636596 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:55.636516    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff"
Sep 18 17:28:57.711100 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:57.711069    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54"]
Sep 18 17:28:57.715841 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:57.715823    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54"
Sep 18 17:28:57.719684 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:57.719660    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-636b80\"/\"kube-root-ca.crt\""
Sep 18 17:28:57.719821 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:57.719661    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-636b80\"/\"openshift-service-ca.crt\""
Sep 18 17:28:57.719821 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:57.719664    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-636b80\"/\"test-model-636b80-kserve-self-signed-certs\""
Sep 18 17:28:57.720122 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:57.720104    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-636b80\"/\"default-dockercfg-rl5zq\""
Sep 18 17:28:57.726831 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:57.726806    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54"]
Sep 18 17:28:57.768940 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:57.768908    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7lnr\" (UniqueName: \"kubernetes.io/projected/246181d7-28f4-453c-bd85-01c6d8fb6e51-kube-api-access-h7lnr\") pod \"test-model-636b80-kserve-68c5cc9b47-ngm54\" (UID: \"246181d7-28f4-453c-bd85-01c6d8fb6e51\") " pod="ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54"
Sep 18 17:28:57.768940 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:57.768940    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/246181d7-28f4-453c-bd85-01c6d8fb6e51-model-cache\") pod \"test-model-636b80-kserve-68c5cc9b47-ngm54\" (UID: \"246181d7-28f4-453c-bd85-01c6d8fb6e51\") " pod="ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54"
Sep 18 17:28:57.769161 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:57.769026    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/246181d7-28f4-453c-bd85-01c6d8fb6e51-home\") pod \"test-model-636b80-kserve-68c5cc9b47-ngm54\" (UID: \"246181d7-28f4-453c-bd85-01c6d8fb6e51\") " pod="ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54"
Sep 18 17:28:57.769161 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:57.769084    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/246181d7-28f4-453c-bd85-01c6d8fb6e51-tls-certs\") pod \"test-model-636b80-kserve-68c5cc9b47-ngm54\" (UID: \"246181d7-28f4-453c-bd85-01c6d8fb6e51\") " pod="ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54"
Sep 18 17:28:57.769161 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:57.769113    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/246181d7-28f4-453c-bd85-01c6d8fb6e51-tmp-dir\") pod \"test-model-636b80-kserve-68c5cc9b47-ngm54\" (UID: \"246181d7-28f4-453c-bd85-01c6d8fb6e51\") " pod="ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54"
Sep 18 17:28:57.769161 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:57.769150    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/246181d7-28f4-453c-bd85-01c6d8fb6e51-dshm\") pod \"test-model-636b80-kserve-68c5cc9b47-ngm54\" (UID: \"246181d7-28f4-453c-bd85-01c6d8fb6e51\") " pod="ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54"
Sep 18 17:28:57.870320 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:57.870284    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/246181d7-28f4-453c-bd85-01c6d8fb6e51-home\") pod \"test-model-636b80-kserve-68c5cc9b47-ngm54\" (UID: \"246181d7-28f4-453c-bd85-01c6d8fb6e51\") " pod="ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54"
Sep 18 17:28:57.870486 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:57.870327    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/246181d7-28f4-453c-bd85-01c6d8fb6e51-tls-certs\") pod \"test-model-636b80-kserve-68c5cc9b47-ngm54\" (UID: \"246181d7-28f4-453c-bd85-01c6d8fb6e51\") " pod="ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54"
Sep 18 17:28:57.870486 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:57.870351    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/246181d7-28f4-453c-bd85-01c6d8fb6e51-tmp-dir\") pod \"test-model-636b80-kserve-68c5cc9b47-ngm54\" (UID: \"246181d7-28f4-453c-bd85-01c6d8fb6e51\") " pod="ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54"
Sep 18 17:28:57.870486 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:57.870464    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/246181d7-28f4-453c-bd85-01c6d8fb6e51-dshm\") pod \"test-model-636b80-kserve-68c5cc9b47-ngm54\" (UID: \"246181d7-28f4-453c-bd85-01c6d8fb6e51\") " pod="ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54"
Sep 18 17:28:57.870596 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:57.870552    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7lnr\" (UniqueName: \"kubernetes.io/projected/246181d7-28f4-453c-bd85-01c6d8fb6e51-kube-api-access-h7lnr\") pod \"test-model-636b80-kserve-68c5cc9b47-ngm54\" (UID: \"246181d7-28f4-453c-bd85-01c6d8fb6e51\") " pod="ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54"
Sep 18 17:28:57.870596 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:57.870584    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/246181d7-28f4-453c-bd85-01c6d8fb6e51-model-cache\") pod \"test-model-636b80-kserve-68c5cc9b47-ngm54\" (UID: \"246181d7-28f4-453c-bd85-01c6d8fb6e51\") " pod="ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54"
Sep 18 17:28:57.870847 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:57.870816    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/246181d7-28f4-453c-bd85-01c6d8fb6e51-home\") pod \"test-model-636b80-kserve-68c5cc9b47-ngm54\" (UID: \"246181d7-28f4-453c-bd85-01c6d8fb6e51\") " pod="ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54"
Sep 18 17:28:57.870999 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:57.870892    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/246181d7-28f4-453c-bd85-01c6d8fb6e51-tmp-dir\") pod \"test-model-636b80-kserve-68c5cc9b47-ngm54\" (UID: \"246181d7-28f4-453c-bd85-01c6d8fb6e51\") " pod="ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54"
Sep 18 17:28:57.871065 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:57.871015    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/246181d7-28f4-453c-bd85-01c6d8fb6e51-model-cache\") pod \"test-model-636b80-kserve-68c5cc9b47-ngm54\" (UID: \"246181d7-28f4-453c-bd85-01c6d8fb6e51\") " pod="ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54"
Sep 18 17:28:57.872723 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:57.872700    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/246181d7-28f4-453c-bd85-01c6d8fb6e51-dshm\") pod \"test-model-636b80-kserve-68c5cc9b47-ngm54\" (UID: \"246181d7-28f4-453c-bd85-01c6d8fb6e51\") " pod="ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54"
Sep 18 17:28:57.872998 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:57.872982    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/246181d7-28f4-453c-bd85-01c6d8fb6e51-tls-certs\") pod \"test-model-636b80-kserve-68c5cc9b47-ngm54\" (UID: \"246181d7-28f4-453c-bd85-01c6d8fb6e51\") " pod="ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54"
Sep 18 17:28:57.884151 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:57.884126    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7lnr\" (UniqueName: \"kubernetes.io/projected/246181d7-28f4-453c-bd85-01c6d8fb6e51-kube-api-access-h7lnr\") pod \"test-model-636b80-kserve-68c5cc9b47-ngm54\" (UID: \"246181d7-28f4-453c-bd85-01c6d8fb6e51\") " pod="ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54"
Sep 18 17:28:58.025123 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:58.025085    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54"
Sep 18 17:28:58.299564 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:58.299536    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54"]
Sep 18 17:28:58.301539 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:28:58.301504    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod246181d7_28f4_453c_bd85_01c6d8fb6e51.slice/crio-ce9e5b096fd917a1726a02d92549a004a01cb188099a9a241a129e9e3beb411d WatchSource:0}: Error finding container ce9e5b096fd917a1726a02d92549a004a01cb188099a9a241a129e9e3beb411d: Status 404 returned error can't find the container with id ce9e5b096fd917a1726a02d92549a004a01cb188099a9a241a129e9e3beb411d
Sep 18 17:28:58.641959 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:58.641838    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54" event={"ID":"246181d7-28f4-453c-bd85-01c6d8fb6e51","Type":"ContainerStarted","Data":"f95cc50647b09dce31b894548b0ad79ccae7cd94c75666f1886ac0c598ca30a7"}
Sep 18 17:28:58.641959 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:58.641922    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54" event={"ID":"246181d7-28f4-453c-bd85-01c6d8fb6e51","Type":"ContainerStarted","Data":"ce9e5b096fd917a1726a02d92549a004a01cb188099a9a241a129e9e3beb411d"}
Sep 18 17:28:58.665885 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:28:58.664894    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54" podStartSLOduration=1.664857101 podStartE2EDuration="1.664857101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:28:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:28:58.661288332 +0000 UTC m=+851.355215887" watchObservedRunningTime="2026-09-18 17:28:58.664857101 +0000 UTC m=+851.358784630"
Sep 18 17:29:00.053745 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.053709    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt"]
Sep 18 17:29:00.054252 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.053987    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt" podUID="d2d35705-eefa-4ad4-be40-7bf3f4fcd392" containerName="maas-api" containerID="cri-o://938d7db3c009ae0b55625450c04f92528f71b72715347e954e971fee198109e1" gracePeriod=30
Sep 18 17:29:00.118357 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.118320    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8bd53078-7c9597dc57-4cmxh"]
Sep 18 17:29:00.118608 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.118579    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8bd53078-7c9597dc57-4cmxh" podUID="b94ea912-c166-4fa4-9381-c4c4e0428f51" containerName="payload-processing" containerID="cri-o://2d302386d9741cf70565c459be4b1e723c66cbdb41f18ea2781d4255e53dee0a" gracePeriod=30
Sep 18 17:29:00.125259 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.125228    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8bd53078-c5d887794-w8bdc"]
Sep 18 17:29:00.125465 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.125436    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8bd53078-c5d887794-w8bdc" podUID="38ace3bc-6017-4e70-962e-9f4cd30fc2bb" containerName="payload-pre-processing" containerID="cri-o://12fccf712fb4d7167f3df0f214bedae8937d6b372d7cb4b134b9a68d120d0bb1" gracePeriod=30
Sep 18 17:29:00.400389 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.400364    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt"
Sep 18 17:29:00.402459 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.402433    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8bd53078-7c9597dc57-4cmxh"
Sep 18 17:29:00.411309 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.411290    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8bd53078-c5d887794-w8bdc"
Sep 18 17:29:00.496783 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.496758    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2d35705-eefa-4ad4-be40-7bf3f4fcd392-maas-api-tls\") pod \"d2d35705-eefa-4ad4-be40-7bf3f4fcd392\" (UID: \"d2d35705-eefa-4ad4-be40-7bf3f4fcd392\") "
Sep 18 17:29:00.496968 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.496813    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2d35705-eefa-4ad4-be40-7bf3f4fcd392-openshift-service-ca\") pod \"d2d35705-eefa-4ad4-be40-7bf3f4fcd392\" (UID: \"d2d35705-eefa-4ad4-be40-7bf3f4fcd392\") "
Sep 18 17:29:00.496968 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.496836    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d35705-eefa-4ad4-be40-7bf3f4fcd392-metrics-certs\") pod \"d2d35705-eefa-4ad4-be40-7bf3f4fcd392\" (UID: \"d2d35705-eefa-4ad4-be40-7bf3f4fcd392\") "
Sep 18 17:29:00.496968 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.496881    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b94ea912-c166-4fa4-9381-c4c4e0428f51-plugins-config-volume\") pod \"b94ea912-c166-4fa4-9381-c4c4e0428f51\" (UID: \"b94ea912-c166-4fa4-9381-c4c4e0428f51\") "
Sep 18 17:29:00.496968 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.496909    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qx5vt\" (UniqueName: \"kubernetes.io/projected/38ace3bc-6017-4e70-962e-9f4cd30fc2bb-kube-api-access-qx5vt\") pod \"38ace3bc-6017-4e70-962e-9f4cd30fc2bb\" (UID: \"38ace3bc-6017-4e70-962e-9f4cd30fc2bb\") "
Sep 18 17:29:00.496968 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.496959    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txgkm\" (UniqueName: \"kubernetes.io/projected/d2d35705-eefa-4ad4-be40-7bf3f4fcd392-kube-api-access-txgkm\") pod \"d2d35705-eefa-4ad4-be40-7bf3f4fcd392\" (UID: \"d2d35705-eefa-4ad4-be40-7bf3f4fcd392\") "
Sep 18 17:29:00.497232 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.496986    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sw25\" (UniqueName: \"kubernetes.io/projected/b94ea912-c166-4fa4-9381-c4c4e0428f51-kube-api-access-5sw25\") pod \"b94ea912-c166-4fa4-9381-c4c4e0428f51\" (UID: \"b94ea912-c166-4fa4-9381-c4c4e0428f51\") "
Sep 18 17:29:00.497232 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.497021    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ace3bc-6017-4e70-962e-9f4cd30fc2bb-plugins-config-volume\") pod \"38ace3bc-6017-4e70-962e-9f4cd30fc2bb\" (UID: \"38ace3bc-6017-4e70-962e-9f4cd30fc2bb\") "
Sep 18 17:29:00.497353 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.497237    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2d35705-eefa-4ad4-be40-7bf3f4fcd392-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d2d35705-eefa-4ad4-be40-7bf3f4fcd392" (UID: "d2d35705-eefa-4ad4-be40-7bf3f4fcd392"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:29:00.497353 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.497271    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b94ea912-c166-4fa4-9381-c4c4e0428f51-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b94ea912-c166-4fa4-9381-c4c4e0428f51" (UID: "b94ea912-c166-4fa4-9381-c4c4e0428f51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:29:00.497580 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.497413    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2d35705-eefa-4ad4-be40-7bf3f4fcd392-openshift-service-ca\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:29:00.497580 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.497580    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b94ea912-c166-4fa4-9381-c4c4e0428f51-plugins-config-volume\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:29:00.497715 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.497600    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ace3bc-6017-4e70-962e-9f4cd30fc2bb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "38ace3bc-6017-4e70-962e-9f4cd30fc2bb" (UID: "38ace3bc-6017-4e70-962e-9f4cd30fc2bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:29:00.499115 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.499090    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ace3bc-6017-4e70-962e-9f4cd30fc2bb-kube-api-access-qx5vt" (OuterVolumeSpecName: "kube-api-access-qx5vt") pod "38ace3bc-6017-4e70-962e-9f4cd30fc2bb" (UID: "38ace3bc-6017-4e70-962e-9f4cd30fc2bb"). InnerVolumeSpecName "kube-api-access-qx5vt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:29:00.499115 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.499106    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2d35705-eefa-4ad4-be40-7bf3f4fcd392-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d2d35705-eefa-4ad4-be40-7bf3f4fcd392" (UID: "d2d35705-eefa-4ad4-be40-7bf3f4fcd392"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:29:00.499252 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.499154    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b94ea912-c166-4fa4-9381-c4c4e0428f51-kube-api-access-5sw25" (OuterVolumeSpecName: "kube-api-access-5sw25") pod "b94ea912-c166-4fa4-9381-c4c4e0428f51" (UID: "b94ea912-c166-4fa4-9381-c4c4e0428f51"). InnerVolumeSpecName "kube-api-access-5sw25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:29:00.499383 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.499361    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2d35705-eefa-4ad4-be40-7bf3f4fcd392-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d2d35705-eefa-4ad4-be40-7bf3f4fcd392" (UID: "d2d35705-eefa-4ad4-be40-7bf3f4fcd392"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:29:00.499469 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.499395    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2d35705-eefa-4ad4-be40-7bf3f4fcd392-kube-api-access-txgkm" (OuterVolumeSpecName: "kube-api-access-txgkm") pod "d2d35705-eefa-4ad4-be40-7bf3f4fcd392" (UID: "d2d35705-eefa-4ad4-be40-7bf3f4fcd392"). InnerVolumeSpecName "kube-api-access-txgkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:29:00.598305 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.598235    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d35705-eefa-4ad4-be40-7bf3f4fcd392-metrics-certs\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:29:00.598305 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.598260    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx5vt\" (UniqueName: \"kubernetes.io/projected/38ace3bc-6017-4e70-962e-9f4cd30fc2bb-kube-api-access-qx5vt\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:29:00.598305 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.598271    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txgkm\" (UniqueName: \"kubernetes.io/projected/d2d35705-eefa-4ad4-be40-7bf3f4fcd392-kube-api-access-txgkm\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:29:00.598305 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.598279    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sw25\" (UniqueName: \"kubernetes.io/projected/b94ea912-c166-4fa4-9381-c4c4e0428f51-kube-api-access-5sw25\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:29:00.598305 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.598287    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ace3bc-6017-4e70-962e-9f4cd30fc2bb-plugins-config-volume\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:29:00.598305 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.598296    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2d35705-eefa-4ad4-be40-7bf3f4fcd392-maas-api-tls\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:29:00.653041 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.653011    2569 generic.go:356] "Generic (PLEG): container finished" podID="d2d35705-eefa-4ad4-be40-7bf3f4fcd392" containerID="938d7db3c009ae0b55625450c04f92528f71b72715347e954e971fee198109e1" exitCode=0
Sep 18 17:29:00.653187 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.653064    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt"
Sep 18 17:29:00.653187 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.653095    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt" event={"ID":"d2d35705-eefa-4ad4-be40-7bf3f4fcd392","Type":"ContainerDied","Data":"938d7db3c009ae0b55625450c04f92528f71b72715347e954e971fee198109e1"}
Sep 18 17:29:00.653187 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.653148    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt" event={"ID":"d2d35705-eefa-4ad4-be40-7bf3f4fcd392","Type":"ContainerDied","Data":"022ed332ef11f1d1b56fe3b9c1727df1b19e68fcfb2dbcc5dea8485b480bebec"}
Sep 18 17:29:00.653187 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.653176    2569 scope.go:117] "RemoveContainer" containerID="938d7db3c009ae0b55625450c04f92528f71b72715347e954e971fee198109e1"
Sep 18 17:29:00.654519 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.654490    2569 generic.go:356] "Generic (PLEG): container finished" podID="38ace3bc-6017-4e70-962e-9f4cd30fc2bb" containerID="12fccf712fb4d7167f3df0f214bedae8937d6b372d7cb4b134b9a68d120d0bb1" exitCode=0
Sep 18 17:29:00.654612 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.654528    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8bd53078-c5d887794-w8bdc"
Sep 18 17:29:00.654612 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.654571    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8bd53078-c5d887794-w8bdc" event={"ID":"38ace3bc-6017-4e70-962e-9f4cd30fc2bb","Type":"ContainerDied","Data":"12fccf712fb4d7167f3df0f214bedae8937d6b372d7cb4b134b9a68d120d0bb1"}
Sep 18 17:29:00.654741 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.654622    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8bd53078-c5d887794-w8bdc" event={"ID":"38ace3bc-6017-4e70-962e-9f4cd30fc2bb","Type":"ContainerDied","Data":"966c45084209384d69e4b9b8fdb1dcf8f9d8f4dbdcb7b25142d0dd0c2f37c69e"}
Sep 18 17:29:00.656091 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.656019    2569 generic.go:356] "Generic (PLEG): container finished" podID="b94ea912-c166-4fa4-9381-c4c4e0428f51" containerID="2d302386d9741cf70565c459be4b1e723c66cbdb41f18ea2781d4255e53dee0a" exitCode=0
Sep 18 17:29:00.656200 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.656049    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8bd53078-7c9597dc57-4cmxh" event={"ID":"b94ea912-c166-4fa4-9381-c4c4e0428f51","Type":"ContainerDied","Data":"2d302386d9741cf70565c459be4b1e723c66cbdb41f18ea2781d4255e53dee0a"}
Sep 18 17:29:00.656200 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.656169    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8bd53078-7c9597dc57-4cmxh" event={"ID":"b94ea912-c166-4fa4-9381-c4c4e0428f51","Type":"ContainerDied","Data":"a9a5a0ba09855f0cb8ff6b066106bbb0b46e06adcd909dfdef097f234786f092"}
Sep 18 17:29:00.656200 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.656062    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8bd53078-7c9597dc57-4cmxh"
Sep 18 17:29:00.664097 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.664079    2569 scope.go:117] "RemoveContainer" containerID="938d7db3c009ae0b55625450c04f92528f71b72715347e954e971fee198109e1"
Sep 18 17:29:00.664465 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:29:00.664417    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"938d7db3c009ae0b55625450c04f92528f71b72715347e954e971fee198109e1\": container with ID starting with 938d7db3c009ae0b55625450c04f92528f71b72715347e954e971fee198109e1 not found: ID does not exist" containerID="938d7db3c009ae0b55625450c04f92528f71b72715347e954e971fee198109e1"
Sep 18 17:29:00.664580 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.664474    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"938d7db3c009ae0b55625450c04f92528f71b72715347e954e971fee198109e1"} err="failed to get container status \"938d7db3c009ae0b55625450c04f92528f71b72715347e954e971fee198109e1\": rpc error: code = NotFound desc = could not find container \"938d7db3c009ae0b55625450c04f92528f71b72715347e954e971fee198109e1\": container with ID starting with 938d7db3c009ae0b55625450c04f92528f71b72715347e954e971fee198109e1 not found: ID does not exist"
Sep 18 17:29:00.664580 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.664497    2569 scope.go:117] "RemoveContainer" containerID="12fccf712fb4d7167f3df0f214bedae8937d6b372d7cb4b134b9a68d120d0bb1"
Sep 18 17:29:00.673067 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.673051    2569 scope.go:117] "RemoveContainer" containerID="12fccf712fb4d7167f3df0f214bedae8937d6b372d7cb4b134b9a68d120d0bb1"
Sep 18 17:29:00.673329 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:29:00.673310    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12fccf712fb4d7167f3df0f214bedae8937d6b372d7cb4b134b9a68d120d0bb1\": container with ID starting with 12fccf712fb4d7167f3df0f214bedae8937d6b372d7cb4b134b9a68d120d0bb1 not found: ID does not exist" containerID="12fccf712fb4d7167f3df0f214bedae8937d6b372d7cb4b134b9a68d120d0bb1"
Sep 18 17:29:00.673381 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.673335    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12fccf712fb4d7167f3df0f214bedae8937d6b372d7cb4b134b9a68d120d0bb1"} err="failed to get container status \"12fccf712fb4d7167f3df0f214bedae8937d6b372d7cb4b134b9a68d120d0bb1\": rpc error: code = NotFound desc = could not find container \"12fccf712fb4d7167f3df0f214bedae8937d6b372d7cb4b134b9a68d120d0bb1\": container with ID starting with 12fccf712fb4d7167f3df0f214bedae8937d6b372d7cb4b134b9a68d120d0bb1 not found: ID does not exist"
Sep 18 17:29:00.673381 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.673351    2569 scope.go:117] "RemoveContainer" containerID="2d302386d9741cf70565c459be4b1e723c66cbdb41f18ea2781d4255e53dee0a"
Sep 18 17:29:00.680820 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.680803    2569 scope.go:117] "RemoveContainer" containerID="2d302386d9741cf70565c459be4b1e723c66cbdb41f18ea2781d4255e53dee0a"
Sep 18 17:29:00.681095 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:29:00.681078    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d302386d9741cf70565c459be4b1e723c66cbdb41f18ea2781d4255e53dee0a\": container with ID starting with 2d302386d9741cf70565c459be4b1e723c66cbdb41f18ea2781d4255e53dee0a not found: ID does not exist" containerID="2d302386d9741cf70565c459be4b1e723c66cbdb41f18ea2781d4255e53dee0a"
Sep 18 17:29:00.681142 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.681100    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d302386d9741cf70565c459be4b1e723c66cbdb41f18ea2781d4255e53dee0a"} err="failed to get container status \"2d302386d9741cf70565c459be4b1e723c66cbdb41f18ea2781d4255e53dee0a\": rpc error: code = NotFound desc = could not find container \"2d302386d9741cf70565c459be4b1e723c66cbdb41f18ea2781d4255e53dee0a\": container with ID starting with 2d302386d9741cf70565c459be4b1e723c66cbdb41f18ea2781d4255e53dee0a not found: ID does not exist"
Sep 18 17:29:00.685117 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.685095    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8bd53078-c5d887794-w8bdc"]
Sep 18 17:29:00.689658 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.689637    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8bd53078-c5d887794-w8bdc"]
Sep 18 17:29:00.700850 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.700828    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt"]
Sep 18 17:29:00.705289 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.705268    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8bd53078-5b8f657c79-zx5zt"]
Sep 18 17:29:00.717840 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.717817    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8bd53078-7c9597dc57-4cmxh"]
Sep 18 17:29:00.719594 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:00.719575    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8bd53078-7c9597dc57-4cmxh"]
Sep 18 17:29:01.765290 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:01.765250    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38ace3bc-6017-4e70-962e-9f4cd30fc2bb" path="/var/lib/kubelet/pods/38ace3bc-6017-4e70-962e-9f4cd30fc2bb/volumes"
Sep 18 17:29:01.765645 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:01.765636    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b94ea912-c166-4fa4-9381-c4c4e0428f51" path="/var/lib/kubelet/pods/b94ea912-c166-4fa4-9381-c4c4e0428f51/volumes"
Sep 18 17:29:01.766010 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:01.765996    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2d35705-eefa-4ad4-be40-7bf3f4fcd392" path="/var/lib/kubelet/pods/d2d35705-eefa-4ad4-be40-7bf3f4fcd392/volumes"
Sep 18 17:29:07.721648 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:07.721600    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fb1cf26a-584f8f98d6-clgqd"]
Sep 18 17:29:07.722156 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:07.722124    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b94ea912-c166-4fa4-9381-c4c4e0428f51" containerName="payload-processing"
Sep 18 17:29:07.722156 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:07.722142    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b94ea912-c166-4fa4-9381-c4c4e0428f51" containerName="payload-processing"
Sep 18 17:29:07.722156 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:07.722158    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2d35705-eefa-4ad4-be40-7bf3f4fcd392" containerName="maas-api"
Sep 18 17:29:07.722347 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:07.722167    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2d35705-eefa-4ad4-be40-7bf3f4fcd392" containerName="maas-api"
Sep 18 17:29:07.722347 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:07.722182    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38ace3bc-6017-4e70-962e-9f4cd30fc2bb" containerName="payload-pre-processing"
Sep 18 17:29:07.722347 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:07.722188    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="38ace3bc-6017-4e70-962e-9f4cd30fc2bb" containerName="payload-pre-processing"
Sep 18 17:29:07.722347 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:07.722259    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="38ace3bc-6017-4e70-962e-9f4cd30fc2bb" containerName="payload-pre-processing"
Sep 18 17:29:07.722347 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:07.722266    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2d35705-eefa-4ad4-be40-7bf3f4fcd392" containerName="maas-api"
Sep 18 17:29:07.722347 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:07.722273    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b94ea912-c166-4fa4-9381-c4c4e0428f51" containerName="payload-processing"
Sep 18 17:29:07.726843 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:07.726823    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb1cf26a-584f8f98d6-clgqd"
Sep 18 17:29:07.729663 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:07.729628    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-fb1cf26a\""
Sep 18 17:29:07.729663 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:07.729632    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-fb1cf26a-dockercfg-64hcc\""
Sep 18 17:29:07.735685 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:07.735658    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fb1cf26a-584f8f98d6-clgqd"]
Sep 18 17:29:07.765351 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:07.765321    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2cgk\" (UniqueName: \"kubernetes.io/projected/032c9db7-ece4-4be0-b426-8cbdf34553c3-kube-api-access-x2cgk\") pod \"payload-pre-processing-e2e-derive-fb1cf26a-584f8f98d6-clgqd\" (UID: \"032c9db7-ece4-4be0-b426-8cbdf34553c3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fb1cf26a-584f8f98d6-clgqd"
Sep 18 17:29:07.765530 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:07.765426    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/032c9db7-ece4-4be0-b426-8cbdf34553c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-fb1cf26a-584f8f98d6-clgqd\" (UID: \"032c9db7-ece4-4be0-b426-8cbdf34553c3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fb1cf26a-584f8f98d6-clgqd"
Sep 18 17:29:07.866176 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:07.866140    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/032c9db7-ece4-4be0-b426-8cbdf34553c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-fb1cf26a-584f8f98d6-clgqd\" (UID: \"032c9db7-ece4-4be0-b426-8cbdf34553c3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fb1cf26a-584f8f98d6-clgqd"
Sep 18 17:29:07.866347 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:07.866216    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2cgk\" (UniqueName: \"kubernetes.io/projected/032c9db7-ece4-4be0-b426-8cbdf34553c3-kube-api-access-x2cgk\") pod \"payload-pre-processing-e2e-derive-fb1cf26a-584f8f98d6-clgqd\" (UID: \"032c9db7-ece4-4be0-b426-8cbdf34553c3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fb1cf26a-584f8f98d6-clgqd"
Sep 18 17:29:07.866851 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:07.866832    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/032c9db7-ece4-4be0-b426-8cbdf34553c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-fb1cf26a-584f8f98d6-clgqd\" (UID: \"032c9db7-ece4-4be0-b426-8cbdf34553c3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fb1cf26a-584f8f98d6-clgqd"
Sep 18 17:29:07.875404 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:07.875370    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2cgk\" (UniqueName: \"kubernetes.io/projected/032c9db7-ece4-4be0-b426-8cbdf34553c3-kube-api-access-x2cgk\") pod \"payload-pre-processing-e2e-derive-fb1cf26a-584f8f98d6-clgqd\" (UID: \"032c9db7-ece4-4be0-b426-8cbdf34553c3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fb1cf26a-584f8f98d6-clgqd"
Sep 18 17:29:08.025450 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:08.025410    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54"
Sep 18 17:29:08.025637 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:08.025470    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54"
Sep 18 17:29:08.037682 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:08.037653    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb1cf26a-584f8f98d6-clgqd"
Sep 18 17:29:08.037888 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:08.037848    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54"
Sep 18 17:29:08.320436 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:08.320407    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fb1cf26a-584f8f98d6-clgqd"]
Sep 18 17:29:08.322635 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:29:08.322609    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod032c9db7_ece4_4be0_b426_8cbdf34553c3.slice/crio-df8421dc6768636e20b4c62694562f10bb5c89713304b70d3ef792e796237cee WatchSource:0}: Error finding container df8421dc6768636e20b4c62694562f10bb5c89713304b70d3ef792e796237cee: Status 404 returned error can't find the container with id df8421dc6768636e20b4c62694562f10bb5c89713304b70d3ef792e796237cee
Sep 18 17:29:08.709144 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:08.709041    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb1cf26a-584f8f98d6-clgqd" event={"ID":"032c9db7-ece4-4be0-b426-8cbdf34553c3","Type":"ContainerStarted","Data":"e5ffd71e68ee6d75313d4559064857c1ea609f15c2fd73b181f99c306d2c7efa"}
Sep 18 17:29:08.709144 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:08.709100    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb1cf26a-584f8f98d6-clgqd" event={"ID":"032c9db7-ece4-4be0-b426-8cbdf34553c3","Type":"ContainerStarted","Data":"df8421dc6768636e20b4c62694562f10bb5c89713304b70d3ef792e796237cee"}
Sep 18 17:29:08.709368 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:08.709248    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb1cf26a-584f8f98d6-clgqd"
Sep 18 17:29:08.720402 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:08.720370    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54"
Sep 18 17:29:08.765967 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:08.765912    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb1cf26a-584f8f98d6-clgqd" podStartSLOduration=1.765896293 podStartE2EDuration="1.765896293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:29:08.763947852 +0000 UTC m=+861.457875391" watchObservedRunningTime="2026-09-18 17:29:08.765896293 +0000 UTC m=+861.459823817"
Sep 18 17:29:19.715569 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:19.715533    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb1cf26a-584f8f98d6-clgqd"
Sep 18 17:29:20.332003 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:20.331964    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-535fd3-29qb8"]
Sep 18 17:29:20.335447 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:20.335423    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-535fd3-29qb8"
Sep 18 17:29:20.338222 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:20.338202    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qt978\""
Sep 18 17:29:20.348398 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:20.348378    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-535fd3-29qb8"]
Sep 18 17:29:20.483512 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:20.483475    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/464aeb1b-cc23-4eb7-9847-9175b29c6d3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-535fd3-29qb8\" (UID: \"464aeb1b-cc23-4eb7-9847-9175b29c6d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-535fd3-29qb8"
Sep 18 17:29:20.584098 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:20.584004    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/464aeb1b-cc23-4eb7-9847-9175b29c6d3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-535fd3-29qb8\" (UID: \"464aeb1b-cc23-4eb7-9847-9175b29c6d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-535fd3-29qb8"
Sep 18 17:29:20.584640 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:20.584617    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/464aeb1b-cc23-4eb7-9847-9175b29c6d3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-535fd3-29qb8\" (UID: \"464aeb1b-cc23-4eb7-9847-9175b29c6d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-535fd3-29qb8"
Sep 18 17:29:20.645184 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:20.645152    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-535fd3-29qb8"
Sep 18 17:29:20.916613 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:20.916429    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-535fd3-29qb8"]
Sep 18 17:29:20.919415 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:29:20.919386    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod464aeb1b_cc23_4eb7_9847_9175b29c6d3b.slice/crio-21ea338fd6a179affdde34799e1fc77385a81b652e6158d413ab86abdc8cf5e2 WatchSource:0}: Error finding container 21ea338fd6a179affdde34799e1fc77385a81b652e6158d413ab86abdc8cf5e2: Status 404 returned error can't find the container with id 21ea338fd6a179affdde34799e1fc77385a81b652e6158d413ab86abdc8cf5e2
Sep 18 17:29:21.778153 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:21.778117    2569 generic.go:356] "Generic (PLEG): container finished" podID="464aeb1b-cc23-4eb7-9847-9175b29c6d3b" containerID="0d1ed69721bcf66c2f6f39b2d04ae4a25982a8eaf50be09b64817d8273c79661" exitCode=0
Sep 18 17:29:21.778331 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:21.778195    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-535fd3-29qb8" event={"ID":"464aeb1b-cc23-4eb7-9847-9175b29c6d3b","Type":"ContainerDied","Data":"0d1ed69721bcf66c2f6f39b2d04ae4a25982a8eaf50be09b64817d8273c79661"}
Sep 18 17:29:21.778331 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:21.778248    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-535fd3-29qb8" event={"ID":"464aeb1b-cc23-4eb7-9847-9175b29c6d3b","Type":"ContainerStarted","Data":"21ea338fd6a179affdde34799e1fc77385a81b652e6158d413ab86abdc8cf5e2"}
Sep 18 17:29:22.927462 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:22.927440    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-535fd3-29qb8"
Sep 18 17:29:23.005750 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:23.005715    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/464aeb1b-cc23-4eb7-9847-9175b29c6d3b-maas-api-service-ca\") pod \"464aeb1b-cc23-4eb7-9847-9175b29c6d3b\" (UID: \"464aeb1b-cc23-4eb7-9847-9175b29c6d3b\") "
Sep 18 17:29:23.006070 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:23.006049    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/464aeb1b-cc23-4eb7-9847-9175b29c6d3b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "464aeb1b-cc23-4eb7-9847-9175b29c6d3b" (UID: "464aeb1b-cc23-4eb7-9847-9175b29c6d3b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:29:23.107428 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:23.107329    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/464aeb1b-cc23-4eb7-9847-9175b29c6d3b-maas-api-service-ca\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:29:23.790128 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:23.790096    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-535fd3-29qb8" event={"ID":"464aeb1b-cc23-4eb7-9847-9175b29c6d3b","Type":"ContainerDied","Data":"21ea338fd6a179affdde34799e1fc77385a81b652e6158d413ab86abdc8cf5e2"}
Sep 18 17:29:23.790275 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:23.790144    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21ea338fd6a179affdde34799e1fc77385a81b652e6158d413ab86abdc8cf5e2"
Sep 18 17:29:23.790275 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:23.790106    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-535fd3-29qb8"
Sep 18 17:29:30.000454 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:30.000415    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb1cf26a-jsjgg"]
Sep 18 17:29:30.001000 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:30.000981    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="464aeb1b-cc23-4eb7-9847-9175b29c6d3b" containerName="revoke-keys"
Sep 18 17:29:30.001089 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:30.001002    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="464aeb1b-cc23-4eb7-9847-9175b29c6d3b" containerName="revoke-keys"
Sep 18 17:29:30.001147 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:30.001123    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="464aeb1b-cc23-4eb7-9847-9175b29c6d3b" containerName="revoke-keys"
Sep 18 17:29:30.004260 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:30.004240    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb1cf26a-jsjgg"
Sep 18 17:29:30.007233 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:30.007217    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qt978\""
Sep 18 17:29:30.014422 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:30.014386    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb1cf26a-jsjgg"]
Sep 18 17:29:30.172542 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:30.172497    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68cd8210-e3ed-460c-a681-cc9a1b7ba8c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fb1cf26a-jsjgg\" (UID: \"68cd8210-e3ed-460c-a681-cc9a1b7ba8c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb1cf26a-jsjgg"
Sep 18 17:29:30.273659 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:30.273578    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68cd8210-e3ed-460c-a681-cc9a1b7ba8c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fb1cf26a-jsjgg\" (UID: \"68cd8210-e3ed-460c-a681-cc9a1b7ba8c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb1cf26a-jsjgg"
Sep 18 17:29:30.274262 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:30.274239    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68cd8210-e3ed-460c-a681-cc9a1b7ba8c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fb1cf26a-jsjgg\" (UID: \"68cd8210-e3ed-460c-a681-cc9a1b7ba8c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb1cf26a-jsjgg"
Sep 18 17:29:30.316552 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:30.316510    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb1cf26a-jsjgg"
Sep 18 17:29:30.589401 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:30.589325    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb1cf26a-jsjgg"]
Sep 18 17:29:30.827468 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:30.827431    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb1cf26a-jsjgg" event={"ID":"68cd8210-e3ed-460c-a681-cc9a1b7ba8c3","Type":"ContainerStarted","Data":"9f28dc20e105d3f817e8ebe2ceb4d66e5acb0aa7dae32e960a8fb0ceb04cb354"}
Sep 18 17:29:31.834173 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:31.834127    2569 generic.go:356] "Generic (PLEG): container finished" podID="68cd8210-e3ed-460c-a681-cc9a1b7ba8c3" containerID="50e8a5907e7c22255b7b99ec5a2a1c7f827b8245d6eef9061aed9f9e13190e3f" exitCode=0
Sep 18 17:29:31.834571 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:31.834176    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb1cf26a-jsjgg" event={"ID":"68cd8210-e3ed-460c-a681-cc9a1b7ba8c3","Type":"ContainerDied","Data":"50e8a5907e7c22255b7b99ec5a2a1c7f827b8245d6eef9061aed9f9e13190e3f"}
Sep 18 17:29:32.581333 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:32.581303    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh"]
Sep 18 17:29:32.581514 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:32.581487    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh" podUID="801c76f4-6045-42d3-8cf1-a464f147e1b4" containerName="maas-api" containerID="cri-o://05cf6d944da56f93931352d8227ae14e02b121f1620d66bd12fb6b9541687dc6" gracePeriod=30
Sep 18 17:29:32.681057 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:32.681021    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-535fd3-6ff4c66bf8-4c5nk"]
Sep 18 17:29:32.681258 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:32.681229    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-535fd3-6ff4c66bf8-4c5nk" podUID="bb37e50a-97c7-4c6d-9721-b9eefead8bc5" containerName="payload-pre-processing" containerID="cri-o://91ac6fd4b814ceecc159faf834e8454697e42d89c0c2c59cf314654051ea3117" gracePeriod=30
Sep 18 17:29:32.835746 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:32.835676    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh"
Sep 18 17:29:32.840699 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:32.840672    2569 generic.go:356] "Generic (PLEG): container finished" podID="801c76f4-6045-42d3-8cf1-a464f147e1b4" containerID="05cf6d944da56f93931352d8227ae14e02b121f1620d66bd12fb6b9541687dc6" exitCode=0
Sep 18 17:29:32.840843 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:32.840715    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh"
Sep 18 17:29:32.840843 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:32.840749    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh" event={"ID":"801c76f4-6045-42d3-8cf1-a464f147e1b4","Type":"ContainerDied","Data":"05cf6d944da56f93931352d8227ae14e02b121f1620d66bd12fb6b9541687dc6"}
Sep 18 17:29:32.840843 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:32.840801    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh" event={"ID":"801c76f4-6045-42d3-8cf1-a464f147e1b4","Type":"ContainerDied","Data":"f9d328cd10b0fcea1b5c993e1dee7233c602fe4036bb6295afe2aa63d57825d9"}
Sep 18 17:29:32.840843 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:32.840816    2569 scope.go:117] "RemoveContainer" containerID="05cf6d944da56f93931352d8227ae14e02b121f1620d66bd12fb6b9541687dc6"
Sep 18 17:29:32.842514 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:32.842490    2569 generic.go:356] "Generic (PLEG): container finished" podID="bb37e50a-97c7-4c6d-9721-b9eefead8bc5" containerID="91ac6fd4b814ceecc159faf834e8454697e42d89c0c2c59cf314654051ea3117" exitCode=0
Sep 18 17:29:32.842610 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:32.842558    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-535fd3-6ff4c66bf8-4c5nk" event={"ID":"bb37e50a-97c7-4c6d-9721-b9eefead8bc5","Type":"ContainerDied","Data":"91ac6fd4b814ceecc159faf834e8454697e42d89c0c2c59cf314654051ea3117"}
Sep 18 17:29:32.850397 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:32.850372    2569 scope.go:117] "RemoveContainer" containerID="05cf6d944da56f93931352d8227ae14e02b121f1620d66bd12fb6b9541687dc6"
Sep 18 17:29:32.850786 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:29:32.850764    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05cf6d944da56f93931352d8227ae14e02b121f1620d66bd12fb6b9541687dc6\": container with ID starting with 05cf6d944da56f93931352d8227ae14e02b121f1620d66bd12fb6b9541687dc6 not found: ID does not exist" containerID="05cf6d944da56f93931352d8227ae14e02b121f1620d66bd12fb6b9541687dc6"
Sep 18 17:29:32.850882 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:32.850793    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05cf6d944da56f93931352d8227ae14e02b121f1620d66bd12fb6b9541687dc6"} err="failed to get container status \"05cf6d944da56f93931352d8227ae14e02b121f1620d66bd12fb6b9541687dc6\": rpc error: code = NotFound desc = could not find container \"05cf6d944da56f93931352d8227ae14e02b121f1620d66bd12fb6b9541687dc6\": container with ID starting with 05cf6d944da56f93931352d8227ae14e02b121f1620d66bd12fb6b9541687dc6 not found: ID does not exist"
Sep 18 17:29:32.911204 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:32.911172    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-535fd3-29qb8"]
Sep 18 17:29:32.915712 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:32.915681    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-535fd3-29qb8"]
Sep 18 17:29:32.959784 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:32.959750    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-535fd3-6ff4c66bf8-4c5nk"
Sep 18 17:29:32.999507 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:32.999473    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb1cf26a-jsjgg"
Sep 18 17:29:33.001948 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:33.001928    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/801c76f4-6045-42d3-8cf1-a464f147e1b4-maas-api-tls\") pod \"801c76f4-6045-42d3-8cf1-a464f147e1b4\" (UID: \"801c76f4-6045-42d3-8cf1-a464f147e1b4\") "
Sep 18 17:29:33.002054 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:33.001966    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/801c76f4-6045-42d3-8cf1-a464f147e1b4-openshift-service-ca\") pod \"801c76f4-6045-42d3-8cf1-a464f147e1b4\" (UID: \"801c76f4-6045-42d3-8cf1-a464f147e1b4\") "
Sep 18 17:29:33.002054 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:33.002008    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrxsf\" (UniqueName: \"kubernetes.io/projected/801c76f4-6045-42d3-8cf1-a464f147e1b4-kube-api-access-zrxsf\") pod \"801c76f4-6045-42d3-8cf1-a464f147e1b4\" (UID: \"801c76f4-6045-42d3-8cf1-a464f147e1b4\") "
Sep 18 17:29:33.002147 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:33.002067    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/801c76f4-6045-42d3-8cf1-a464f147e1b4-metrics-certs\") pod \"801c76f4-6045-42d3-8cf1-a464f147e1b4\" (UID: \"801c76f4-6045-42d3-8cf1-a464f147e1b4\") "
Sep 18 17:29:33.002389 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:33.002366    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/801c76f4-6045-42d3-8cf1-a464f147e1b4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "801c76f4-6045-42d3-8cf1-a464f147e1b4" (UID: "801c76f4-6045-42d3-8cf1-a464f147e1b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:29:33.004025 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:33.004002    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/801c76f4-6045-42d3-8cf1-a464f147e1b4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "801c76f4-6045-42d3-8cf1-a464f147e1b4" (UID: "801c76f4-6045-42d3-8cf1-a464f147e1b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:29:33.004141 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:33.004115    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/801c76f4-6045-42d3-8cf1-a464f147e1b4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "801c76f4-6045-42d3-8cf1-a464f147e1b4" (UID: "801c76f4-6045-42d3-8cf1-a464f147e1b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:29:33.004365 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:33.004343    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/801c76f4-6045-42d3-8cf1-a464f147e1b4-kube-api-access-zrxsf" (OuterVolumeSpecName: "kube-api-access-zrxsf") pod "801c76f4-6045-42d3-8cf1-a464f147e1b4" (UID: "801c76f4-6045-42d3-8cf1-a464f147e1b4"). InnerVolumeSpecName "kube-api-access-zrxsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:29:33.103437 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:33.103341    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68cd8210-e3ed-460c-a681-cc9a1b7ba8c3-maas-api-service-ca\") pod \"68cd8210-e3ed-460c-a681-cc9a1b7ba8c3\" (UID: \"68cd8210-e3ed-460c-a681-cc9a1b7ba8c3\") "
Sep 18 17:29:33.103437 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:33.103402    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb37e50a-97c7-4c6d-9721-b9eefead8bc5-plugins-config-volume\") pod \"bb37e50a-97c7-4c6d-9721-b9eefead8bc5\" (UID: \"bb37e50a-97c7-4c6d-9721-b9eefead8bc5\") "
Sep 18 17:29:33.103437 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:33.103439    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rgkd\" (UniqueName: \"kubernetes.io/projected/bb37e50a-97c7-4c6d-9721-b9eefead8bc5-kube-api-access-5rgkd\") pod \"bb37e50a-97c7-4c6d-9721-b9eefead8bc5\" (UID: \"bb37e50a-97c7-4c6d-9721-b9eefead8bc5\") "
Sep 18 17:29:33.103667 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:33.103645    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/801c76f4-6045-42d3-8cf1-a464f147e1b4-maas-api-tls\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:29:33.103667 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:33.103657    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/801c76f4-6045-42d3-8cf1-a464f147e1b4-openshift-service-ca\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:29:33.103736 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:33.103668    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrxsf\" (UniqueName: \"kubernetes.io/projected/801c76f4-6045-42d3-8cf1-a464f147e1b4-kube-api-access-zrxsf\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:29:33.103736 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:33.103688    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/801c76f4-6045-42d3-8cf1-a464f147e1b4-metrics-certs\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:29:33.103736 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:33.103717    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68cd8210-e3ed-460c-a681-cc9a1b7ba8c3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "68cd8210-e3ed-460c-a681-cc9a1b7ba8c3" (UID: "68cd8210-e3ed-460c-a681-cc9a1b7ba8c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:29:33.103848 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:33.103825    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb37e50a-97c7-4c6d-9721-b9eefead8bc5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bb37e50a-97c7-4c6d-9721-b9eefead8bc5" (UID: "bb37e50a-97c7-4c6d-9721-b9eefead8bc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:29:33.105477 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:33.105460    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb37e50a-97c7-4c6d-9721-b9eefead8bc5-kube-api-access-5rgkd" (OuterVolumeSpecName: "kube-api-access-5rgkd") pod "bb37e50a-97c7-4c6d-9721-b9eefead8bc5" (UID: "bb37e50a-97c7-4c6d-9721-b9eefead8bc5"). InnerVolumeSpecName "kube-api-access-5rgkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:29:33.164790 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:33.164751    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh"]
Sep 18 17:29:33.167003 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:33.166974    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-535fd3-5c65ffbdc-mkzxh"]
Sep 18 17:29:33.205167 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:33.205138    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68cd8210-e3ed-460c-a681-cc9a1b7ba8c3-maas-api-service-ca\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:29:33.205167 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:33.205164    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb37e50a-97c7-4c6d-9721-b9eefead8bc5-plugins-config-volume\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:29:33.205167 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:33.205175    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rgkd\" (UniqueName: \"kubernetes.io/projected/bb37e50a-97c7-4c6d-9721-b9eefead8bc5-kube-api-access-5rgkd\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:29:33.765957 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:33.765922    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="464aeb1b-cc23-4eb7-9847-9175b29c6d3b" path="/var/lib/kubelet/pods/464aeb1b-cc23-4eb7-9847-9175b29c6d3b/volumes"
Sep 18 17:29:33.766338 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:33.766323    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="801c76f4-6045-42d3-8cf1-a464f147e1b4" path="/var/lib/kubelet/pods/801c76f4-6045-42d3-8cf1-a464f147e1b4/volumes"
Sep 18 17:29:33.849324 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:33.849233    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb1cf26a-jsjgg"
Sep 18 17:29:33.849324 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:33.849268    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb1cf26a-jsjgg" event={"ID":"68cd8210-e3ed-460c-a681-cc9a1b7ba8c3","Type":"ContainerDied","Data":"9f28dc20e105d3f817e8ebe2ceb4d66e5acb0aa7dae32e960a8fb0ceb04cb354"}
Sep 18 17:29:33.849324 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:33.849318    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f28dc20e105d3f817e8ebe2ceb4d66e5acb0aa7dae32e960a8fb0ceb04cb354"
Sep 18 17:29:33.851783 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:33.851758    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-535fd3-6ff4c66bf8-4c5nk" event={"ID":"bb37e50a-97c7-4c6d-9721-b9eefead8bc5","Type":"ContainerDied","Data":"f4dc13f42b5f2d6cc5acf802a65c4d6de9a87183310bc9c9d9ff672e33062fff"}
Sep 18 17:29:33.851933 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:33.851811    2569 scope.go:117] "RemoveContainer" containerID="91ac6fd4b814ceecc159faf834e8454697e42d89c0c2c59cf314654051ea3117"
Sep 18 17:29:33.851933 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:33.851841    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-535fd3-6ff4c66bf8-4c5nk"
Sep 18 17:29:33.894846 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:33.894805    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-535fd3-6ff4c66bf8-4c5nk"]
Sep 18 17:29:33.908371 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:33.908340    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-535fd3-6ff4c66bf8-4c5nk"]
Sep 18 17:29:35.765272 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:35.765238    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb37e50a-97c7-4c6d-9721-b9eefead8bc5" path="/var/lib/kubelet/pods/bb37e50a-97c7-4c6d-9721-b9eefead8bc5/volumes"
Sep 18 17:29:37.136723 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:37.136634    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff"]
Sep 18 17:29:38.599522 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:38.599475    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fb1cf26a-584f8f98d6-clgqd"]
Sep 18 17:29:38.600007 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:38.599708    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb1cf26a-584f8f98d6-clgqd" podUID="032c9db7-ece4-4be0-b426-8cbdf34553c3" containerName="payload-pre-processing" containerID="cri-o://e5ffd71e68ee6d75313d4559064857c1ea609f15c2fd73b181f99c306d2c7efa" gracePeriod=30
Sep 18 17:29:38.777640 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:38.777608    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb1cf26a-jsjgg"]
Sep 18 17:29:38.781855 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:38.781828    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb1cf26a-jsjgg"]
Sep 18 17:29:38.858619 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:38.858558    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb1cf26a-584f8f98d6-clgqd"
Sep 18 17:29:38.881828 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:38.881792    2569 generic.go:356] "Generic (PLEG): container finished" podID="032c9db7-ece4-4be0-b426-8cbdf34553c3" containerID="e5ffd71e68ee6d75313d4559064857c1ea609f15c2fd73b181f99c306d2c7efa" exitCode=0
Sep 18 17:29:38.882073 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:38.881842    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb1cf26a-584f8f98d6-clgqd" event={"ID":"032c9db7-ece4-4be0-b426-8cbdf34553c3","Type":"ContainerDied","Data":"e5ffd71e68ee6d75313d4559064857c1ea609f15c2fd73b181f99c306d2c7efa"}
Sep 18 17:29:38.882221 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:38.882117    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb1cf26a-584f8f98d6-clgqd" event={"ID":"032c9db7-ece4-4be0-b426-8cbdf34553c3","Type":"ContainerDied","Data":"df8421dc6768636e20b4c62694562f10bb5c89713304b70d3ef792e796237cee"}
Sep 18 17:29:38.882221 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:38.881856    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fb1cf26a-584f8f98d6-clgqd"
Sep 18 17:29:38.882221 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:38.882153    2569 scope.go:117] "RemoveContainer" containerID="e5ffd71e68ee6d75313d4559064857c1ea609f15c2fd73b181f99c306d2c7efa"
Sep 18 17:29:38.891960 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:38.891941    2569 scope.go:117] "RemoveContainer" containerID="e5ffd71e68ee6d75313d4559064857c1ea609f15c2fd73b181f99c306d2c7efa"
Sep 18 17:29:38.892273 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:29:38.892243    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5ffd71e68ee6d75313d4559064857c1ea609f15c2fd73b181f99c306d2c7efa\": container with ID starting with e5ffd71e68ee6d75313d4559064857c1ea609f15c2fd73b181f99c306d2c7efa not found: ID does not exist" containerID="e5ffd71e68ee6d75313d4559064857c1ea609f15c2fd73b181f99c306d2c7efa"
Sep 18 17:29:38.892330 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:38.892273    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5ffd71e68ee6d75313d4559064857c1ea609f15c2fd73b181f99c306d2c7efa"} err="failed to get container status \"e5ffd71e68ee6d75313d4559064857c1ea609f15c2fd73b181f99c306d2c7efa\": rpc error: code = NotFound desc = could not find container \"e5ffd71e68ee6d75313d4559064857c1ea609f15c2fd73b181f99c306d2c7efa\": container with ID starting with e5ffd71e68ee6d75313d4559064857c1ea609f15c2fd73b181f99c306d2c7efa not found: ID does not exist"
Sep 18 17:29:38.959977 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:38.959945    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/032c9db7-ece4-4be0-b426-8cbdf34553c3-plugins-config-volume\") pod \"032c9db7-ece4-4be0-b426-8cbdf34553c3\" (UID: \"032c9db7-ece4-4be0-b426-8cbdf34553c3\") "
Sep 18 17:29:38.960145 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:38.960034    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2cgk\" (UniqueName: \"kubernetes.io/projected/032c9db7-ece4-4be0-b426-8cbdf34553c3-kube-api-access-x2cgk\") pod \"032c9db7-ece4-4be0-b426-8cbdf34553c3\" (UID: \"032c9db7-ece4-4be0-b426-8cbdf34553c3\") "
Sep 18 17:29:38.960306 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:38.960282    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/032c9db7-ece4-4be0-b426-8cbdf34553c3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "032c9db7-ece4-4be0-b426-8cbdf34553c3" (UID: "032c9db7-ece4-4be0-b426-8cbdf34553c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:29:38.962069 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:38.962042    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/032c9db7-ece4-4be0-b426-8cbdf34553c3-kube-api-access-x2cgk" (OuterVolumeSpecName: "kube-api-access-x2cgk") pod "032c9db7-ece4-4be0-b426-8cbdf34553c3" (UID: "032c9db7-ece4-4be0-b426-8cbdf34553c3"). InnerVolumeSpecName "kube-api-access-x2cgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:29:39.061561 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:39.061521    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/032c9db7-ece4-4be0-b426-8cbdf34553c3-plugins-config-volume\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:29:39.061561 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:39.061552    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2cgk\" (UniqueName: \"kubernetes.io/projected/032c9db7-ece4-4be0-b426-8cbdf34553c3-kube-api-access-x2cgk\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:29:39.204704 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:39.204670    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fb1cf26a-584f8f98d6-clgqd"]
Sep 18 17:29:39.206455 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:39.206435    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fb1cf26a-584f8f98d6-clgqd"]
Sep 18 17:29:39.764633 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:39.764600    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="032c9db7-ece4-4be0-b426-8cbdf34553c3" path="/var/lib/kubelet/pods/032c9db7-ece4-4be0-b426-8cbdf34553c3/volumes"
Sep 18 17:29:39.765018 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:39.764988    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68cd8210-e3ed-460c-a681-cc9a1b7ba8c3" path="/var/lib/kubelet/pods/68cd8210-e3ed-460c-a681-cc9a1b7ba8c3/volumes"
Sep 18 17:29:52.155607 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:52.155550    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff" podUID="5e72e7b6-5c6f-4754-9ade-f027ccbc98ef" containerName="main" containerID="cri-o://f3bed3db5778ff5bd54fc137af381e5e572fb01d475e213ee78ae13947d613de" gracePeriod=45
Sep 18 17:29:52.413341 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:52.413283    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff"
Sep 18 17:29:52.481057 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:52.481026    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-model-cache\") pod \"5e72e7b6-5c6f-4754-9ade-f027ccbc98ef\" (UID: \"5e72e7b6-5c6f-4754-9ade-f027ccbc98ef\") "
Sep 18 17:29:52.481195 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:52.481065    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-home\") pod \"5e72e7b6-5c6f-4754-9ade-f027ccbc98ef\" (UID: \"5e72e7b6-5c6f-4754-9ade-f027ccbc98ef\") "
Sep 18 17:29:52.481195 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:52.481104    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-dshm\") pod \"5e72e7b6-5c6f-4754-9ade-f027ccbc98ef\" (UID: \"5e72e7b6-5c6f-4754-9ade-f027ccbc98ef\") "
Sep 18 17:29:52.481195 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:52.481163    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-tmp-dir\") pod \"5e72e7b6-5c6f-4754-9ade-f027ccbc98ef\" (UID: \"5e72e7b6-5c6f-4754-9ade-f027ccbc98ef\") "
Sep 18 17:29:52.481339 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:52.481227    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-tls-certs\") pod \"5e72e7b6-5c6f-4754-9ade-f027ccbc98ef\" (UID: \"5e72e7b6-5c6f-4754-9ade-f027ccbc98ef\") "
Sep 18 17:29:52.481339 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:52.481314    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5948f\" (UniqueName: \"kubernetes.io/projected/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-kube-api-access-5948f\") pod \"5e72e7b6-5c6f-4754-9ade-f027ccbc98ef\" (UID: \"5e72e7b6-5c6f-4754-9ade-f027ccbc98ef\") "
Sep 18 17:29:52.481455 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:52.481382    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-model-cache" (OuterVolumeSpecName: "model-cache") pod "5e72e7b6-5c6f-4754-9ade-f027ccbc98ef" (UID: "5e72e7b6-5c6f-4754-9ade-f027ccbc98ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:29:52.481455 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:52.481400    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-home" (OuterVolumeSpecName: "home") pod "5e72e7b6-5c6f-4754-9ade-f027ccbc98ef" (UID: "5e72e7b6-5c6f-4754-9ade-f027ccbc98ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:29:52.481565 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:52.481509    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5e72e7b6-5c6f-4754-9ade-f027ccbc98ef" (UID: "5e72e7b6-5c6f-4754-9ade-f027ccbc98ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:29:52.481679 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:52.481663    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-model-cache\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:29:52.481733 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:52.481683    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-home\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:29:52.481733 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:52.481696    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-tmp-dir\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:29:52.483382 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:52.483360    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-dshm" (OuterVolumeSpecName: "dshm") pod "5e72e7b6-5c6f-4754-9ade-f027ccbc98ef" (UID: "5e72e7b6-5c6f-4754-9ade-f027ccbc98ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:29:52.483491 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:52.483385    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5e72e7b6-5c6f-4754-9ade-f027ccbc98ef" (UID: "5e72e7b6-5c6f-4754-9ade-f027ccbc98ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:29:52.483491 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:52.483465    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-kube-api-access-5948f" (OuterVolumeSpecName: "kube-api-access-5948f") pod "5e72e7b6-5c6f-4754-9ade-f027ccbc98ef" (UID: "5e72e7b6-5c6f-4754-9ade-f027ccbc98ef"). InnerVolumeSpecName "kube-api-access-5948f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:29:52.582509 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:52.582470    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5948f\" (UniqueName: \"kubernetes.io/projected/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-kube-api-access-5948f\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:29:52.582509 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:52.582502    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-dshm\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:29:52.582509 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:52.582511    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef-tls-certs\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:29:52.955144 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:52.955112    2569 generic.go:356] "Generic (PLEG): container finished" podID="5e72e7b6-5c6f-4754-9ade-f027ccbc98ef" containerID="f3bed3db5778ff5bd54fc137af381e5e572fb01d475e213ee78ae13947d613de" exitCode=0
Sep 18 17:29:52.955305 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:52.955153    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff"
Sep 18 17:29:52.955305 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:52.955202    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff" event={"ID":"5e72e7b6-5c6f-4754-9ade-f027ccbc98ef","Type":"ContainerDied","Data":"f3bed3db5778ff5bd54fc137af381e5e572fb01d475e213ee78ae13947d613de"}
Sep 18 17:29:52.955305 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:52.955241    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff" event={"ID":"5e72e7b6-5c6f-4754-9ade-f027ccbc98ef","Type":"ContainerDied","Data":"ab28f18a2c720b37d6066058ccab6dd587db142ffd6a0dd3d4cd7dfd26bdd903"}
Sep 18 17:29:52.955305 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:52.955258    2569 scope.go:117] "RemoveContainer" containerID="f3bed3db5778ff5bd54fc137af381e5e572fb01d475e213ee78ae13947d613de"
Sep 18 17:29:52.969838 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:52.969817    2569 scope.go:117] "RemoveContainer" containerID="f3bed3db5778ff5bd54fc137af381e5e572fb01d475e213ee78ae13947d613de"
Sep 18 17:29:52.970175 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:29:52.970158    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3bed3db5778ff5bd54fc137af381e5e572fb01d475e213ee78ae13947d613de\": container with ID starting with f3bed3db5778ff5bd54fc137af381e5e572fb01d475e213ee78ae13947d613de not found: ID does not exist" containerID="f3bed3db5778ff5bd54fc137af381e5e572fb01d475e213ee78ae13947d613de"
Sep 18 17:29:52.970236 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:52.970181    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3bed3db5778ff5bd54fc137af381e5e572fb01d475e213ee78ae13947d613de"} err="failed to get container status \"f3bed3db5778ff5bd54fc137af381e5e572fb01d475e213ee78ae13947d613de\": rpc error: code = NotFound desc = could not find container \"f3bed3db5778ff5bd54fc137af381e5e572fb01d475e213ee78ae13947d613de\": container with ID starting with f3bed3db5778ff5bd54fc137af381e5e572fb01d475e213ee78ae13947d613de not found: ID does not exist"
Sep 18 17:29:52.980759 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:52.980733    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff"]
Sep 18 17:29:52.985562 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:52.985540    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-535fd3/autores-model-535fd3-kserve-76b546457f-27zff"]
Sep 18 17:29:53.764646 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:29:53.764615    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e72e7b6-5c6f-4754-9ade-f027ccbc98ef" path="/var/lib/kubelet/pods/5e72e7b6-5c6f-4754-9ade-f027ccbc98ef/volumes"
Sep 18 17:30:00.151420 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.151385    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-5b6d69-29829210-ln4kp"]
Sep 18 17:30:00.153899 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.151710    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e72e7b6-5c6f-4754-9ade-f027ccbc98ef" containerName="main"
Sep 18 17:30:00.153899 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.151720    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e72e7b6-5c6f-4754-9ade-f027ccbc98ef" containerName="main"
Sep 18 17:30:00.153899 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.151732    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb37e50a-97c7-4c6d-9721-b9eefead8bc5" containerName="payload-pre-processing"
Sep 18 17:30:00.153899 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.151738    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb37e50a-97c7-4c6d-9721-b9eefead8bc5" containerName="payload-pre-processing"
Sep 18 17:30:00.153899 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.151752    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="801c76f4-6045-42d3-8cf1-a464f147e1b4" containerName="maas-api"
Sep 18 17:30:00.153899 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.151759    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="801c76f4-6045-42d3-8cf1-a464f147e1b4" containerName="maas-api"
Sep 18 17:30:00.153899 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.151765    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="032c9db7-ece4-4be0-b426-8cbdf34553c3" containerName="payload-pre-processing"
Sep 18 17:30:00.153899 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.151770    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="032c9db7-ece4-4be0-b426-8cbdf34553c3" containerName="payload-pre-processing"
Sep 18 17:30:00.153899 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.151776    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68cd8210-e3ed-460c-a681-cc9a1b7ba8c3" containerName="revoke-keys"
Sep 18 17:30:00.153899 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.151782    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="68cd8210-e3ed-460c-a681-cc9a1b7ba8c3" containerName="revoke-keys"
Sep 18 17:30:00.153899 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.151838    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="032c9db7-ece4-4be0-b426-8cbdf34553c3" containerName="payload-pre-processing"
Sep 18 17:30:00.153899 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.151847    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb37e50a-97c7-4c6d-9721-b9eefead8bc5" containerName="payload-pre-processing"
Sep 18 17:30:00.153899 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.151853    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="68cd8210-e3ed-460c-a681-cc9a1b7ba8c3" containerName="revoke-keys"
Sep 18 17:30:00.153899 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.151872    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e72e7b6-5c6f-4754-9ade-f027ccbc98ef" containerName="main"
Sep 18 17:30:00.153899 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.151880    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="801c76f4-6045-42d3-8cf1-a464f147e1b4" containerName="maas-api"
Sep 18 17:30:00.154693 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.154678    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-5b6d69-29829210-ln4kp"
Sep 18 17:30:00.157361 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.157327    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qt978\""
Sep 18 17:30:00.160365 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.160343    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aa65a-29829210-b8g9l"]
Sep 18 17:30:00.164019 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.163999    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a2e13-29829210-fpjfn"]
Sep 18 17:30:00.164152 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.164119    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aa65a-29829210-b8g9l"
Sep 18 17:30:00.165778 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.165757    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-5b6d69-29829210-ln4kp"
Sep 18 17:30:00.167603 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.167583    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829210-mn2xl"]
Sep 18 17:30:00.167731 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.167702    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a2e13-29829210-fpjfn"
Sep 18 17:30:00.171161 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.171143    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-5b6d69-29829210-ln4kp"]
Sep 18 17:30:00.171421 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.171406    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829210-mn2xl"
Sep 18 17:30:00.173949 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.173474    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aa65a-29829210-b8g9l"]
Sep 18 17:30:00.177401 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.177382    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a2e13-29829210-fpjfn"]
Sep 18 17:30:00.179623 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.179460    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aa65a-29829210-b8g9l"
Sep 18 17:30:00.179727 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.179632    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829210-mn2xl"]
Sep 18 17:30:00.220388 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.220366    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a2e13-29829210-fpjfn"
Sep 18 17:30:00.231885 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.231624    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829210-mn2xl"
Sep 18 17:30:00.451512 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.451484    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-5b6d69-29829210-ln4kp"]
Sep 18 17:30:00.453904 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:30:00.453854    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b79e5e_54f1_4eb5_88ca_3c4f4c8885ae.slice/crio-e0050181d4979449afc34950d9f9153967eb3ca913b3e8a35977173042eefb1f WatchSource:0}: Error finding container e0050181d4979449afc34950d9f9153967eb3ca913b3e8a35977173042eefb1f: Status 404 returned error can't find the container with id e0050181d4979449afc34950d9f9153967eb3ca913b3e8a35977173042eefb1f
Sep 18 17:30:00.473150 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.473125    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aa65a-29829210-b8g9l"]
Sep 18 17:30:00.473796 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:30:00.473773    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53add21d_9226_4dab_ae83_80ea86ba930b.slice/crio-7b2ba42024855d0548496aacea027fa7ddb0e95bc74dfa4c6083cb6e3b2c6924 WatchSource:0}: Error finding container 7b2ba42024855d0548496aacea027fa7ddb0e95bc74dfa4c6083cb6e3b2c6924: Status 404 returned error can't find the container with id 7b2ba42024855d0548496aacea027fa7ddb0e95bc74dfa4c6083cb6e3b2c6924
Sep 18 17:30:00.519074 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.519047    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829210-mn2xl"]
Sep 18 17:30:00.521592 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:30:00.521266    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e1310bd_d032_45ee_bbc6_7adee2636dad.slice/crio-30fa141b541fc23802e614f32f46eff3a7819a60f91ff832d9d1e46999a86c60 WatchSource:0}: Error finding container 30fa141b541fc23802e614f32f46eff3a7819a60f91ff832d9d1e46999a86c60: Status 404 returned error can't find the container with id 30fa141b541fc23802e614f32f46eff3a7819a60f91ff832d9d1e46999a86c60
Sep 18 17:30:00.523166 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.523132    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a2e13-29829210-fpjfn"]
Sep 18 17:30:00.526240 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:30:00.526218    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d51e20_2f48_4fbb_ac4e_5ad2377ad376.slice/crio-adc0a93fcaf77f1ef17b04532df550204aac7f1c0e58af07e69d06c82cd9bae5 WatchSource:0}: Error finding container adc0a93fcaf77f1ef17b04532df550204aac7f1c0e58af07e69d06c82cd9bae5: Status 404 returned error can't find the container with id adc0a93fcaf77f1ef17b04532df550204aac7f1c0e58af07e69d06c82cd9bae5
Sep 18 17:30:00.999767 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.999723    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-5b6d69-29829210-ln4kp" event={"ID":"e8b79e5e-54f1-4eb5-88ca-3c4f4c8885ae","Type":"ContainerStarted","Data":"530e44366f27cefed9791912efd0cda29a829cc4abb66e8825504f02dc7f9394"}
Sep 18 17:30:00.999982 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:00.999786    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-5b6d69-29829210-ln4kp" event={"ID":"e8b79e5e-54f1-4eb5-88ca-3c4f4c8885ae","Type":"ContainerStarted","Data":"e0050181d4979449afc34950d9f9153967eb3ca913b3e8a35977173042eefb1f"}
Sep 18 17:30:01.001307 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:01.001275    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829210-mn2xl" event={"ID":"3e1310bd-d032-45ee-bbc6-7adee2636dad","Type":"ContainerStarted","Data":"a28819e2cc681610d6353e0bac65e65026e7e2e0a9bd3e852f92c52f8c5d1a6d"}
Sep 18 17:30:01.001410 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:01.001331    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829210-mn2xl" event={"ID":"3e1310bd-d032-45ee-bbc6-7adee2636dad","Type":"ContainerStarted","Data":"30fa141b541fc23802e614f32f46eff3a7819a60f91ff832d9d1e46999a86c60"}
Sep 18 17:30:01.002773 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:01.002747    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a2e13-29829210-fpjfn" event={"ID":"63d51e20-2f48-4fbb-ac4e-5ad2377ad376","Type":"ContainerStarted","Data":"604aab1db04837e4b458e53c9fe65f96500da32ea4347f3109dcd88144dd0a84"}
Sep 18 17:30:01.002876 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:01.002794    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a2e13-29829210-fpjfn" event={"ID":"63d51e20-2f48-4fbb-ac4e-5ad2377ad376","Type":"ContainerStarted","Data":"adc0a93fcaf77f1ef17b04532df550204aac7f1c0e58af07e69d06c82cd9bae5"}
Sep 18 17:30:01.004295 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:01.004244    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aa65a-29829210-b8g9l" event={"ID":"53add21d-9226-4dab-ae83-80ea86ba930b","Type":"ContainerStarted","Data":"034258b763cb0612896b9ea4ba64c7a329a20e4203142b089784adc473f0e490"}
Sep 18 17:30:01.004295 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:01.004279    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aa65a-29829210-b8g9l" event={"ID":"53add21d-9226-4dab-ae83-80ea86ba930b","Type":"ContainerStarted","Data":"7b2ba42024855d0548496aacea027fa7ddb0e95bc74dfa4c6083cb6e3b2c6924"}
Sep 18 17:30:01.037528 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:01.037475    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-5b6d69-29829210-ln4kp" podStartSLOduration=1.037461991 podStartE2EDuration="1.037461991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:30:01.035930701 +0000 UTC m=+913.729858251" watchObservedRunningTime="2026-09-18 17:30:01.037461991 +0000 UTC m=+913.731389518"
Sep 18 17:30:01.075356 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:01.075311    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829210-mn2xl" podStartSLOduration=1.075295265 podStartE2EDuration="1.075295265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:30:01.075184007 +0000 UTC m=+913.769111537" watchObservedRunningTime="2026-09-18 17:30:01.075295265 +0000 UTC m=+913.769222794"
Sep 18 17:30:01.116346 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:01.116300    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aa65a-29829210-b8g9l" podStartSLOduration=1.116285179 podStartE2EDuration="1.116285179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:30:01.115303791 +0000 UTC m=+913.809231343" watchObservedRunningTime="2026-09-18 17:30:01.116285179 +0000 UTC m=+913.810212749"
Sep 18 17:30:01.144809 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:01.144745    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a2e13-29829210-fpjfn" podStartSLOduration=1.144726319 podStartE2EDuration="1.144726319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:30:01.141930009 +0000 UTC m=+913.835857539" watchObservedRunningTime="2026-09-18 17:30:01.144726319 +0000 UTC m=+913.838653847"
Sep 18 17:30:02.011803 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:02.011767    2569 generic.go:356] "Generic (PLEG): container finished" podID="63d51e20-2f48-4fbb-ac4e-5ad2377ad376" containerID="604aab1db04837e4b458e53c9fe65f96500da32ea4347f3109dcd88144dd0a84" exitCode=0
Sep 18 17:30:02.012264 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:02.011836    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a2e13-29829210-fpjfn" event={"ID":"63d51e20-2f48-4fbb-ac4e-5ad2377ad376","Type":"ContainerDied","Data":"604aab1db04837e4b458e53c9fe65f96500da32ea4347f3109dcd88144dd0a84"}
Sep 18 17:30:02.013372 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:02.013347    2569 generic.go:356] "Generic (PLEG): container finished" podID="53add21d-9226-4dab-ae83-80ea86ba930b" containerID="034258b763cb0612896b9ea4ba64c7a329a20e4203142b089784adc473f0e490" exitCode=0
Sep 18 17:30:02.013486 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:02.013416    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aa65a-29829210-b8g9l" event={"ID":"53add21d-9226-4dab-ae83-80ea86ba930b","Type":"ContainerDied","Data":"034258b763cb0612896b9ea4ba64c7a329a20e4203142b089784adc473f0e490"}
Sep 18 17:30:02.014918 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:02.014895    2569 generic.go:356] "Generic (PLEG): container finished" podID="e8b79e5e-54f1-4eb5-88ca-3c4f4c8885ae" containerID="530e44366f27cefed9791912efd0cda29a829cc4abb66e8825504f02dc7f9394" exitCode=0
Sep 18 17:30:02.015018 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:02.014954    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-5b6d69-29829210-ln4kp" event={"ID":"e8b79e5e-54f1-4eb5-88ca-3c4f4c8885ae","Type":"ContainerDied","Data":"530e44366f27cefed9791912efd0cda29a829cc4abb66e8825504f02dc7f9394"}
Sep 18 17:30:02.016554 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:02.016532    2569 generic.go:356] "Generic (PLEG): container finished" podID="3e1310bd-d032-45ee-bbc6-7adee2636dad" containerID="a28819e2cc681610d6353e0bac65e65026e7e2e0a9bd3e852f92c52f8c5d1a6d" exitCode=0
Sep 18 17:30:02.016641 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:02.016557    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829210-mn2xl" event={"ID":"3e1310bd-d032-45ee-bbc6-7adee2636dad","Type":"ContainerDied","Data":"a28819e2cc681610d6353e0bac65e65026e7e2e0a9bd3e852f92c52f8c5d1a6d"}
Sep 18 17:30:02.913112 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:02.913078    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9"]
Sep 18 17:30:02.916446 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:02.916423    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9"
Sep 18 17:30:02.918969 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:02.918946    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-5b6d69\"/\"premium-w4-5b6d69-kserve-self-signed-certs\""
Sep 18 17:30:02.925569 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:02.925547    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9"]
Sep 18 17:30:03.073174 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:03.073143    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hggj\" (UniqueName: \"kubernetes.io/projected/718a9c7a-e629-4cb5-aa54-520a737523d8-kube-api-access-8hggj\") pod \"premium-w4-5b6d69-kserve-5485464d4-5j8k9\" (UID: \"718a9c7a-e629-4cb5-aa54-520a737523d8\") " pod="e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9"
Sep 18 17:30:03.073641 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:03.073200    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/718a9c7a-e629-4cb5-aa54-520a737523d8-tls-certs\") pod \"premium-w4-5b6d69-kserve-5485464d4-5j8k9\" (UID: \"718a9c7a-e629-4cb5-aa54-520a737523d8\") " pod="e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9"
Sep 18 17:30:03.073641 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:03.073289    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/718a9c7a-e629-4cb5-aa54-520a737523d8-home\") pod \"premium-w4-5b6d69-kserve-5485464d4-5j8k9\" (UID: \"718a9c7a-e629-4cb5-aa54-520a737523d8\") " pod="e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9"
Sep 18 17:30:03.073641 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:03.073370    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/718a9c7a-e629-4cb5-aa54-520a737523d8-dshm\") pod \"premium-w4-5b6d69-kserve-5485464d4-5j8k9\" (UID: \"718a9c7a-e629-4cb5-aa54-520a737523d8\") " pod="e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9"
Sep 18 17:30:03.073641 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:03.073419    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/718a9c7a-e629-4cb5-aa54-520a737523d8-tmp-dir\") pod \"premium-w4-5b6d69-kserve-5485464d4-5j8k9\" (UID: \"718a9c7a-e629-4cb5-aa54-520a737523d8\") " pod="e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9"
Sep 18 17:30:03.073641 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:03.073450    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/718a9c7a-e629-4cb5-aa54-520a737523d8-model-cache\") pod \"premium-w4-5b6d69-kserve-5485464d4-5j8k9\" (UID: \"718a9c7a-e629-4cb5-aa54-520a737523d8\") " pod="e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9"
Sep 18 17:30:03.170271 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:03.170208    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a2e13-29829210-fpjfn"
Sep 18 17:30:03.175021 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:03.174097    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hggj\" (UniqueName: \"kubernetes.io/projected/718a9c7a-e629-4cb5-aa54-520a737523d8-kube-api-access-8hggj\") pod \"premium-w4-5b6d69-kserve-5485464d4-5j8k9\" (UID: \"718a9c7a-e629-4cb5-aa54-520a737523d8\") " pod="e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9"
Sep 18 17:30:03.175021 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:03.174162    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/718a9c7a-e629-4cb5-aa54-520a737523d8-tls-certs\") pod \"premium-w4-5b6d69-kserve-5485464d4-5j8k9\" (UID: \"718a9c7a-e629-4cb5-aa54-520a737523d8\") " pod="e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9"
Sep 18 17:30:03.175021 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:03.174200    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/718a9c7a-e629-4cb5-aa54-520a737523d8-home\") pod \"premium-w4-5b6d69-kserve-5485464d4-5j8k9\" (UID: \"718a9c7a-e629-4cb5-aa54-520a737523d8\") " pod="e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9"
Sep 18 17:30:03.175021 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:03.174261    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/718a9c7a-e629-4cb5-aa54-520a737523d8-dshm\") pod \"premium-w4-5b6d69-kserve-5485464d4-5j8k9\" (UID: \"718a9c7a-e629-4cb5-aa54-520a737523d8\") " pod="e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9"
Sep 18 17:30:03.175021 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:03.174304    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/718a9c7a-e629-4cb5-aa54-520a737523d8-tmp-dir\") pod \"premium-w4-5b6d69-kserve-5485464d4-5j8k9\" (UID: \"718a9c7a-e629-4cb5-aa54-520a737523d8\") " pod="e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9"
Sep 18 17:30:03.175021 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:03.174332    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/718a9c7a-e629-4cb5-aa54-520a737523d8-model-cache\") pod \"premium-w4-5b6d69-kserve-5485464d4-5j8k9\" (UID: \"718a9c7a-e629-4cb5-aa54-520a737523d8\") " pod="e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9"
Sep 18 17:30:03.175021 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:03.174699    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/718a9c7a-e629-4cb5-aa54-520a737523d8-home\") pod \"premium-w4-5b6d69-kserve-5485464d4-5j8k9\" (UID: \"718a9c7a-e629-4cb5-aa54-520a737523d8\") " pod="e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9"
Sep 18 17:30:03.175021 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:03.174755    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/718a9c7a-e629-4cb5-aa54-520a737523d8-model-cache\") pod \"premium-w4-5b6d69-kserve-5485464d4-5j8k9\" (UID: \"718a9c7a-e629-4cb5-aa54-520a737523d8\") " pod="e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9"
Sep 18 17:30:03.175021 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:03.174981    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/718a9c7a-e629-4cb5-aa54-520a737523d8-tmp-dir\") pod \"premium-w4-5b6d69-kserve-5485464d4-5j8k9\" (UID: \"718a9c7a-e629-4cb5-aa54-520a737523d8\") " pod="e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9"
Sep 18 17:30:03.177208 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:03.177186    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/718a9c7a-e629-4cb5-aa54-520a737523d8-tls-certs\") pod \"premium-w4-5b6d69-kserve-5485464d4-5j8k9\" (UID: \"718a9c7a-e629-4cb5-aa54-520a737523d8\") " pod="e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9"
Sep 18 17:30:03.177815 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:03.177793    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/718a9c7a-e629-4cb5-aa54-520a737523d8-dshm\") pod \"premium-w4-5b6d69-kserve-5485464d4-5j8k9\" (UID: \"718a9c7a-e629-4cb5-aa54-520a737523d8\") " pod="e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9"
Sep 18 17:30:03.182796 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:03.182768    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hggj\" (UniqueName: \"kubernetes.io/projected/718a9c7a-e629-4cb5-aa54-520a737523d8-kube-api-access-8hggj\") pod \"premium-w4-5b6d69-kserve-5485464d4-5j8k9\" (UID: \"718a9c7a-e629-4cb5-aa54-520a737523d8\") " pod="e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9"
Sep 18 17:30:03.211184 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:03.211158    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-5b6d69-29829210-ln4kp"
Sep 18 17:30:03.226769 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:03.226736    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9"
Sep 18 17:30:03.251934 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:03.251908    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aa65a-29829210-b8g9l"
Sep 18 17:30:03.254016 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:03.253993    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829210-mn2xl"
Sep 18 17:30:03.517759 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:03.517726    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9"]
Sep 18 17:30:03.519468 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:30:03.519437    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod718a9c7a_e629_4cb5_aa54_520a737523d8.slice/crio-b706cb3a7ed4980302fa467bd09b07bc1702892398ee5058d978764c63d0990f WatchSource:0}: Error finding container b706cb3a7ed4980302fa467bd09b07bc1702892398ee5058d978764c63d0990f: Status 404 returned error can't find the container with id b706cb3a7ed4980302fa467bd09b07bc1702892398ee5058d978764c63d0990f
Sep 18 17:30:04.028832 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:04.028792    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9" event={"ID":"718a9c7a-e629-4cb5-aa54-520a737523d8","Type":"ContainerStarted","Data":"dddbc5af93ac946b0a79c9abf85e5de9fac5a4030c8d512f602ba833f3e4f744"}
Sep 18 17:30:04.029015 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:04.028849    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9" event={"ID":"718a9c7a-e629-4cb5-aa54-520a737523d8","Type":"ContainerStarted","Data":"b706cb3a7ed4980302fa467bd09b07bc1702892398ee5058d978764c63d0990f"}
Sep 18 17:30:04.030183 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:04.030163    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829210-mn2xl"
Sep 18 17:30:04.030289 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:04.030207    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829210-mn2xl" event={"ID":"3e1310bd-d032-45ee-bbc6-7adee2636dad","Type":"ContainerDied","Data":"30fa141b541fc23802e614f32f46eff3a7819a60f91ff832d9d1e46999a86c60"}
Sep 18 17:30:04.030289 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:04.030249    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30fa141b541fc23802e614f32f46eff3a7819a60f91ff832d9d1e46999a86c60"
Sep 18 17:30:04.031689 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:04.031629    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a2e13-29829210-fpjfn" event={"ID":"63d51e20-2f48-4fbb-ac4e-5ad2377ad376","Type":"ContainerDied","Data":"adc0a93fcaf77f1ef17b04532df550204aac7f1c0e58af07e69d06c82cd9bae5"}
Sep 18 17:30:04.031689 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:04.031640    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a2e13-29829210-fpjfn"
Sep 18 17:30:04.031689 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:04.031662    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adc0a93fcaf77f1ef17b04532df550204aac7f1c0e58af07e69d06c82cd9bae5"
Sep 18 17:30:04.033375 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:04.033356    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aa65a-29829210-b8g9l"
Sep 18 17:30:04.033483 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:04.033385    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aa65a-29829210-b8g9l" event={"ID":"53add21d-9226-4dab-ae83-80ea86ba930b","Type":"ContainerDied","Data":"7b2ba42024855d0548496aacea027fa7ddb0e95bc74dfa4c6083cb6e3b2c6924"}
Sep 18 17:30:04.033483 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:04.033423    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b2ba42024855d0548496aacea027fa7ddb0e95bc74dfa4c6083cb6e3b2c6924"
Sep 18 17:30:04.034882 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:04.034848    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-5b6d69-29829210-ln4kp" event={"ID":"e8b79e5e-54f1-4eb5-88ca-3c4f4c8885ae","Type":"ContainerDied","Data":"e0050181d4979449afc34950d9f9153967eb3ca913b3e8a35977173042eefb1f"}
Sep 18 17:30:04.034978 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:04.034896    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0050181d4979449afc34950d9f9153967eb3ca913b3e8a35977173042eefb1f"
Sep 18 17:30:04.034978 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:04.034853    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-5b6d69-29829210-ln4kp"
Sep 18 17:30:04.049945 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:04.049898    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9" podStartSLOduration=2.049883435 podStartE2EDuration="2.049883435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:30:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:30:04.047940753 +0000 UTC m=+916.741868302" watchObservedRunningTime="2026-09-18 17:30:04.049883435 +0000 UTC m=+916.743810956"
Sep 18 17:30:13.227414 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:13.227373    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9"
Sep 18 17:30:13.227906 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:13.227431    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9"
Sep 18 17:30:13.239832 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:13.239804    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9"
Sep 18 17:30:14.099877 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:14.099834    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9"
Sep 18 17:30:16.709123 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:16.709089    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l"]
Sep 18 17:30:16.709473 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:16.709414    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e1310bd-d032-45ee-bbc6-7adee2636dad" containerName="cleanup"
Sep 18 17:30:16.709473 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:16.709425    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e1310bd-d032-45ee-bbc6-7adee2636dad" containerName="cleanup"
Sep 18 17:30:16.709473 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:16.709440    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53add21d-9226-4dab-ae83-80ea86ba930b" containerName="cleanup"
Sep 18 17:30:16.709473 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:16.709446    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="53add21d-9226-4dab-ae83-80ea86ba930b" containerName="cleanup"
Sep 18 17:30:16.709473 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:16.709460    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8b79e5e-54f1-4eb5-88ca-3c4f4c8885ae" containerName="cleanup"
Sep 18 17:30:16.709473 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:16.709467    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8b79e5e-54f1-4eb5-88ca-3c4f4c8885ae" containerName="cleanup"
Sep 18 17:30:16.709473 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:16.709473    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63d51e20-2f48-4fbb-ac4e-5ad2377ad376" containerName="cleanup"
Sep 18 17:30:16.709701 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:16.709479    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="63d51e20-2f48-4fbb-ac4e-5ad2377ad376" containerName="cleanup"
Sep 18 17:30:16.709701 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:16.709547    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="53add21d-9226-4dab-ae83-80ea86ba930b" containerName="cleanup"
Sep 18 17:30:16.709701 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:16.709556    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="63d51e20-2f48-4fbb-ac4e-5ad2377ad376" containerName="cleanup"
Sep 18 17:30:16.709701 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:16.709562    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e1310bd-d032-45ee-bbc6-7adee2636dad" containerName="cleanup"
Sep 18 17:30:16.709701 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:16.709569    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8b79e5e-54f1-4eb5-88ca-3c4f4c8885ae" containerName="cleanup"
Sep 18 17:30:16.712797 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:16.712778    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l"
Sep 18 17:30:16.715252 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:16.715230    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0a2e13\"/\"premium-w0-0a2e13-kserve-self-signed-certs\""
Sep 18 17:30:16.723202 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:16.723175    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l"]
Sep 18 17:30:16.905368 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:16.905333    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwpm8\" (UniqueName: \"kubernetes.io/projected/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-kube-api-access-zwpm8\") pod \"premium-w0-0a2e13-kserve-58b9c99c94-8xr5l\" (UID: \"0b29ad1f-339b-48f8-bc20-8e51ba37f30c\") " pod="e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l"
Sep 18 17:30:16.905569 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:16.905403    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-model-cache\") pod \"premium-w0-0a2e13-kserve-58b9c99c94-8xr5l\" (UID: \"0b29ad1f-339b-48f8-bc20-8e51ba37f30c\") " pod="e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l"
Sep 18 17:30:16.905569 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:16.905466    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-tmp-dir\") pod \"premium-w0-0a2e13-kserve-58b9c99c94-8xr5l\" (UID: \"0b29ad1f-339b-48f8-bc20-8e51ba37f30c\") " pod="e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l"
Sep 18 17:30:16.905569 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:16.905527    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-home\") pod \"premium-w0-0a2e13-kserve-58b9c99c94-8xr5l\" (UID: \"0b29ad1f-339b-48f8-bc20-8e51ba37f30c\") " pod="e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l"
Sep 18 17:30:16.905569 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:16.905560    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-dshm\") pod \"premium-w0-0a2e13-kserve-58b9c99c94-8xr5l\" (UID: \"0b29ad1f-339b-48f8-bc20-8e51ba37f30c\") " pod="e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l"
Sep 18 17:30:16.905778 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:16.905578    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-tls-certs\") pod \"premium-w0-0a2e13-kserve-58b9c99c94-8xr5l\" (UID: \"0b29ad1f-339b-48f8-bc20-8e51ba37f30c\") " pod="e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l"
Sep 18 17:30:17.006200 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:17.006171    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwpm8\" (UniqueName: \"kubernetes.io/projected/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-kube-api-access-zwpm8\") pod \"premium-w0-0a2e13-kserve-58b9c99c94-8xr5l\" (UID: \"0b29ad1f-339b-48f8-bc20-8e51ba37f30c\") " pod="e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l"
Sep 18 17:30:17.006398 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:17.006242    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-model-cache\") pod \"premium-w0-0a2e13-kserve-58b9c99c94-8xr5l\" (UID: \"0b29ad1f-339b-48f8-bc20-8e51ba37f30c\") " pod="e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l"
Sep 18 17:30:17.006398 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:17.006273    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-tmp-dir\") pod \"premium-w0-0a2e13-kserve-58b9c99c94-8xr5l\" (UID: \"0b29ad1f-339b-48f8-bc20-8e51ba37f30c\") " pod="e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l"
Sep 18 17:30:17.006398 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:17.006313    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-home\") pod \"premium-w0-0a2e13-kserve-58b9c99c94-8xr5l\" (UID: \"0b29ad1f-339b-48f8-bc20-8e51ba37f30c\") " pod="e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l"
Sep 18 17:30:17.006398 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:17.006339    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-dshm\") pod \"premium-w0-0a2e13-kserve-58b9c99c94-8xr5l\" (UID: \"0b29ad1f-339b-48f8-bc20-8e51ba37f30c\") " pod="e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l"
Sep 18 17:30:17.006398 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:17.006359    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-tls-certs\") pod \"premium-w0-0a2e13-kserve-58b9c99c94-8xr5l\" (UID: \"0b29ad1f-339b-48f8-bc20-8e51ba37f30c\") " pod="e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l"
Sep 18 17:30:17.006714 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:17.006691    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-model-cache\") pod \"premium-w0-0a2e13-kserve-58b9c99c94-8xr5l\" (UID: \"0b29ad1f-339b-48f8-bc20-8e51ba37f30c\") " pod="e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l"
Sep 18 17:30:17.006763 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:17.006742    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-tmp-dir\") pod \"premium-w0-0a2e13-kserve-58b9c99c94-8xr5l\" (UID: \"0b29ad1f-339b-48f8-bc20-8e51ba37f30c\") " pod="e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l"
Sep 18 17:30:17.006816 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:17.006800    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-home\") pod \"premium-w0-0a2e13-kserve-58b9c99c94-8xr5l\" (UID: \"0b29ad1f-339b-48f8-bc20-8e51ba37f30c\") " pod="e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l"
Sep 18 17:30:17.008630 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:17.008603    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-dshm\") pod \"premium-w0-0a2e13-kserve-58b9c99c94-8xr5l\" (UID: \"0b29ad1f-339b-48f8-bc20-8e51ba37f30c\") " pod="e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l"
Sep 18 17:30:17.008793 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:17.008777    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-tls-certs\") pod \"premium-w0-0a2e13-kserve-58b9c99c94-8xr5l\" (UID: \"0b29ad1f-339b-48f8-bc20-8e51ba37f30c\") " pod="e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l"
Sep 18 17:30:17.025022 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:17.024997    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwpm8\" (UniqueName: \"kubernetes.io/projected/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-kube-api-access-zwpm8\") pod \"premium-w0-0a2e13-kserve-58b9c99c94-8xr5l\" (UID: \"0b29ad1f-339b-48f8-bc20-8e51ba37f30c\") " pod="e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l"
Sep 18 17:30:17.322993 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:17.322909    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l"
Sep 18 17:30:17.625965 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:17.625938    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l"]
Sep 18 17:30:17.627852 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:30:17.627800    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b29ad1f_339b_48f8_bc20_8e51ba37f30c.slice/crio-254a79edce1904920931dcf484ca4e2a1e865b90ff2793121fdcd683daa32988 WatchSource:0}: Error finding container 254a79edce1904920931dcf484ca4e2a1e865b90ff2793121fdcd683daa32988: Status 404 returned error can't find the container with id 254a79edce1904920931dcf484ca4e2a1e865b90ff2793121fdcd683daa32988
Sep 18 17:30:18.111088 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:18.111050    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l" event={"ID":"0b29ad1f-339b-48f8-bc20-8e51ba37f30c","Type":"ContainerStarted","Data":"2f07db007f1fa6f9befdd2f740a9cd7b1678e09a59f5c1dca41e8a0c3867938b"}
Sep 18 17:30:18.111488 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:18.111098    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l" event={"ID":"0b29ad1f-339b-48f8-bc20-8e51ba37f30c","Type":"ContainerStarted","Data":"254a79edce1904920931dcf484ca4e2a1e865b90ff2793121fdcd683daa32988"}
Sep 18 17:30:25.073480 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:25.073420    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l" podStartSLOduration=9.073401033 podStartE2EDuration="9.073401033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:30:18.134409691 +0000 UTC m=+930.828337231" watchObservedRunningTime="2026-09-18 17:30:25.073401033 +0000 UTC m=+937.767328560"
Sep 18 17:30:25.074532 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:25.074513    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-97798a-cff5c65cc-bthtz"]
Sep 18 17:30:25.077801 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:25.077784    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-97798a-cff5c65cc-bthtz"
Sep 18 17:30:25.085934 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:25.085911    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 17:30:25.086615 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:25.086592    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 18 17:30:25.086936 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:25.086918    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 17:30:25.094141 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:25.094120    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-97798a-serving-cert\""
Sep 18 17:30:25.095229 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:25.095210    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sjtq4\""
Sep 18 17:30:25.118431 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:25.118402    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-97798a-cff5c65cc-bthtz"]
Sep 18 17:30:25.178877 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:25.178833    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e32f5a1-a455-413e-8d51-52565ce0df1e-metrics-certs\") pod \"maas-api-e2e-explicit-97798a-cff5c65cc-bthtz\" (UID: \"4e32f5a1-a455-413e-8d51-52565ce0df1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-97798a-cff5c65cc-bthtz"
Sep 18 17:30:25.179053 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:25.178912    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkqvq\" (UniqueName: \"kubernetes.io/projected/4e32f5a1-a455-413e-8d51-52565ce0df1e-kube-api-access-pkqvq\") pod \"maas-api-e2e-explicit-97798a-cff5c65cc-bthtz\" (UID: \"4e32f5a1-a455-413e-8d51-52565ce0df1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-97798a-cff5c65cc-bthtz"
Sep 18 17:30:25.179053 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:25.178944    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e32f5a1-a455-413e-8d51-52565ce0df1e-openshift-service-ca\") pod \"maas-api-e2e-explicit-97798a-cff5c65cc-bthtz\" (UID: \"4e32f5a1-a455-413e-8d51-52565ce0df1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-97798a-cff5c65cc-bthtz"
Sep 18 17:30:25.179053 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:25.179019    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e32f5a1-a455-413e-8d51-52565ce0df1e-maas-api-tls\") pod \"maas-api-e2e-explicit-97798a-cff5c65cc-bthtz\" (UID: \"4e32f5a1-a455-413e-8d51-52565ce0df1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-97798a-cff5c65cc-bthtz"
Sep 18 17:30:25.279921 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:25.279889    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e32f5a1-a455-413e-8d51-52565ce0df1e-metrics-certs\") pod \"maas-api-e2e-explicit-97798a-cff5c65cc-bthtz\" (UID: \"4e32f5a1-a455-413e-8d51-52565ce0df1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-97798a-cff5c65cc-bthtz"
Sep 18 17:30:25.280086 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:25.279927    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkqvq\" (UniqueName: \"kubernetes.io/projected/4e32f5a1-a455-413e-8d51-52565ce0df1e-kube-api-access-pkqvq\") pod \"maas-api-e2e-explicit-97798a-cff5c65cc-bthtz\" (UID: \"4e32f5a1-a455-413e-8d51-52565ce0df1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-97798a-cff5c65cc-bthtz"
Sep 18 17:30:25.280086 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:25.279955    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e32f5a1-a455-413e-8d51-52565ce0df1e-openshift-service-ca\") pod \"maas-api-e2e-explicit-97798a-cff5c65cc-bthtz\" (UID: \"4e32f5a1-a455-413e-8d51-52565ce0df1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-97798a-cff5c65cc-bthtz"
Sep 18 17:30:25.280086 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:25.280011    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e32f5a1-a455-413e-8d51-52565ce0df1e-maas-api-tls\") pod \"maas-api-e2e-explicit-97798a-cff5c65cc-bthtz\" (UID: \"4e32f5a1-a455-413e-8d51-52565ce0df1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-97798a-cff5c65cc-bthtz"
Sep 18 17:30:25.280196 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:30:25.280159    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-97798a-serving-cert: secret "maas-api-e2e-explicit-97798a-serving-cert" not found
Sep 18 17:30:25.280242 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:30:25.280231    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e32f5a1-a455-413e-8d51-52565ce0df1e-maas-api-tls podName:4e32f5a1-a455-413e-8d51-52565ce0df1e nodeName:}" failed. No retries permitted until 2026-09-18 17:30:25.780212365 +0000 UTC m=+938.474139887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4e32f5a1-a455-413e-8d51-52565ce0df1e-maas-api-tls") pod "maas-api-e2e-explicit-97798a-cff5c65cc-bthtz" (UID: "4e32f5a1-a455-413e-8d51-52565ce0df1e") : secret "maas-api-e2e-explicit-97798a-serving-cert" not found
Sep 18 17:30:25.280659 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:25.280633    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e32f5a1-a455-413e-8d51-52565ce0df1e-openshift-service-ca\") pod \"maas-api-e2e-explicit-97798a-cff5c65cc-bthtz\" (UID: \"4e32f5a1-a455-413e-8d51-52565ce0df1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-97798a-cff5c65cc-bthtz"
Sep 18 17:30:25.282250 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:25.282224    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e32f5a1-a455-413e-8d51-52565ce0df1e-metrics-certs\") pod \"maas-api-e2e-explicit-97798a-cff5c65cc-bthtz\" (UID: \"4e32f5a1-a455-413e-8d51-52565ce0df1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-97798a-cff5c65cc-bthtz"
Sep 18 17:30:25.288579 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:25.288554    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkqvq\" (UniqueName: \"kubernetes.io/projected/4e32f5a1-a455-413e-8d51-52565ce0df1e-kube-api-access-pkqvq\") pod \"maas-api-e2e-explicit-97798a-cff5c65cc-bthtz\" (UID: \"4e32f5a1-a455-413e-8d51-52565ce0df1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-97798a-cff5c65cc-bthtz"
Sep 18 17:30:25.785805 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:25.785773    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e32f5a1-a455-413e-8d51-52565ce0df1e-maas-api-tls\") pod \"maas-api-e2e-explicit-97798a-cff5c65cc-bthtz\" (UID: \"4e32f5a1-a455-413e-8d51-52565ce0df1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-97798a-cff5c65cc-bthtz"
Sep 18 17:30:25.788349 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:25.788318    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e32f5a1-a455-413e-8d51-52565ce0df1e-maas-api-tls\") pod \"maas-api-e2e-explicit-97798a-cff5c65cc-bthtz\" (UID: \"4e32f5a1-a455-413e-8d51-52565ce0df1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-97798a-cff5c65cc-bthtz"
Sep 18 17:30:25.988669 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:25.988637    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-97798a-cff5c65cc-bthtz"
Sep 18 17:30:26.061819 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:26.061724    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-97798a-5755fcf6c7-c7qmd"]
Sep 18 17:30:26.067115 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:26.067091    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-97798a-5755fcf6c7-c7qmd"
Sep 18 17:30:26.069958 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:26.069932    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-97798a-dockercfg-b8m75\""
Sep 18 17:30:26.070105 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:26.070013    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-97798a\""
Sep 18 17:30:26.079540 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:26.079517    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-97798a-5755fcf6c7-c7qmd"]
Sep 18 17:30:26.088236 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:26.088213    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cbad065-e91d-4b19-a7c1-2857773e3d22-plugins-config-volume\") pod \"payload-processing-e2e-explicit-97798a-5755fcf6c7-c7qmd\" (UID: \"7cbad065-e91d-4b19-a7c1-2857773e3d22\") " pod="openshift-ingress/payload-processing-e2e-explicit-97798a-5755fcf6c7-c7qmd"
Sep 18 17:30:26.088343 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:26.088268    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rtml\" (UniqueName: \"kubernetes.io/projected/7cbad065-e91d-4b19-a7c1-2857773e3d22-kube-api-access-9rtml\") pod \"payload-processing-e2e-explicit-97798a-5755fcf6c7-c7qmd\" (UID: \"7cbad065-e91d-4b19-a7c1-2857773e3d22\") " pod="openshift-ingress/payload-processing-e2e-explicit-97798a-5755fcf6c7-c7qmd"
Sep 18 17:30:26.188980 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:26.188950    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cbad065-e91d-4b19-a7c1-2857773e3d22-plugins-config-volume\") pod \"payload-processing-e2e-explicit-97798a-5755fcf6c7-c7qmd\" (UID: \"7cbad065-e91d-4b19-a7c1-2857773e3d22\") " pod="openshift-ingress/payload-processing-e2e-explicit-97798a-5755fcf6c7-c7qmd"
Sep 18 17:30:26.189095 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:26.189007    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rtml\" (UniqueName: \"kubernetes.io/projected/7cbad065-e91d-4b19-a7c1-2857773e3d22-kube-api-access-9rtml\") pod \"payload-processing-e2e-explicit-97798a-5755fcf6c7-c7qmd\" (UID: \"7cbad065-e91d-4b19-a7c1-2857773e3d22\") " pod="openshift-ingress/payload-processing-e2e-explicit-97798a-5755fcf6c7-c7qmd"
Sep 18 17:30:26.189650 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:26.189628    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cbad065-e91d-4b19-a7c1-2857773e3d22-plugins-config-volume\") pod \"payload-processing-e2e-explicit-97798a-5755fcf6c7-c7qmd\" (UID: \"7cbad065-e91d-4b19-a7c1-2857773e3d22\") " pod="openshift-ingress/payload-processing-e2e-explicit-97798a-5755fcf6c7-c7qmd"
Sep 18 17:30:26.197927 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:26.197893    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rtml\" (UniqueName: \"kubernetes.io/projected/7cbad065-e91d-4b19-a7c1-2857773e3d22-kube-api-access-9rtml\") pod \"payload-processing-e2e-explicit-97798a-5755fcf6c7-c7qmd\" (UID: \"7cbad065-e91d-4b19-a7c1-2857773e3d22\") " pod="openshift-ingress/payload-processing-e2e-explicit-97798a-5755fcf6c7-c7qmd"
Sep 18 17:30:26.261666 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:26.261638    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-97798a-cff5c65cc-bthtz"]
Sep 18 17:30:26.263578 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:30:26.263553    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e32f5a1_a455_413e_8d51_52565ce0df1e.slice/crio-7c512b02710456ddacc1f7a38bf909fd352ca0cf6562d3fc5c920810f84842d5 WatchSource:0}: Error finding container 7c512b02710456ddacc1f7a38bf909fd352ca0cf6562d3fc5c920810f84842d5: Status 404 returned error can't find the container with id 7c512b02710456ddacc1f7a38bf909fd352ca0cf6562d3fc5c920810f84842d5
Sep 18 17:30:26.265033 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:26.265011    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 17:30:26.378565 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:26.378484    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-97798a-5755fcf6c7-c7qmd"
Sep 18 17:30:26.677837 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:26.677809    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-97798a-5755fcf6c7-c7qmd"]
Sep 18 17:30:26.679807 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:30:26.679780    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cbad065_e91d_4b19_a7c1_2857773e3d22.slice/crio-bc1b75209d09895d39e768a47364761c20dcba00480db98697c41da52f61dd7d WatchSource:0}: Error finding container bc1b75209d09895d39e768a47364761c20dcba00480db98697c41da52f61dd7d: Status 404 returned error can't find the container with id bc1b75209d09895d39e768a47364761c20dcba00480db98697c41da52f61dd7d
Sep 18 17:30:27.161245 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:27.161201    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-97798a-cff5c65cc-bthtz" event={"ID":"4e32f5a1-a455-413e-8d51-52565ce0df1e","Type":"ContainerStarted","Data":"705a4f480b4a1dcb8540868fa4a85c593131b788ee6cef2415d4e28e630a0746"}
Sep 18 17:30:27.161694 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:27.161275    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-97798a-cff5c65cc-bthtz"
Sep 18 17:30:27.161694 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:27.161307    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-97798a-cff5c65cc-bthtz" event={"ID":"4e32f5a1-a455-413e-8d51-52565ce0df1e","Type":"ContainerStarted","Data":"7c512b02710456ddacc1f7a38bf909fd352ca0cf6562d3fc5c920810f84842d5"}
Sep 18 17:30:27.162960 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:27.162930    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-97798a-5755fcf6c7-c7qmd" event={"ID":"7cbad065-e91d-4b19-a7c1-2857773e3d22","Type":"ContainerStarted","Data":"6af6f570c4974adff50ec97972e1f2c04f974a6a350961635563edeffd78a0f3"}
Sep 18 17:30:27.163094 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:27.162981    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-97798a-5755fcf6c7-c7qmd" event={"ID":"7cbad065-e91d-4b19-a7c1-2857773e3d22","Type":"ContainerStarted","Data":"bc1b75209d09895d39e768a47364761c20dcba00480db98697c41da52f61dd7d"}
Sep 18 17:30:27.163094 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:27.163091    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-97798a-5755fcf6c7-c7qmd"
Sep 18 17:30:27.203251 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:27.203203    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-97798a-cff5c65cc-bthtz" podStartSLOduration=1.7212675100000001 podStartE2EDuration="2.20318685s" totalImagesPullingTime="481.91934ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:30:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:30:27.200853372 +0000 UTC m=+939.894780912" watchObservedRunningTime="2026-09-18 17:30:27.20318685 +0000 UTC m=+939.897114528"
Sep 18 17:30:27.323974 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:27.323934    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l"
Sep 18 17:30:27.324157 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:27.323988    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l"
Sep 18 17:30:27.336535 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:27.336511    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l"
Sep 18 17:30:27.373846 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:27.373791    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-97798a-5755fcf6c7-c7qmd" podStartSLOduration=1.373775616 podStartE2EDuration="1.373775616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:30:27.237606397 +0000 UTC m=+939.931533918" watchObservedRunningTime="2026-09-18 17:30:27.373775616 +0000 UTC m=+940.067703143"
Sep 18 17:30:28.179090 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:28.179062    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l"
Sep 18 17:30:33.173174 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:33.173146    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-97798a-cff5c65cc-bthtz"
Sep 18 17:30:38.170448 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:30:38.170415    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-97798a-5755fcf6c7-c7qmd"
Sep 18 17:31:33.051429 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:33.051394    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-55a2c1b6-556fc86789-5hx2n"]
Sep 18 17:31:33.055035 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:33.055018    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-55a2c1b6-556fc86789-5hx2n"
Sep 18 17:31:33.057524 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:33.057502    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-55a2c1b6\""
Sep 18 17:31:33.057672 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:33.057600    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-55a2c1b6-dockercfg-pfdns\""
Sep 18 17:31:33.065128 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:33.065100    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-55a2c1b6-556fc86789-5hx2n"]
Sep 18 17:31:33.223817 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:33.223782    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/797b324e-173a-45ab-af25-14f04abfda25-plugins-config-volume\") pod \"payload-processing-e2e-mt-55a2c1b6-556fc86789-5hx2n\" (UID: \"797b324e-173a-45ab-af25-14f04abfda25\") " pod="openshift-ingress/payload-processing-e2e-mt-55a2c1b6-556fc86789-5hx2n"
Sep 18 17:31:33.223992 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:33.223933    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z8kq\" (UniqueName: \"kubernetes.io/projected/797b324e-173a-45ab-af25-14f04abfda25-kube-api-access-8z8kq\") pod \"payload-processing-e2e-mt-55a2c1b6-556fc86789-5hx2n\" (UID: \"797b324e-173a-45ab-af25-14f04abfda25\") " pod="openshift-ingress/payload-processing-e2e-mt-55a2c1b6-556fc86789-5hx2n"
Sep 18 17:31:33.324777 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:33.324690    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/797b324e-173a-45ab-af25-14f04abfda25-plugins-config-volume\") pod \"payload-processing-e2e-mt-55a2c1b6-556fc86789-5hx2n\" (UID: \"797b324e-173a-45ab-af25-14f04abfda25\") " pod="openshift-ingress/payload-processing-e2e-mt-55a2c1b6-556fc86789-5hx2n"
Sep 18 17:31:33.324956 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:33.324781    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z8kq\" (UniqueName: \"kubernetes.io/projected/797b324e-173a-45ab-af25-14f04abfda25-kube-api-access-8z8kq\") pod \"payload-processing-e2e-mt-55a2c1b6-556fc86789-5hx2n\" (UID: \"797b324e-173a-45ab-af25-14f04abfda25\") " pod="openshift-ingress/payload-processing-e2e-mt-55a2c1b6-556fc86789-5hx2n"
Sep 18 17:31:33.325420 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:33.325399    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/797b324e-173a-45ab-af25-14f04abfda25-plugins-config-volume\") pod \"payload-processing-e2e-mt-55a2c1b6-556fc86789-5hx2n\" (UID: \"797b324e-173a-45ab-af25-14f04abfda25\") " pod="openshift-ingress/payload-processing-e2e-mt-55a2c1b6-556fc86789-5hx2n"
Sep 18 17:31:33.333809 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:33.333774    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z8kq\" (UniqueName: \"kubernetes.io/projected/797b324e-173a-45ab-af25-14f04abfda25-kube-api-access-8z8kq\") pod \"payload-processing-e2e-mt-55a2c1b6-556fc86789-5hx2n\" (UID: \"797b324e-173a-45ab-af25-14f04abfda25\") " pod="openshift-ingress/payload-processing-e2e-mt-55a2c1b6-556fc86789-5hx2n"
Sep 18 17:31:33.365747 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:33.365720    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-55a2c1b6-556fc86789-5hx2n"
Sep 18 17:31:33.639180 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:33.639154    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-55a2c1b6-556fc86789-5hx2n"]
Sep 18 17:31:33.641396 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:31:33.641367    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod797b324e_173a_45ab_af25_14f04abfda25.slice/crio-75e486437aff40bb81f66ce4d7c52fdb8f1d40589d2f362b69c6bd702d6c4aba WatchSource:0}: Error finding container 75e486437aff40bb81f66ce4d7c52fdb8f1d40589d2f362b69c6bd702d6c4aba: Status 404 returned error can't find the container with id 75e486437aff40bb81f66ce4d7c52fdb8f1d40589d2f362b69c6bd702d6c4aba
Sep 18 17:31:34.521989 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:34.521950    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-55a2c1b6-556fc86789-5hx2n" event={"ID":"797b324e-173a-45ab-af25-14f04abfda25","Type":"ContainerStarted","Data":"b4cf56b783de2370a85b3a0923afd03338a935a9cf249716289bcaf6bd554aa6"}
Sep 18 17:31:34.522444 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:34.522017    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-55a2c1b6-556fc86789-5hx2n" event={"ID":"797b324e-173a-45ab-af25-14f04abfda25","Type":"ContainerStarted","Data":"75e486437aff40bb81f66ce4d7c52fdb8f1d40589d2f362b69c6bd702d6c4aba"}
Sep 18 17:31:34.522444 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:34.522154    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-55a2c1b6-556fc86789-5hx2n"
Sep 18 17:31:34.541786 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:34.541741    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-55a2c1b6-556fc86789-5hx2n" podStartSLOduration=1.541727517 podStartE2EDuration="1.541727517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:31:34.539569434 +0000 UTC m=+1007.233496977" watchObservedRunningTime="2026-09-18 17:31:34.541727517 +0000 UTC m=+1007.235655070"
Sep 18 17:31:43.216785 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:43.216753    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv"]
Sep 18 17:31:43.220249 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:43.220232    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv"
Sep 18 17:31:43.222930 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:43.222909    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-97798a\"/\"openshift-service-ca.crt\""
Sep 18 17:31:43.223693 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:43.223674    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-97798a\"/\"kube-root-ca.crt\""
Sep 18 17:31:43.223785 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:43.223677    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-97798a\"/\"default-dockercfg-64p26\""
Sep 18 17:31:43.223848 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:43.223677    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-97798a\"/\"explicit-model-97798a-kserve-self-signed-certs\""
Sep 18 17:31:43.235215 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:43.235193    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv"]
Sep 18 17:31:43.315211 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:43.315164    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-dshm\") pod \"explicit-model-97798a-kserve-6d6d9764f5-wpdqv\" (UID: \"58aeb0b0-d9f4-412c-b669-5a8a6331c9b4\") " pod="ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv"
Sep 18 17:31:43.315402 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:43.315230    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-tls-certs\") pod \"explicit-model-97798a-kserve-6d6d9764f5-wpdqv\" (UID: \"58aeb0b0-d9f4-412c-b669-5a8a6331c9b4\") " pod="ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv"
Sep 18 17:31:43.315402 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:43.315271    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-home\") pod \"explicit-model-97798a-kserve-6d6d9764f5-wpdqv\" (UID: \"58aeb0b0-d9f4-412c-b669-5a8a6331c9b4\") " pod="ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv"
Sep 18 17:31:43.315402 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:43.315320    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-tmp-dir\") pod \"explicit-model-97798a-kserve-6d6d9764f5-wpdqv\" (UID: \"58aeb0b0-d9f4-412c-b669-5a8a6331c9b4\") " pod="ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv"
Sep 18 17:31:43.315402 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:43.315369    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-model-cache\") pod \"explicit-model-97798a-kserve-6d6d9764f5-wpdqv\" (UID: \"58aeb0b0-d9f4-412c-b669-5a8a6331c9b4\") " pod="ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv"
Sep 18 17:31:43.315402 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:43.315390    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9qh4\" (UniqueName: \"kubernetes.io/projected/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-kube-api-access-x9qh4\") pod \"explicit-model-97798a-kserve-6d6d9764f5-wpdqv\" (UID: \"58aeb0b0-d9f4-412c-b669-5a8a6331c9b4\") " pod="ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv"
Sep 18 17:31:43.416629 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:43.416597    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-home\") pod \"explicit-model-97798a-kserve-6d6d9764f5-wpdqv\" (UID: \"58aeb0b0-d9f4-412c-b669-5a8a6331c9b4\") " pod="ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv"
Sep 18 17:31:43.416840 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:43.416666    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-tmp-dir\") pod \"explicit-model-97798a-kserve-6d6d9764f5-wpdqv\" (UID: \"58aeb0b0-d9f4-412c-b669-5a8a6331c9b4\") " pod="ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv"
Sep 18 17:31:43.416840 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:43.416740    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-model-cache\") pod \"explicit-model-97798a-kserve-6d6d9764f5-wpdqv\" (UID: \"58aeb0b0-d9f4-412c-b669-5a8a6331c9b4\") " pod="ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv"
Sep 18 17:31:43.416840 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:43.416769    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9qh4\" (UniqueName: \"kubernetes.io/projected/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-kube-api-access-x9qh4\") pod \"explicit-model-97798a-kserve-6d6d9764f5-wpdqv\" (UID: \"58aeb0b0-d9f4-412c-b669-5a8a6331c9b4\") " pod="ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv"
Sep 18 17:31:43.416840 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:43.416822    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-dshm\") pod \"explicit-model-97798a-kserve-6d6d9764f5-wpdqv\" (UID: \"58aeb0b0-d9f4-412c-b669-5a8a6331c9b4\") " pod="ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv"
Sep 18 17:31:43.417097 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:43.416876    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-tls-certs\") pod \"explicit-model-97798a-kserve-6d6d9764f5-wpdqv\" (UID: \"58aeb0b0-d9f4-412c-b669-5a8a6331c9b4\") " pod="ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv"
Sep 18 17:31:43.417097 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:43.417059    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-home\") pod \"explicit-model-97798a-kserve-6d6d9764f5-wpdqv\" (UID: \"58aeb0b0-d9f4-412c-b669-5a8a6331c9b4\") " pod="ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv"
Sep 18 17:31:43.417205 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:43.417118    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-tmp-dir\") pod \"explicit-model-97798a-kserve-6d6d9764f5-wpdqv\" (UID: \"58aeb0b0-d9f4-412c-b669-5a8a6331c9b4\") " pod="ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv"
Sep 18 17:31:43.417205 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:43.417141    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-model-cache\") pod \"explicit-model-97798a-kserve-6d6d9764f5-wpdqv\" (UID: \"58aeb0b0-d9f4-412c-b669-5a8a6331c9b4\") " pod="ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv"
Sep 18 17:31:43.419014 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:43.418987    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-dshm\") pod \"explicit-model-97798a-kserve-6d6d9764f5-wpdqv\" (UID: \"58aeb0b0-d9f4-412c-b669-5a8a6331c9b4\") " pod="ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv"
Sep 18 17:31:43.419388 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:43.419367    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-tls-certs\") pod \"explicit-model-97798a-kserve-6d6d9764f5-wpdqv\" (UID: \"58aeb0b0-d9f4-412c-b669-5a8a6331c9b4\") " pod="ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv"
Sep 18 17:31:43.425908 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:43.425856    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9qh4\" (UniqueName: \"kubernetes.io/projected/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-kube-api-access-x9qh4\") pod \"explicit-model-97798a-kserve-6d6d9764f5-wpdqv\" (UID: \"58aeb0b0-d9f4-412c-b669-5a8a6331c9b4\") " pod="ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv"
Sep 18 17:31:43.529429 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:43.529399    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv"
Sep 18 17:31:43.801941 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:43.801915    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv"]
Sep 18 17:31:43.803933 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:31:43.803905    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58aeb0b0_d9f4_412c_b669_5a8a6331c9b4.slice/crio-653166709b16f0ccf00aff9b8e97b425e4742c6a27636d046bc5c687c7153f91 WatchSource:0}: Error finding container 653166709b16f0ccf00aff9b8e97b425e4742c6a27636d046bc5c687c7153f91: Status 404 returned error can't find the container with id 653166709b16f0ccf00aff9b8e97b425e4742c6a27636d046bc5c687c7153f91
Sep 18 17:31:44.579646 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:44.579614    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv" event={"ID":"58aeb0b0-d9f4-412c-b669-5a8a6331c9b4","Type":"ContainerStarted","Data":"00528e2f37a6d7ba2e8ce3bc4bea941659360f102286f9b4f4e5ca52ccfc3d3b"}
Sep 18 17:31:44.580063 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:44.579654    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv" event={"ID":"58aeb0b0-d9f4-412c-b669-5a8a6331c9b4","Type":"ContainerStarted","Data":"653166709b16f0ccf00aff9b8e97b425e4742c6a27636d046bc5c687c7153f91"}
Sep 18 17:31:44.598155 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:44.598107    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv" podStartSLOduration=1.598091761 podStartE2EDuration="1.598091761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:31:44.597575687 +0000 UTC m=+1017.291503239" watchObservedRunningTime="2026-09-18 17:31:44.598091761 +0000 UTC m=+1017.292019289"
Sep 18 17:31:45.529034 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:45.529003    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-55a2c1b6-556fc86789-5hx2n"
Sep 18 17:31:53.529780 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:53.529739    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv"
Sep 18 17:31:53.530211 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:53.529797    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv"
Sep 18 17:31:53.542742 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:53.542709    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv"
Sep 18 17:31:53.645570 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:31:53.645538    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv"
Sep 18 17:32:03.720267 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:03.720229    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv"]
Sep 18 17:32:03.834756 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:03.834713    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-97798a-cbxd2"]
Sep 18 17:32:03.838566 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:03.838548    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-97798a-cbxd2"
Sep 18 17:32:03.841166 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:03.841146    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qt978\""
Sep 18 17:32:03.846503 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:03.846439    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-97798a-cbxd2"]
Sep 18 17:32:03.912626 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:03.912586    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eba86cb-56ad-4b84-a667-7979fc9c0614-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-97798a-cbxd2\" (UID: \"2eba86cb-56ad-4b84-a667-7979fc9c0614\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-97798a-cbxd2"
Sep 18 17:32:04.013858 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:04.013809    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eba86cb-56ad-4b84-a667-7979fc9c0614-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-97798a-cbxd2\" (UID: \"2eba86cb-56ad-4b84-a667-7979fc9c0614\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-97798a-cbxd2"
Sep 18 17:32:04.014456 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:04.014437    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eba86cb-56ad-4b84-a667-7979fc9c0614-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-97798a-cbxd2\" (UID: \"2eba86cb-56ad-4b84-a667-7979fc9c0614\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-97798a-cbxd2"
Sep 18 17:32:04.150528 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:04.150491    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-97798a-cbxd2"
Sep 18 17:32:04.420030 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:04.420005    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-97798a-cbxd2"]
Sep 18 17:32:04.422365 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:32:04.422328    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eba86cb_56ad_4b84_a667_7979fc9c0614.slice/crio-250c20854b0102167997023111e4c8125fb33f32bde6b7441339afd9bc8d11ac WatchSource:0}: Error finding container 250c20854b0102167997023111e4c8125fb33f32bde6b7441339afd9bc8d11ac: Status 404 returned error can't find the container with id 250c20854b0102167997023111e4c8125fb33f32bde6b7441339afd9bc8d11ac
Sep 18 17:32:04.696718 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:04.696645    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-97798a-cbxd2" event={"ID":"2eba86cb-56ad-4b84-a667-7979fc9c0614","Type":"ContainerStarted","Data":"250c20854b0102167997023111e4c8125fb33f32bde6b7441339afd9bc8d11ac"}
Sep 18 17:32:05.076325 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:05.076280    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55a2c1b6-cs76n"]
Sep 18 17:32:05.080265 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:05.080239    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55a2c1b6-cs76n"
Sep 18 17:32:05.088766 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:05.088737    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55a2c1b6-cs76n"]
Sep 18 17:32:05.226755 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:05.226702    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0c97202-9fa0-4e88-b485-5797a89ce1a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-55a2c1b6-cs76n\" (UID: \"d0c97202-9fa0-4e88-b485-5797a89ce1a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55a2c1b6-cs76n"
Sep 18 17:32:05.328357 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:05.328275    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0c97202-9fa0-4e88-b485-5797a89ce1a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-55a2c1b6-cs76n\" (UID: \"d0c97202-9fa0-4e88-b485-5797a89ce1a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55a2c1b6-cs76n"
Sep 18 17:32:05.328857 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:05.328838    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0c97202-9fa0-4e88-b485-5797a89ce1a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-55a2c1b6-cs76n\" (UID: \"d0c97202-9fa0-4e88-b485-5797a89ce1a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55a2c1b6-cs76n"
Sep 18 17:32:05.392387 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:05.392350    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55a2c1b6-cs76n"
Sep 18 17:32:05.669558 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:05.669535    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55a2c1b6-cs76n"]
Sep 18 17:32:05.671357 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:32:05.671327    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c97202_9fa0_4e88_b485_5797a89ce1a8.slice/crio-c945ebe38b4dd3fb4db2490837840cec5d0892472f1444a55e4a3dc2fbca5cc0 WatchSource:0}: Error finding container c945ebe38b4dd3fb4db2490837840cec5d0892472f1444a55e4a3dc2fbca5cc0: Status 404 returned error can't find the container with id c945ebe38b4dd3fb4db2490837840cec5d0892472f1444a55e4a3dc2fbca5cc0
Sep 18 17:32:05.703098 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:05.703070    2569 generic.go:356] "Generic (PLEG): container finished" podID="2eba86cb-56ad-4b84-a667-7979fc9c0614" containerID="d6d61dbc8fe39315137ef5e87735ffe9a1e1944ebf5ecc40c1ed336b6a15aab0" exitCode=0
Sep 18 17:32:05.703195 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:05.703163    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-97798a-cbxd2" event={"ID":"2eba86cb-56ad-4b84-a667-7979fc9c0614","Type":"ContainerDied","Data":"d6d61dbc8fe39315137ef5e87735ffe9a1e1944ebf5ecc40c1ed336b6a15aab0"}
Sep 18 17:32:05.704354 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:05.704335    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55a2c1b6-cs76n" event={"ID":"d0c97202-9fa0-4e88-b485-5797a89ce1a8","Type":"ContainerStarted","Data":"c945ebe38b4dd3fb4db2490837840cec5d0892472f1444a55e4a3dc2fbca5cc0"}
Sep 18 17:32:06.711431 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:06.711342    2569 generic.go:356] "Generic (PLEG): container finished" podID="d0c97202-9fa0-4e88-b485-5797a89ce1a8" containerID="cc4b2a885d1f76aa21a6334aef6d2ed1523aed477f79300e93b7aeb2297b6f8a" exitCode=0
Sep 18 17:32:06.711896 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:06.711430    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55a2c1b6-cs76n" event={"ID":"d0c97202-9fa0-4e88-b485-5797a89ce1a8","Type":"ContainerDied","Data":"cc4b2a885d1f76aa21a6334aef6d2ed1523aed477f79300e93b7aeb2297b6f8a"}
Sep 18 17:32:06.847051 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:06.847019    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-97798a-cbxd2"
Sep 18 17:32:06.951414 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:06.951379    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eba86cb-56ad-4b84-a667-7979fc9c0614-maas-api-service-ca\") pod \"2eba86cb-56ad-4b84-a667-7979fc9c0614\" (UID: \"2eba86cb-56ad-4b84-a667-7979fc9c0614\") "
Sep 18 17:32:06.951726 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:06.951702    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eba86cb-56ad-4b84-a667-7979fc9c0614-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2eba86cb-56ad-4b84-a667-7979fc9c0614" (UID: "2eba86cb-56ad-4b84-a667-7979fc9c0614"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:32:07.052698 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:07.052662    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eba86cb-56ad-4b84-a667-7979fc9c0614-maas-api-service-ca\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:32:07.718414 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:07.718376    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-97798a-cbxd2" event={"ID":"2eba86cb-56ad-4b84-a667-7979fc9c0614","Type":"ContainerDied","Data":"250c20854b0102167997023111e4c8125fb33f32bde6b7441339afd9bc8d11ac"}
Sep 18 17:32:07.718827 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:07.718435    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="250c20854b0102167997023111e4c8125fb33f32bde6b7441339afd9bc8d11ac"
Sep 18 17:32:07.718827 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:07.718384    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-97798a-cbxd2"
Sep 18 17:32:07.844241 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:07.844218    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55a2c1b6-cs76n"
Sep 18 17:32:07.962668 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:07.962637    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0c97202-9fa0-4e88-b485-5797a89ce1a8-maas-api-service-ca\") pod \"d0c97202-9fa0-4e88-b485-5797a89ce1a8\" (UID: \"d0c97202-9fa0-4e88-b485-5797a89ce1a8\") "
Sep 18 17:32:07.963031 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:07.963009    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c97202-9fa0-4e88-b485-5797a89ce1a8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d0c97202-9fa0-4e88-b485-5797a89ce1a8" (UID: "d0c97202-9fa0-4e88-b485-5797a89ce1a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:32:08.063925 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:08.063887    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0c97202-9fa0-4e88-b485-5797a89ce1a8-maas-api-service-ca\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:32:08.725875 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:08.725842    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55a2c1b6-cs76n"
Sep 18 17:32:08.726347 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:08.725874    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55a2c1b6-cs76n" event={"ID":"d0c97202-9fa0-4e88-b485-5797a89ce1a8","Type":"ContainerDied","Data":"c945ebe38b4dd3fb4db2490837840cec5d0892472f1444a55e4a3dc2fbca5cc0"}
Sep 18 17:32:08.726347 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:08.725920    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c945ebe38b4dd3fb4db2490837840cec5d0892472f1444a55e4a3dc2fbca5cc0"
Sep 18 17:32:14.657272 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:14.657236    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-97798a-cff5c65cc-bthtz"]
Sep 18 17:32:14.657781 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:14.657488    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-97798a-cff5c65cc-bthtz" podUID="4e32f5a1-a455-413e-8d51-52565ce0df1e" containerName="maas-api" containerID="cri-o://705a4f480b4a1dcb8540868fa4a85c593131b788ee6cef2415d4e28e630a0746" gracePeriod=30
Sep 18 17:32:14.714370 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:14.714333    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-97798a-5755fcf6c7-c7qmd"]
Sep 18 17:32:14.714618 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:14.714590    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-97798a-5755fcf6c7-c7qmd" podUID="7cbad065-e91d-4b19-a7c1-2857773e3d22" containerName="payload-processing" containerID="cri-o://6af6f570c4974adff50ec97972e1f2c04f974a6a350961635563edeffd78a0f3" gracePeriod=30
Sep 18 17:32:14.874550 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:14.874513    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-97798a-cbxd2"]
Sep 18 17:32:14.879191 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:14.879157    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-97798a-cbxd2"]
Sep 18 17:32:14.966624 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:14.966594    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-97798a-cff5c65cc-bthtz"
Sep 18 17:32:15.003109 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.003077    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-97798a-5755fcf6c7-c7qmd"
Sep 18 17:32:15.031404 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.031372    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkqvq\" (UniqueName: \"kubernetes.io/projected/4e32f5a1-a455-413e-8d51-52565ce0df1e-kube-api-access-pkqvq\") pod \"4e32f5a1-a455-413e-8d51-52565ce0df1e\" (UID: \"4e32f5a1-a455-413e-8d51-52565ce0df1e\") "
Sep 18 17:32:15.031404 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.031401    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cbad065-e91d-4b19-a7c1-2857773e3d22-plugins-config-volume\") pod \"7cbad065-e91d-4b19-a7c1-2857773e3d22\" (UID: \"7cbad065-e91d-4b19-a7c1-2857773e3d22\") "
Sep 18 17:32:15.031632 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.031422    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rtml\" (UniqueName: \"kubernetes.io/projected/7cbad065-e91d-4b19-a7c1-2857773e3d22-kube-api-access-9rtml\") pod \"7cbad065-e91d-4b19-a7c1-2857773e3d22\" (UID: \"7cbad065-e91d-4b19-a7c1-2857773e3d22\") "
Sep 18 17:32:15.031632 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.031536    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e32f5a1-a455-413e-8d51-52565ce0df1e-metrics-certs\") pod \"4e32f5a1-a455-413e-8d51-52565ce0df1e\" (UID: \"4e32f5a1-a455-413e-8d51-52565ce0df1e\") "
Sep 18 17:32:15.031632 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.031570    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e32f5a1-a455-413e-8d51-52565ce0df1e-openshift-service-ca\") pod \"4e32f5a1-a455-413e-8d51-52565ce0df1e\" (UID: \"4e32f5a1-a455-413e-8d51-52565ce0df1e\") "
Sep 18 17:32:15.031632 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.031629    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e32f5a1-a455-413e-8d51-52565ce0df1e-maas-api-tls\") pod \"4e32f5a1-a455-413e-8d51-52565ce0df1e\" (UID: \"4e32f5a1-a455-413e-8d51-52565ce0df1e\") "
Sep 18 17:32:15.031900 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.031782    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cbad065-e91d-4b19-a7c1-2857773e3d22-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7cbad065-e91d-4b19-a7c1-2857773e3d22" (UID: "7cbad065-e91d-4b19-a7c1-2857773e3d22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:32:15.032021 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.031981    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cbad065-e91d-4b19-a7c1-2857773e3d22-plugins-config-volume\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:32:15.032417 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.032384    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e32f5a1-a455-413e-8d51-52565ce0df1e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4e32f5a1-a455-413e-8d51-52565ce0df1e" (UID: "4e32f5a1-a455-413e-8d51-52565ce0df1e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:32:15.033951 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.033921    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e32f5a1-a455-413e-8d51-52565ce0df1e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4e32f5a1-a455-413e-8d51-52565ce0df1e" (UID: "4e32f5a1-a455-413e-8d51-52565ce0df1e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:32:15.034085 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.034062    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e32f5a1-a455-413e-8d51-52565ce0df1e-kube-api-access-pkqvq" (OuterVolumeSpecName: "kube-api-access-pkqvq") pod "4e32f5a1-a455-413e-8d51-52565ce0df1e" (UID: "4e32f5a1-a455-413e-8d51-52565ce0df1e"). InnerVolumeSpecName "kube-api-access-pkqvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:32:15.034128 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.034060    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cbad065-e91d-4b19-a7c1-2857773e3d22-kube-api-access-9rtml" (OuterVolumeSpecName: "kube-api-access-9rtml") pod "7cbad065-e91d-4b19-a7c1-2857773e3d22" (UID: "7cbad065-e91d-4b19-a7c1-2857773e3d22"). InnerVolumeSpecName "kube-api-access-9rtml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:32:15.034291 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.034264    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e32f5a1-a455-413e-8d51-52565ce0df1e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4e32f5a1-a455-413e-8d51-52565ce0df1e" (UID: "4e32f5a1-a455-413e-8d51-52565ce0df1e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:32:15.132621 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.132575    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e32f5a1-a455-413e-8d51-52565ce0df1e-metrics-certs\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:32:15.132621 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.132614    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e32f5a1-a455-413e-8d51-52565ce0df1e-openshift-service-ca\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:32:15.132621 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.132628    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e32f5a1-a455-413e-8d51-52565ce0df1e-maas-api-tls\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:32:15.132909 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.132640    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkqvq\" (UniqueName: \"kubernetes.io/projected/4e32f5a1-a455-413e-8d51-52565ce0df1e-kube-api-access-pkqvq\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:32:15.132909 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.132652    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rtml\" (UniqueName: \"kubernetes.io/projected/7cbad065-e91d-4b19-a7c1-2857773e3d22-kube-api-access-9rtml\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:32:15.765660 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.765624    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2eba86cb-56ad-4b84-a667-7979fc9c0614" path="/var/lib/kubelet/pods/2eba86cb-56ad-4b84-a667-7979fc9c0614/volumes"
Sep 18 17:32:15.767673 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.767642    2569 generic.go:356] "Generic (PLEG): container finished" podID="7cbad065-e91d-4b19-a7c1-2857773e3d22" containerID="6af6f570c4974adff50ec97972e1f2c04f974a6a350961635563edeffd78a0f3" exitCode=0
Sep 18 17:32:15.767832 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.767697    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-97798a-5755fcf6c7-c7qmd"
Sep 18 17:32:15.767832 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.767736    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-97798a-5755fcf6c7-c7qmd" event={"ID":"7cbad065-e91d-4b19-a7c1-2857773e3d22","Type":"ContainerDied","Data":"6af6f570c4974adff50ec97972e1f2c04f974a6a350961635563edeffd78a0f3"}
Sep 18 17:32:15.767832 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.767786    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-97798a-5755fcf6c7-c7qmd" event={"ID":"7cbad065-e91d-4b19-a7c1-2857773e3d22","Type":"ContainerDied","Data":"bc1b75209d09895d39e768a47364761c20dcba00480db98697c41da52f61dd7d"}
Sep 18 17:32:15.767832 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.767814    2569 scope.go:117] "RemoveContainer" containerID="6af6f570c4974adff50ec97972e1f2c04f974a6a350961635563edeffd78a0f3"
Sep 18 17:32:15.769373 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.769351    2569 generic.go:356] "Generic (PLEG): container finished" podID="4e32f5a1-a455-413e-8d51-52565ce0df1e" containerID="705a4f480b4a1dcb8540868fa4a85c593131b788ee6cef2415d4e28e630a0746" exitCode=0
Sep 18 17:32:15.769456 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.769410    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-97798a-cff5c65cc-bthtz"
Sep 18 17:32:15.769510 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.769408    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-97798a-cff5c65cc-bthtz" event={"ID":"4e32f5a1-a455-413e-8d51-52565ce0df1e","Type":"ContainerDied","Data":"705a4f480b4a1dcb8540868fa4a85c593131b788ee6cef2415d4e28e630a0746"}
Sep 18 17:32:15.769581 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.769547    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-97798a-cff5c65cc-bthtz" event={"ID":"4e32f5a1-a455-413e-8d51-52565ce0df1e","Type":"ContainerDied","Data":"7c512b02710456ddacc1f7a38bf909fd352ca0cf6562d3fc5c920810f84842d5"}
Sep 18 17:32:15.778941 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.778930    2569 scope.go:117] "RemoveContainer" containerID="6af6f570c4974adff50ec97972e1f2c04f974a6a350961635563edeffd78a0f3"
Sep 18 17:32:15.779197 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:32:15.779176    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6af6f570c4974adff50ec97972e1f2c04f974a6a350961635563edeffd78a0f3\": container with ID starting with 6af6f570c4974adff50ec97972e1f2c04f974a6a350961635563edeffd78a0f3 not found: ID does not exist" containerID="6af6f570c4974adff50ec97972e1f2c04f974a6a350961635563edeffd78a0f3"
Sep 18 17:32:15.779266 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.779202    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6af6f570c4974adff50ec97972e1f2c04f974a6a350961635563edeffd78a0f3"} err="failed to get container status \"6af6f570c4974adff50ec97972e1f2c04f974a6a350961635563edeffd78a0f3\": rpc error: code = NotFound desc = could not find container \"6af6f570c4974adff50ec97972e1f2c04f974a6a350961635563edeffd78a0f3\": container with ID starting with 6af6f570c4974adff50ec97972e1f2c04f974a6a350961635563edeffd78a0f3 not found: ID does not exist"
Sep 18 17:32:15.779266 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.779218    2569 scope.go:117] "RemoveContainer" containerID="705a4f480b4a1dcb8540868fa4a85c593131b788ee6cef2415d4e28e630a0746"
Sep 18 17:32:15.787634 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.787608    2569 scope.go:117] "RemoveContainer" containerID="705a4f480b4a1dcb8540868fa4a85c593131b788ee6cef2415d4e28e630a0746"
Sep 18 17:32:15.787922 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:32:15.787899    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"705a4f480b4a1dcb8540868fa4a85c593131b788ee6cef2415d4e28e630a0746\": container with ID starting with 705a4f480b4a1dcb8540868fa4a85c593131b788ee6cef2415d4e28e630a0746 not found: ID does not exist" containerID="705a4f480b4a1dcb8540868fa4a85c593131b788ee6cef2415d4e28e630a0746"
Sep 18 17:32:15.787983 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.787931    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"705a4f480b4a1dcb8540868fa4a85c593131b788ee6cef2415d4e28e630a0746"} err="failed to get container status \"705a4f480b4a1dcb8540868fa4a85c593131b788ee6cef2415d4e28e630a0746\": rpc error: code = NotFound desc = could not find container \"705a4f480b4a1dcb8540868fa4a85c593131b788ee6cef2415d4e28e630a0746\": container with ID starting with 705a4f480b4a1dcb8540868fa4a85c593131b788ee6cef2415d4e28e630a0746 not found: ID does not exist"
Sep 18 17:32:15.802773 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.802745    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-97798a-5755fcf6c7-c7qmd"]
Sep 18 17:32:15.805401 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.805380    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-97798a-5755fcf6c7-c7qmd"]
Sep 18 17:32:15.828971 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.828939    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-97798a-cff5c65cc-bthtz"]
Sep 18 17:32:15.831765 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:15.831741    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-97798a-cff5c65cc-bthtz"]
Sep 18 17:32:17.769014 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:17.768981    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e32f5a1-a455-413e-8d51-52565ce0df1e" path="/var/lib/kubelet/pods/4e32f5a1-a455-413e-8d51-52565ce0df1e/volumes"
Sep 18 17:32:17.769474 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:17.769458    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cbad065-e91d-4b19-a7c1-2857773e3d22" path="/var/lib/kubelet/pods/7cbad065-e91d-4b19-a7c1-2857773e3d22/volumes"
Sep 18 17:32:17.769749 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:17.769734    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-55a2c1b6-556fc86789-5hx2n"]
Sep 18 17:32:17.769970 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:17.769947    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-55a2c1b6-556fc86789-5hx2n" podUID="797b324e-173a-45ab-af25-14f04abfda25" containerName="payload-processing" containerID="cri-o://b4cf56b783de2370a85b3a0923afd03338a935a9cf249716289bcaf6bd554aa6" gracePeriod=30
Sep 18 17:32:17.972065 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:17.972036    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55a2c1b6-cs76n"]
Sep 18 17:32:17.975482 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:17.975449    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55a2c1b6-cs76n"]
Sep 18 17:32:18.023055 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:18.022996    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-55a2c1b6-556fc86789-5hx2n"
Sep 18 17:32:18.057740 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:18.057710    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/797b324e-173a-45ab-af25-14f04abfda25-plugins-config-volume\") pod \"797b324e-173a-45ab-af25-14f04abfda25\" (UID: \"797b324e-173a-45ab-af25-14f04abfda25\") "
Sep 18 17:32:18.057915 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:18.057853    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8z8kq\" (UniqueName: \"kubernetes.io/projected/797b324e-173a-45ab-af25-14f04abfda25-kube-api-access-8z8kq\") pod \"797b324e-173a-45ab-af25-14f04abfda25\" (UID: \"797b324e-173a-45ab-af25-14f04abfda25\") "
Sep 18 17:32:18.058088 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:18.058066    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/797b324e-173a-45ab-af25-14f04abfda25-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "797b324e-173a-45ab-af25-14f04abfda25" (UID: "797b324e-173a-45ab-af25-14f04abfda25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:32:18.058194 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:18.058172    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/797b324e-173a-45ab-af25-14f04abfda25-plugins-config-volume\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:32:18.060194 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:18.060164    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/797b324e-173a-45ab-af25-14f04abfda25-kube-api-access-8z8kq" (OuterVolumeSpecName: "kube-api-access-8z8kq") pod "797b324e-173a-45ab-af25-14f04abfda25" (UID: "797b324e-173a-45ab-af25-14f04abfda25"). InnerVolumeSpecName "kube-api-access-8z8kq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:32:18.159074 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:18.159042    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z8kq\" (UniqueName: \"kubernetes.io/projected/797b324e-173a-45ab-af25-14f04abfda25-kube-api-access-8z8kq\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:32:18.739747 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:18.739707    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv" podUID="58aeb0b0-d9f4-412c-b669-5a8a6331c9b4" containerName="main" containerID="cri-o://00528e2f37a6d7ba2e8ce3bc4bea941659360f102286f9b4f4e5ca52ccfc3d3b" gracePeriod=45
Sep 18 17:32:18.789432 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:18.789400    2569 generic.go:356] "Generic (PLEG): container finished" podID="797b324e-173a-45ab-af25-14f04abfda25" containerID="b4cf56b783de2370a85b3a0923afd03338a935a9cf249716289bcaf6bd554aa6" exitCode=0
Sep 18 17:32:18.789775 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:18.789444    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-55a2c1b6-556fc86789-5hx2n"
Sep 18 17:32:18.789775 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:18.789477    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-55a2c1b6-556fc86789-5hx2n" event={"ID":"797b324e-173a-45ab-af25-14f04abfda25","Type":"ContainerDied","Data":"b4cf56b783de2370a85b3a0923afd03338a935a9cf249716289bcaf6bd554aa6"}
Sep 18 17:32:18.789775 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:18.789530    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-55a2c1b6-556fc86789-5hx2n" event={"ID":"797b324e-173a-45ab-af25-14f04abfda25","Type":"ContainerDied","Data":"75e486437aff40bb81f66ce4d7c52fdb8f1d40589d2f362b69c6bd702d6c4aba"}
Sep 18 17:32:18.789775 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:18.789551    2569 scope.go:117] "RemoveContainer" containerID="b4cf56b783de2370a85b3a0923afd03338a935a9cf249716289bcaf6bd554aa6"
Sep 18 17:32:18.829359 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:18.829333    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-55a2c1b6-556fc86789-5hx2n"]
Sep 18 17:32:18.836138 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:18.836117    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-55a2c1b6-556fc86789-5hx2n"]
Sep 18 17:32:18.851279 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:18.851259    2569 scope.go:117] "RemoveContainer" containerID="b4cf56b783de2370a85b3a0923afd03338a935a9cf249716289bcaf6bd554aa6"
Sep 18 17:32:18.851649 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:32:18.851626    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4cf56b783de2370a85b3a0923afd03338a935a9cf249716289bcaf6bd554aa6\": container with ID starting with b4cf56b783de2370a85b3a0923afd03338a935a9cf249716289bcaf6bd554aa6 not found: ID does not exist" containerID="b4cf56b783de2370a85b3a0923afd03338a935a9cf249716289bcaf6bd554aa6"
Sep 18 17:32:18.851753 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:18.851656    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4cf56b783de2370a85b3a0923afd03338a935a9cf249716289bcaf6bd554aa6"} err="failed to get container status \"b4cf56b783de2370a85b3a0923afd03338a935a9cf249716289bcaf6bd554aa6\": rpc error: code = NotFound desc = could not find container \"b4cf56b783de2370a85b3a0923afd03338a935a9cf249716289bcaf6bd554aa6\": container with ID starting with b4cf56b783de2370a85b3a0923afd03338a935a9cf249716289bcaf6bd554aa6 not found: ID does not exist"
Sep 18 17:32:19.003275 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:19.003216    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv"
Sep 18 17:32:19.066807 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:19.066778    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9qh4\" (UniqueName: \"kubernetes.io/projected/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-kube-api-access-x9qh4\") pod \"58aeb0b0-d9f4-412c-b669-5a8a6331c9b4\" (UID: \"58aeb0b0-d9f4-412c-b669-5a8a6331c9b4\") "
Sep 18 17:32:19.066987 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:19.066820    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-model-cache\") pod \"58aeb0b0-d9f4-412c-b669-5a8a6331c9b4\" (UID: \"58aeb0b0-d9f4-412c-b669-5a8a6331c9b4\") "
Sep 18 17:32:19.066987 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:19.066856    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-tls-certs\") pod \"58aeb0b0-d9f4-412c-b669-5a8a6331c9b4\" (UID: \"58aeb0b0-d9f4-412c-b669-5a8a6331c9b4\") "
Sep 18 17:32:19.067700 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:19.067304    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-home\") pod \"58aeb0b0-d9f4-412c-b669-5a8a6331c9b4\" (UID: \"58aeb0b0-d9f4-412c-b669-5a8a6331c9b4\") "
Sep 18 17:32:19.067700 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:19.067391    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-dshm\") pod \"58aeb0b0-d9f4-412c-b669-5a8a6331c9b4\" (UID: \"58aeb0b0-d9f4-412c-b669-5a8a6331c9b4\") "
Sep 18 17:32:19.067700 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:19.067439    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-tmp-dir\") pod \"58aeb0b0-d9f4-412c-b669-5a8a6331c9b4\" (UID: \"58aeb0b0-d9f4-412c-b669-5a8a6331c9b4\") "
Sep 18 17:32:19.067700 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:19.067584    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-model-cache" (OuterVolumeSpecName: "model-cache") pod "58aeb0b0-d9f4-412c-b669-5a8a6331c9b4" (UID: "58aeb0b0-d9f4-412c-b669-5a8a6331c9b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:32:19.068062 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:19.067811    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-home" (OuterVolumeSpecName: "home") pod "58aeb0b0-d9f4-412c-b669-5a8a6331c9b4" (UID: "58aeb0b0-d9f4-412c-b669-5a8a6331c9b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:32:19.068922 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:19.068174    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "58aeb0b0-d9f4-412c-b669-5a8a6331c9b4" (UID: "58aeb0b0-d9f4-412c-b669-5a8a6331c9b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:32:19.068922 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:19.068780    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-model-cache\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:32:19.068922 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:19.068806    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-home\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:32:19.069980 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:19.069937    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-kube-api-access-x9qh4" (OuterVolumeSpecName: "kube-api-access-x9qh4") pod "58aeb0b0-d9f4-412c-b669-5a8a6331c9b4" (UID: "58aeb0b0-d9f4-412c-b669-5a8a6331c9b4"). InnerVolumeSpecName "kube-api-access-x9qh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:32:19.072909 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:19.070791    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-dshm" (OuterVolumeSpecName: "dshm") pod "58aeb0b0-d9f4-412c-b669-5a8a6331c9b4" (UID: "58aeb0b0-d9f4-412c-b669-5a8a6331c9b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:32:19.072909 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:19.071958    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "58aeb0b0-d9f4-412c-b669-5a8a6331c9b4" (UID: "58aeb0b0-d9f4-412c-b669-5a8a6331c9b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:32:19.169766 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:19.169730    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-tls-certs\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:32:19.169766 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:19.169760    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-dshm\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:32:19.169766 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:19.169768    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-tmp-dir\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:32:19.169766 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:19.169776    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9qh4\" (UniqueName: \"kubernetes.io/projected/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4-kube-api-access-x9qh4\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:32:19.765761 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:19.765723    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="797b324e-173a-45ab-af25-14f04abfda25" path="/var/lib/kubelet/pods/797b324e-173a-45ab-af25-14f04abfda25/volumes"
Sep 18 17:32:19.766147 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:19.766131    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0c97202-9fa0-4e88-b485-5797a89ce1a8" path="/var/lib/kubelet/pods/d0c97202-9fa0-4e88-b485-5797a89ce1a8/volumes"
Sep 18 17:32:19.806468 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:19.806434    2569 generic.go:356] "Generic (PLEG): container finished" podID="58aeb0b0-d9f4-412c-b669-5a8a6331c9b4" containerID="00528e2f37a6d7ba2e8ce3bc4bea941659360f102286f9b4f4e5ca52ccfc3d3b" exitCode=0
Sep 18 17:32:19.807348 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:19.807079    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv"
Sep 18 17:32:19.808047 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:19.807103    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv" event={"ID":"58aeb0b0-d9f4-412c-b669-5a8a6331c9b4","Type":"ContainerDied","Data":"00528e2f37a6d7ba2e8ce3bc4bea941659360f102286f9b4f4e5ca52ccfc3d3b"}
Sep 18 17:32:19.808447 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:19.808426    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv" event={"ID":"58aeb0b0-d9f4-412c-b669-5a8a6331c9b4","Type":"ContainerDied","Data":"653166709b16f0ccf00aff9b8e97b425e4742c6a27636d046bc5c687c7153f91"}
Sep 18 17:32:19.808653 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:19.808578    2569 scope.go:117] "RemoveContainer" containerID="00528e2f37a6d7ba2e8ce3bc4bea941659360f102286f9b4f4e5ca52ccfc3d3b"
Sep 18 17:32:19.832566 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:19.832538    2569 scope.go:117] "RemoveContainer" containerID="00528e2f37a6d7ba2e8ce3bc4bea941659360f102286f9b4f4e5ca52ccfc3d3b"
Sep 18 17:32:19.833130 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:32:19.833105    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00528e2f37a6d7ba2e8ce3bc4bea941659360f102286f9b4f4e5ca52ccfc3d3b\": container with ID starting with 00528e2f37a6d7ba2e8ce3bc4bea941659360f102286f9b4f4e5ca52ccfc3d3b not found: ID does not exist" containerID="00528e2f37a6d7ba2e8ce3bc4bea941659360f102286f9b4f4e5ca52ccfc3d3b"
Sep 18 17:32:19.833305 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:19.833277    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00528e2f37a6d7ba2e8ce3bc4bea941659360f102286f9b4f4e5ca52ccfc3d3b"} err="failed to get container status \"00528e2f37a6d7ba2e8ce3bc4bea941659360f102286f9b4f4e5ca52ccfc3d3b\": rpc error: code = NotFound desc = could not find container \"00528e2f37a6d7ba2e8ce3bc4bea941659360f102286f9b4f4e5ca52ccfc3d3b\": container with ID starting with 00528e2f37a6d7ba2e8ce3bc4bea941659360f102286f9b4f4e5ca52ccfc3d3b not found: ID does not exist"
Sep 18 17:32:19.846798 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:19.846768    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv"]
Sep 18 17:32:19.850636 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:19.850605    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-97798a/explicit-model-97798a-kserve-6d6d9764f5-wpdqv"]
Sep 18 17:32:21.765288 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:21.765251    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58aeb0b0-d9f4-412c-b669-5a8a6331c9b4" path="/var/lib/kubelet/pods/58aeb0b0-d9f4-412c-b669-5a8a6331c9b4/volumes"
Sep 18 17:32:49.530985 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.530950    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl"]
Sep 18 17:32:49.531386 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.531325    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0c97202-9fa0-4e88-b485-5797a89ce1a8" containerName="revoke-keys"
Sep 18 17:32:49.531386 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.531337    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0c97202-9fa0-4e88-b485-5797a89ce1a8" containerName="revoke-keys"
Sep 18 17:32:49.531386 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.531349    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="797b324e-173a-45ab-af25-14f04abfda25" containerName="payload-processing"
Sep 18 17:32:49.531386 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.531355    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="797b324e-173a-45ab-af25-14f04abfda25" containerName="payload-processing"
Sep 18 17:32:49.531386 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.531361    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58aeb0b0-d9f4-412c-b669-5a8a6331c9b4" containerName="main"
Sep 18 17:32:49.531386 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.531367    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="58aeb0b0-d9f4-412c-b669-5a8a6331c9b4" containerName="main"
Sep 18 17:32:49.531386 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.531376    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eba86cb-56ad-4b84-a667-7979fc9c0614" containerName="revoke-keys"
Sep 18 17:32:49.531386 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.531382    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eba86cb-56ad-4b84-a667-7979fc9c0614" containerName="revoke-keys"
Sep 18 17:32:49.531668 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.531394    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cbad065-e91d-4b19-a7c1-2857773e3d22" containerName="payload-processing"
Sep 18 17:32:49.531668 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.531399    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cbad065-e91d-4b19-a7c1-2857773e3d22" containerName="payload-processing"
Sep 18 17:32:49.531668 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.531410    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e32f5a1-a455-413e-8d51-52565ce0df1e" containerName="maas-api"
Sep 18 17:32:49.531668 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.531415    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e32f5a1-a455-413e-8d51-52565ce0df1e" containerName="maas-api"
Sep 18 17:32:49.531668 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.531468    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="58aeb0b0-d9f4-412c-b669-5a8a6331c9b4" containerName="main"
Sep 18 17:32:49.531668 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.531475    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="797b324e-173a-45ab-af25-14f04abfda25" containerName="payload-processing"
Sep 18 17:32:49.531668 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.531482    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e32f5a1-a455-413e-8d51-52565ce0df1e" containerName="maas-api"
Sep 18 17:32:49.531668 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.531489    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cbad065-e91d-4b19-a7c1-2857773e3d22" containerName="payload-processing"
Sep 18 17:32:49.531668 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.531495    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0c97202-9fa0-4e88-b485-5797a89ce1a8" containerName="revoke-keys"
Sep 18 17:32:49.531668 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.531499    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2eba86cb-56ad-4b84-a667-7979fc9c0614" containerName="revoke-keys"
Sep 18 17:32:49.535843 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.535824    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl"
Sep 18 17:32:49.539404 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.539382    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-dd235c-serving-cert\""
Sep 18 17:32:49.539520 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.539410    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 17:32:49.539520 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.539415    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sjtq4\""
Sep 18 17:32:49.540759 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.540745    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 18 17:32:49.540852 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.540834    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 17:32:49.556570 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.556543    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl"]
Sep 18 17:32:49.652268 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.652232    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fqzv\" (UniqueName: \"kubernetes.io/projected/87dc18db-0ad6-4e73-9c6d-8be180920810-kube-api-access-6fqzv\") pod \"maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl\" (UID: \"87dc18db-0ad6-4e73-9c6d-8be180920810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl"
Sep 18 17:32:49.652465 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.652319    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87dc18db-0ad6-4e73-9c6d-8be180920810-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl\" (UID: \"87dc18db-0ad6-4e73-9c6d-8be180920810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl"
Sep 18 17:32:49.652465 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.652355    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87dc18db-0ad6-4e73-9c6d-8be180920810-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl\" (UID: \"87dc18db-0ad6-4e73-9c6d-8be180920810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl"
Sep 18 17:32:49.652465 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.652389    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87dc18db-0ad6-4e73-9c6d-8be180920810-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl\" (UID: \"87dc18db-0ad6-4e73-9c6d-8be180920810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl"
Sep 18 17:32:49.753093 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.753054    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87dc18db-0ad6-4e73-9c6d-8be180920810-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl\" (UID: \"87dc18db-0ad6-4e73-9c6d-8be180920810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl"
Sep 18 17:32:49.753283 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.753102    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fqzv\" (UniqueName: \"kubernetes.io/projected/87dc18db-0ad6-4e73-9c6d-8be180920810-kube-api-access-6fqzv\") pod \"maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl\" (UID: \"87dc18db-0ad6-4e73-9c6d-8be180920810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl"
Sep 18 17:32:49.753283 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.753170    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87dc18db-0ad6-4e73-9c6d-8be180920810-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl\" (UID: \"87dc18db-0ad6-4e73-9c6d-8be180920810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl"
Sep 18 17:32:49.753283 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.753191    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87dc18db-0ad6-4e73-9c6d-8be180920810-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl\" (UID: \"87dc18db-0ad6-4e73-9c6d-8be180920810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl"
Sep 18 17:32:49.753283 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:32:49.753284    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dd235c-serving-cert: secret "maas-api-e2e-shared-a-w2-dd235c-serving-cert" not found
Sep 18 17:32:49.753505 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:32:49.753339    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87dc18db-0ad6-4e73-9c6d-8be180920810-maas-api-tls podName:87dc18db-0ad6-4e73-9c6d-8be180920810 nodeName:}" failed. No retries permitted until 2026-09-18 17:32:50.253323573 +0000 UTC m=+1082.947251083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/87dc18db-0ad6-4e73-9c6d-8be180920810-maas-api-tls") pod "maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl" (UID: "87dc18db-0ad6-4e73-9c6d-8be180920810") : secret "maas-api-e2e-shared-a-w2-dd235c-serving-cert" not found
Sep 18 17:32:49.753795 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.753776    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87dc18db-0ad6-4e73-9c6d-8be180920810-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl\" (UID: \"87dc18db-0ad6-4e73-9c6d-8be180920810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl"
Sep 18 17:32:49.755596 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.755574    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87dc18db-0ad6-4e73-9c6d-8be180920810-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl\" (UID: \"87dc18db-0ad6-4e73-9c6d-8be180920810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl"
Sep 18 17:32:49.764146 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:49.764121    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fqzv\" (UniqueName: \"kubernetes.io/projected/87dc18db-0ad6-4e73-9c6d-8be180920810-kube-api-access-6fqzv\") pod \"maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl\" (UID: \"87dc18db-0ad6-4e73-9c6d-8be180920810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl"
Sep 18 17:32:50.259131 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:50.259089    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87dc18db-0ad6-4e73-9c6d-8be180920810-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl\" (UID: \"87dc18db-0ad6-4e73-9c6d-8be180920810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl"
Sep 18 17:32:50.261483 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:50.261462    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87dc18db-0ad6-4e73-9c6d-8be180920810-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl\" (UID: \"87dc18db-0ad6-4e73-9c6d-8be180920810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl"
Sep 18 17:32:50.446172 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:50.446131    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl"
Sep 18 17:32:50.468508 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:50.468477    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dd235c-575594b894-wvv67"]
Sep 18 17:32:50.472102 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:50.472083    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dd235c-575594b894-wvv67"
Sep 18 17:32:50.475297 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:50.475270    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-dd235c\""
Sep 18 17:32:50.475398 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:50.475315    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-dd235c-dockercfg-wxhjc\""
Sep 18 17:32:50.483653 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:50.483577    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dd235c-6cf55ffd4c-k4gv9"]
Sep 18 17:32:50.487252 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:50.487231    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dd235c-6cf55ffd4c-k4gv9"
Sep 18 17:32:50.493422 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:50.493371    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dd235c-575594b894-wvv67"]
Sep 18 17:32:50.501654 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:50.501624    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dd235c-6cf55ffd4c-k4gv9"]
Sep 18 17:32:50.561735 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:50.561704    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/986f5132-20cd-4154-b364-b7d352e5324b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-dd235c-575594b894-wvv67\" (UID: \"986f5132-20cd-4154-b364-b7d352e5324b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dd235c-575594b894-wvv67"
Sep 18 17:32:50.562137 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:50.561787    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twwh7\" (UniqueName: \"kubernetes.io/projected/986f5132-20cd-4154-b364-b7d352e5324b-kube-api-access-twwh7\") pod \"payload-pre-processing-e2e-shared-a-w2-dd235c-575594b894-wvv67\" (UID: \"986f5132-20cd-4154-b364-b7d352e5324b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dd235c-575594b894-wvv67"
Sep 18 17:32:50.662605 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:50.662562    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/986f5132-20cd-4154-b364-b7d352e5324b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-dd235c-575594b894-wvv67\" (UID: \"986f5132-20cd-4154-b364-b7d352e5324b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dd235c-575594b894-wvv67"
Sep 18 17:32:50.662605 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:50.662612    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92ng9\" (UniqueName: \"kubernetes.io/projected/73c3ef72-b53a-41af-899b-e3c9b7aed1d6-kube-api-access-92ng9\") pod \"payload-processing-e2e-shared-a-w2-dd235c-6cf55ffd4c-k4gv9\" (UID: \"73c3ef72-b53a-41af-899b-e3c9b7aed1d6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dd235c-6cf55ffd4c-k4gv9"
Sep 18 17:32:50.662891 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:50.662653    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twwh7\" (UniqueName: \"kubernetes.io/projected/986f5132-20cd-4154-b364-b7d352e5324b-kube-api-access-twwh7\") pod \"payload-pre-processing-e2e-shared-a-w2-dd235c-575594b894-wvv67\" (UID: \"986f5132-20cd-4154-b364-b7d352e5324b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dd235c-575594b894-wvv67"
Sep 18 17:32:50.662891 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:50.662673    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c3ef72-b53a-41af-899b-e3c9b7aed1d6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-dd235c-6cf55ffd4c-k4gv9\" (UID: \"73c3ef72-b53a-41af-899b-e3c9b7aed1d6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dd235c-6cf55ffd4c-k4gv9"
Sep 18 17:32:50.663334 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:50.663313    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/986f5132-20cd-4154-b364-b7d352e5324b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-dd235c-575594b894-wvv67\" (UID: \"986f5132-20cd-4154-b364-b7d352e5324b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dd235c-575594b894-wvv67"
Sep 18 17:32:50.673231 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:50.673208    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twwh7\" (UniqueName: \"kubernetes.io/projected/986f5132-20cd-4154-b364-b7d352e5324b-kube-api-access-twwh7\") pod \"payload-pre-processing-e2e-shared-a-w2-dd235c-575594b894-wvv67\" (UID: \"986f5132-20cd-4154-b364-b7d352e5324b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dd235c-575594b894-wvv67"
Sep 18 17:32:50.737732 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:50.737707    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl"]
Sep 18 17:32:50.739624 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:32:50.739598    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87dc18db_0ad6_4e73_9c6d_8be180920810.slice/crio-7afb66d7754960f1131d7d5d5fa045c888cf48dcfaf738297d5502694b923034 WatchSource:0}: Error finding container 7afb66d7754960f1131d7d5d5fa045c888cf48dcfaf738297d5502694b923034: Status 404 returned error can't find the container with id 7afb66d7754960f1131d7d5d5fa045c888cf48dcfaf738297d5502694b923034
Sep 18 17:32:50.764398 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:50.764328    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92ng9\" (UniqueName: \"kubernetes.io/projected/73c3ef72-b53a-41af-899b-e3c9b7aed1d6-kube-api-access-92ng9\") pod \"payload-processing-e2e-shared-a-w2-dd235c-6cf55ffd4c-k4gv9\" (UID: \"73c3ef72-b53a-41af-899b-e3c9b7aed1d6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dd235c-6cf55ffd4c-k4gv9"
Sep 18 17:32:50.764398 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:50.764391    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c3ef72-b53a-41af-899b-e3c9b7aed1d6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-dd235c-6cf55ffd4c-k4gv9\" (UID: \"73c3ef72-b53a-41af-899b-e3c9b7aed1d6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dd235c-6cf55ffd4c-k4gv9"
Sep 18 17:32:50.765165 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:50.765136    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c3ef72-b53a-41af-899b-e3c9b7aed1d6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-dd235c-6cf55ffd4c-k4gv9\" (UID: \"73c3ef72-b53a-41af-899b-e3c9b7aed1d6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dd235c-6cf55ffd4c-k4gv9"
Sep 18 17:32:50.777638 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:50.777612    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92ng9\" (UniqueName: \"kubernetes.io/projected/73c3ef72-b53a-41af-899b-e3c9b7aed1d6-kube-api-access-92ng9\") pod \"payload-processing-e2e-shared-a-w2-dd235c-6cf55ffd4c-k4gv9\" (UID: \"73c3ef72-b53a-41af-899b-e3c9b7aed1d6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dd235c-6cf55ffd4c-k4gv9"
Sep 18 17:32:50.787455 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:50.787432    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dd235c-575594b894-wvv67"
Sep 18 17:32:50.800511 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:50.800484    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dd235c-6cf55ffd4c-k4gv9"
Sep 18 17:32:50.979327 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:50.979293    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl" event={"ID":"87dc18db-0ad6-4e73-9c6d-8be180920810","Type":"ContainerStarted","Data":"7afb66d7754960f1131d7d5d5fa045c888cf48dcfaf738297d5502694b923034"}
Sep 18 17:32:51.076231 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:51.076201    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dd235c-575594b894-wvv67"]
Sep 18 17:32:51.077624 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:32:51.077599    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod986f5132_20cd_4154_b364_b7d352e5324b.slice/crio-df97cccebbf31e6ad0f01d0c6258e55055fac24cffd41bf73acb6cb27e245e45 WatchSource:0}: Error finding container df97cccebbf31e6ad0f01d0c6258e55055fac24cffd41bf73acb6cb27e245e45: Status 404 returned error can't find the container with id df97cccebbf31e6ad0f01d0c6258e55055fac24cffd41bf73acb6cb27e245e45
Sep 18 17:32:51.086618 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:51.086595    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dd235c-6cf55ffd4c-k4gv9"]
Sep 18 17:32:51.089770 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:32:51.089747    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c3ef72_b53a_41af_899b_e3c9b7aed1d6.slice/crio-13526d3aace981831b457463d47421aa6fcf91715c77e0c037e781eab394772d WatchSource:0}: Error finding container 13526d3aace981831b457463d47421aa6fcf91715c77e0c037e781eab394772d: Status 404 returned error can't find the container with id 13526d3aace981831b457463d47421aa6fcf91715c77e0c037e781eab394772d
Sep 18 17:32:51.986976 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:51.986936    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dd235c-6cf55ffd4c-k4gv9" event={"ID":"73c3ef72-b53a-41af-899b-e3c9b7aed1d6","Type":"ContainerStarted","Data":"9e2fae374220f7034ea25bf4c86fa9e46231a57ede4b836621507fd3427b4910"}
Sep 18 17:32:51.987453 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:51.987001    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dd235c-6cf55ffd4c-k4gv9" event={"ID":"73c3ef72-b53a-41af-899b-e3c9b7aed1d6","Type":"ContainerStarted","Data":"13526d3aace981831b457463d47421aa6fcf91715c77e0c037e781eab394772d"}
Sep 18 17:32:51.987453 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:51.987026    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dd235c-6cf55ffd4c-k4gv9"
Sep 18 17:32:51.988645 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:51.988621    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl" event={"ID":"87dc18db-0ad6-4e73-9c6d-8be180920810","Type":"ContainerStarted","Data":"97f88de0a52477c8fefe09506ac6b9d623ef6242843cd2a822b7a44a7fbd94d4"}
Sep 18 17:32:51.988754 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:51.988695    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl"
Sep 18 17:32:51.990232 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:51.990211    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dd235c-575594b894-wvv67" event={"ID":"986f5132-20cd-4154-b364-b7d352e5324b","Type":"ContainerStarted","Data":"c3649d8a019a83942a706bb90c59efec831022576433a33ff024e13e28409722"}
Sep 18 17:32:51.990342 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:51.990256    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dd235c-575594b894-wvv67" event={"ID":"986f5132-20cd-4154-b364-b7d352e5324b","Type":"ContainerStarted","Data":"df97cccebbf31e6ad0f01d0c6258e55055fac24cffd41bf73acb6cb27e245e45"}
Sep 18 17:32:51.990342 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:51.990283    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dd235c-575594b894-wvv67"
Sep 18 17:32:52.026320 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:52.026269    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dd235c-6cf55ffd4c-k4gv9" podStartSLOduration=2.026253955 podStartE2EDuration="2.026253955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:32:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:32:52.025464876 +0000 UTC m=+1084.719392404" watchObservedRunningTime="2026-09-18 17:32:52.026253955 +0000 UTC m=+1084.720181483"
Sep 18 17:32:52.054613 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:52.054559    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl" podStartSLOduration=2.678854006 podStartE2EDuration="3.054543559s" totalImagesPullingTime="375.689553ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:32:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:32:52.053433593 +0000 UTC m=+1084.747361120" watchObservedRunningTime="2026-09-18 17:32:52.054543559 +0000 UTC m=+1084.748471132"
Sep 18 17:32:52.078094 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:52.078007    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dd235c-575594b894-wvv67" podStartSLOduration=2.077988246 podStartE2EDuration="2.077988246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:32:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:32:52.076246515 +0000 UTC m=+1084.770174043" watchObservedRunningTime="2026-09-18 17:32:52.077988246 +0000 UTC m=+1084.771915775"
Sep 18 17:32:58.001419 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:32:58.001390    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl"
Sep 18 17:33:01.511799 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:01.511757    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54"]
Sep 18 17:33:02.998287 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:02.998246    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dd235c-6cf55ffd4c-k4gv9"
Sep 18 17:33:02.998695 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:02.998389    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dd235c-575594b894-wvv67"
Sep 18 17:33:16.404209 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:16.404172    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636b80-v4km9"]
Sep 18 17:33:16.407570 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:16.407553    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636b80-v4km9"
Sep 18 17:33:16.410352 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:16.410334    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qt978\""
Sep 18 17:33:16.417423 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:16.417396    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636b80-v4km9"]
Sep 18 17:33:16.508324 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:16.508287    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2319387f-0c5d-4e1c-bcc4-ede05f9956ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-636b80-v4km9\" (UID: \"2319387f-0c5d-4e1c-bcc4-ede05f9956ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636b80-v4km9"
Sep 18 17:33:16.530790 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:16.530742    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54" podUID="246181d7-28f4-453c-bd85-01c6d8fb6e51" containerName="main" containerID="cri-o://f95cc50647b09dce31b894548b0ad79ccae7cd94c75666f1886ac0c598ca30a7" gracePeriod=45
Sep 18 17:33:16.609466 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:16.609435    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2319387f-0c5d-4e1c-bcc4-ede05f9956ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-636b80-v4km9\" (UID: \"2319387f-0c5d-4e1c-bcc4-ede05f9956ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636b80-v4km9"
Sep 18 17:33:16.610068 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:16.610048    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2319387f-0c5d-4e1c-bcc4-ede05f9956ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-636b80-v4km9\" (UID: \"2319387f-0c5d-4e1c-bcc4-ede05f9956ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636b80-v4km9"
Sep 18 17:33:16.719329 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:16.719290    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636b80-v4km9"
Sep 18 17:33:16.797936 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:16.797852    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54"
Sep 18 17:33:16.912622 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:16.912590    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/246181d7-28f4-453c-bd85-01c6d8fb6e51-model-cache\") pod \"246181d7-28f4-453c-bd85-01c6d8fb6e51\" (UID: \"246181d7-28f4-453c-bd85-01c6d8fb6e51\") "
Sep 18 17:33:16.912622 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:16.912635    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/246181d7-28f4-453c-bd85-01c6d8fb6e51-home\") pod \"246181d7-28f4-453c-bd85-01c6d8fb6e51\" (UID: \"246181d7-28f4-453c-bd85-01c6d8fb6e51\") "
Sep 18 17:33:16.912849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:16.912662    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/246181d7-28f4-453c-bd85-01c6d8fb6e51-dshm\") pod \"246181d7-28f4-453c-bd85-01c6d8fb6e51\" (UID: \"246181d7-28f4-453c-bd85-01c6d8fb6e51\") "
Sep 18 17:33:16.912849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:16.912738    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/246181d7-28f4-453c-bd85-01c6d8fb6e51-tmp-dir\") pod \"246181d7-28f4-453c-bd85-01c6d8fb6e51\" (UID: \"246181d7-28f4-453c-bd85-01c6d8fb6e51\") "
Sep 18 17:33:16.912849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:16.912773    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/246181d7-28f4-453c-bd85-01c6d8fb6e51-tls-certs\") pod \"246181d7-28f4-453c-bd85-01c6d8fb6e51\" (UID: \"246181d7-28f4-453c-bd85-01c6d8fb6e51\") "
Sep 18 17:33:16.912849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:16.912806    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7lnr\" (UniqueName: \"kubernetes.io/projected/246181d7-28f4-453c-bd85-01c6d8fb6e51-kube-api-access-h7lnr\") pod \"246181d7-28f4-453c-bd85-01c6d8fb6e51\" (UID: \"246181d7-28f4-453c-bd85-01c6d8fb6e51\") "
Sep 18 17:33:16.913099 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:16.912933    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246181d7-28f4-453c-bd85-01c6d8fb6e51-home" (OuterVolumeSpecName: "home") pod "246181d7-28f4-453c-bd85-01c6d8fb6e51" (UID: "246181d7-28f4-453c-bd85-01c6d8fb6e51"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:33:16.913099 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:16.912948    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246181d7-28f4-453c-bd85-01c6d8fb6e51-model-cache" (OuterVolumeSpecName: "model-cache") pod "246181d7-28f4-453c-bd85-01c6d8fb6e51" (UID: "246181d7-28f4-453c-bd85-01c6d8fb6e51"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:33:16.913099 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:16.913080    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246181d7-28f4-453c-bd85-01c6d8fb6e51-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "246181d7-28f4-453c-bd85-01c6d8fb6e51" (UID: "246181d7-28f4-453c-bd85-01c6d8fb6e51"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:33:16.913263 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:16.913210    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/246181d7-28f4-453c-bd85-01c6d8fb6e51-model-cache\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:33:16.913263 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:16.913229    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/246181d7-28f4-453c-bd85-01c6d8fb6e51-home\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:33:16.913263 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:16.913244    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/246181d7-28f4-453c-bd85-01c6d8fb6e51-tmp-dir\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:33:16.914988 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:16.914964    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246181d7-28f4-453c-bd85-01c6d8fb6e51-dshm" (OuterVolumeSpecName: "dshm") pod "246181d7-28f4-453c-bd85-01c6d8fb6e51" (UID: "246181d7-28f4-453c-bd85-01c6d8fb6e51"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:33:16.915076 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:16.914990    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/246181d7-28f4-453c-bd85-01c6d8fb6e51-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "246181d7-28f4-453c-bd85-01c6d8fb6e51" (UID: "246181d7-28f4-453c-bd85-01c6d8fb6e51"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:33:16.915076 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:16.915018    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/246181d7-28f4-453c-bd85-01c6d8fb6e51-kube-api-access-h7lnr" (OuterVolumeSpecName: "kube-api-access-h7lnr") pod "246181d7-28f4-453c-bd85-01c6d8fb6e51" (UID: "246181d7-28f4-453c-bd85-01c6d8fb6e51"). InnerVolumeSpecName "kube-api-access-h7lnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:33:16.998091 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:16.998067    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636b80-v4km9"]
Sep 18 17:33:16.999805 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:33:16.999778    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2319387f_0c5d_4e1c_bcc4_ede05f9956ab.slice/crio-21a8e07986debbdb0bb0eda643bad9acb4a3f9df64392197733135eb72b69c24 WatchSource:0}: Error finding container 21a8e07986debbdb0bb0eda643bad9acb4a3f9df64392197733135eb72b69c24: Status 404 returned error can't find the container with id 21a8e07986debbdb0bb0eda643bad9acb4a3f9df64392197733135eb72b69c24
Sep 18 17:33:17.014675 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:17.014659    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/246181d7-28f4-453c-bd85-01c6d8fb6e51-tls-certs\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:33:17.014760 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:17.014679    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7lnr\" (UniqueName: \"kubernetes.io/projected/246181d7-28f4-453c-bd85-01c6d8fb6e51-kube-api-access-h7lnr\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:33:17.014760 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:17.014689    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/246181d7-28f4-453c-bd85-01c6d8fb6e51-dshm\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:33:17.122839 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:17.122807    2569 generic.go:356] "Generic (PLEG): container finished" podID="246181d7-28f4-453c-bd85-01c6d8fb6e51" containerID="f95cc50647b09dce31b894548b0ad79ccae7cd94c75666f1886ac0c598ca30a7" exitCode=0
Sep 18 17:33:17.123010 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:17.122848    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54"
Sep 18 17:33:17.123010 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:17.122906    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54" event={"ID":"246181d7-28f4-453c-bd85-01c6d8fb6e51","Type":"ContainerDied","Data":"f95cc50647b09dce31b894548b0ad79ccae7cd94c75666f1886ac0c598ca30a7"}
Sep 18 17:33:17.123010 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:17.122944    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54" event={"ID":"246181d7-28f4-453c-bd85-01c6d8fb6e51","Type":"ContainerDied","Data":"ce9e5b096fd917a1726a02d92549a004a01cb188099a9a241a129e9e3beb411d"}
Sep 18 17:33:17.123010 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:17.122961    2569 scope.go:117] "RemoveContainer" containerID="f95cc50647b09dce31b894548b0ad79ccae7cd94c75666f1886ac0c598ca30a7"
Sep 18 17:33:17.124319 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:17.124298    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636b80-v4km9" event={"ID":"2319387f-0c5d-4e1c-bcc4-ede05f9956ab","Type":"ContainerStarted","Data":"21a8e07986debbdb0bb0eda643bad9acb4a3f9df64392197733135eb72b69c24"}
Sep 18 17:33:17.132120 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:17.132103    2569 scope.go:117] "RemoveContainer" containerID="f95cc50647b09dce31b894548b0ad79ccae7cd94c75666f1886ac0c598ca30a7"
Sep 18 17:33:17.132391 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:33:17.132373    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f95cc50647b09dce31b894548b0ad79ccae7cd94c75666f1886ac0c598ca30a7\": container with ID starting with f95cc50647b09dce31b894548b0ad79ccae7cd94c75666f1886ac0c598ca30a7 not found: ID does not exist" containerID="f95cc50647b09dce31b894548b0ad79ccae7cd94c75666f1886ac0c598ca30a7"
Sep 18 17:33:17.132444 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:17.132398    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f95cc50647b09dce31b894548b0ad79ccae7cd94c75666f1886ac0c598ca30a7"} err="failed to get container status \"f95cc50647b09dce31b894548b0ad79ccae7cd94c75666f1886ac0c598ca30a7\": rpc error: code = NotFound desc = could not find container \"f95cc50647b09dce31b894548b0ad79ccae7cd94c75666f1886ac0c598ca30a7\": container with ID starting with f95cc50647b09dce31b894548b0ad79ccae7cd94c75666f1886ac0c598ca30a7 not found: ID does not exist"
Sep 18 17:33:17.151311 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:17.151282    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54"]
Sep 18 17:33:17.156129 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:17.156108    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-636b80/test-model-636b80-kserve-68c5cc9b47-ngm54"]
Sep 18 17:33:17.765392 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:17.765362    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="246181d7-28f4-453c-bd85-01c6d8fb6e51" path="/var/lib/kubelet/pods/246181d7-28f4-453c-bd85-01c6d8fb6e51/volumes"
Sep 18 17:33:18.133339 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:18.133305    2569 generic.go:356] "Generic (PLEG): container finished" podID="2319387f-0c5d-4e1c-bcc4-ede05f9956ab" containerID="26751883c6c307e3d48f9399a19c9b4d5f19072a7c1f8c864c2ec8782c7a04fd" exitCode=0
Sep 18 17:33:18.133581 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:18.133396    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636b80-v4km9" event={"ID":"2319387f-0c5d-4e1c-bcc4-ede05f9956ab","Type":"ContainerDied","Data":"26751883c6c307e3d48f9399a19c9b4d5f19072a7c1f8c864c2ec8782c7a04fd"}
Sep 18 17:33:19.273068 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:19.273043    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636b80-v4km9"
Sep 18 17:33:19.337528 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:19.337500    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2319387f-0c5d-4e1c-bcc4-ede05f9956ab-maas-api-service-ca\") pod \"2319387f-0c5d-4e1c-bcc4-ede05f9956ab\" (UID: \"2319387f-0c5d-4e1c-bcc4-ede05f9956ab\") "
Sep 18 17:33:19.337874 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:19.337841    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2319387f-0c5d-4e1c-bcc4-ede05f9956ab-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2319387f-0c5d-4e1c-bcc4-ede05f9956ab" (UID: "2319387f-0c5d-4e1c-bcc4-ede05f9956ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:33:19.438925 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:19.438811    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2319387f-0c5d-4e1c-bcc4-ede05f9956ab-maas-api-service-ca\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:33:20.147577 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:20.147495    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636b80-v4km9"
Sep 18 17:33:20.147577 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:20.147519    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636b80-v4km9" event={"ID":"2319387f-0c5d-4e1c-bcc4-ede05f9956ab","Type":"ContainerDied","Data":"21a8e07986debbdb0bb0eda643bad9acb4a3f9df64392197733135eb72b69c24"}
Sep 18 17:33:20.147577 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:20.147565    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21a8e07986debbdb0bb0eda643bad9acb4a3f9df64392197733135eb72b69c24"
Sep 18 17:33:22.519618 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:22.519581    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66efa6-587f5466b6-vdf5c"]
Sep 18 17:33:22.520109 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:22.519990    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="246181d7-28f4-453c-bd85-01c6d8fb6e51" containerName="main"
Sep 18 17:33:22.520109 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:22.520004    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="246181d7-28f4-453c-bd85-01c6d8fb6e51" containerName="main"
Sep 18 17:33:22.520109 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:22.520018    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2319387f-0c5d-4e1c-bcc4-ede05f9956ab" containerName="revoke-keys"
Sep 18 17:33:22.520109 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:22.520024    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2319387f-0c5d-4e1c-bcc4-ede05f9956ab" containerName="revoke-keys"
Sep 18 17:33:22.520109 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:22.520093    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2319387f-0c5d-4e1c-bcc4-ede05f9956ab" containerName="revoke-keys"
Sep 18 17:33:22.520109 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:22.520101    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="246181d7-28f4-453c-bd85-01c6d8fb6e51" containerName="main"
Sep 18 17:33:22.525378 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:22.525349    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66efa6-587f5466b6-vdf5c"
Sep 18 17:33:22.528703 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:22.528675    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-66efa6\""
Sep 18 17:33:22.528856 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:22.528681    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-66efa6-dockercfg-wrxzn\""
Sep 18 17:33:22.532534 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:22.532512    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66efa6-587f5466b6-vdf5c"]
Sep 18 17:33:22.537459 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:22.537434    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-66efa6-7b84ffc8c9-2dzhp"]
Sep 18 17:33:22.541612 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:22.541592    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66efa6-7b84ffc8c9-2dzhp"
Sep 18 17:33:22.550359 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:22.550334    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-66efa6-7b84ffc8c9-2dzhp"]
Sep 18 17:33:22.669600 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:22.669569    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl4ck\" (UniqueName: \"kubernetes.io/projected/f4853ad5-9ed6-4d38-9f94-4b0d085c93f2-kube-api-access-xl4ck\") pod \"payload-processing-e2e-shared-b-w2-66efa6-7b84ffc8c9-2dzhp\" (UID: \"f4853ad5-9ed6-4d38-9f94-4b0d085c93f2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66efa6-7b84ffc8c9-2dzhp"
Sep 18 17:33:22.669767 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:22.669619    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4853ad5-9ed6-4d38-9f94-4b0d085c93f2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-66efa6-7b84ffc8c9-2dzhp\" (UID: \"f4853ad5-9ed6-4d38-9f94-4b0d085c93f2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66efa6-7b84ffc8c9-2dzhp"
Sep 18 17:33:22.669767 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:22.669741    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8194a9e1-5e29-4e70-8899-8631ffb2c2d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-66efa6-587f5466b6-vdf5c\" (UID: \"8194a9e1-5e29-4e70-8899-8631ffb2c2d6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66efa6-587f5466b6-vdf5c"
Sep 18 17:33:22.669843 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:22.669773    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvh7v\" (UniqueName: \"kubernetes.io/projected/8194a9e1-5e29-4e70-8899-8631ffb2c2d6-kube-api-access-nvh7v\") pod \"payload-pre-processing-e2e-shared-b-w2-66efa6-587f5466b6-vdf5c\" (UID: \"8194a9e1-5e29-4e70-8899-8631ffb2c2d6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66efa6-587f5466b6-vdf5c"
Sep 18 17:33:22.770481 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:22.770394    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8194a9e1-5e29-4e70-8899-8631ffb2c2d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-66efa6-587f5466b6-vdf5c\" (UID: \"8194a9e1-5e29-4e70-8899-8631ffb2c2d6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66efa6-587f5466b6-vdf5c"
Sep 18 17:33:22.770481 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:22.770432    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvh7v\" (UniqueName: \"kubernetes.io/projected/8194a9e1-5e29-4e70-8899-8631ffb2c2d6-kube-api-access-nvh7v\") pod \"payload-pre-processing-e2e-shared-b-w2-66efa6-587f5466b6-vdf5c\" (UID: \"8194a9e1-5e29-4e70-8899-8631ffb2c2d6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66efa6-587f5466b6-vdf5c"
Sep 18 17:33:22.770481 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:22.770467    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl4ck\" (UniqueName: \"kubernetes.io/projected/f4853ad5-9ed6-4d38-9f94-4b0d085c93f2-kube-api-access-xl4ck\") pod \"payload-processing-e2e-shared-b-w2-66efa6-7b84ffc8c9-2dzhp\" (UID: \"f4853ad5-9ed6-4d38-9f94-4b0d085c93f2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66efa6-7b84ffc8c9-2dzhp"
Sep 18 17:33:22.770709 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:22.770497    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4853ad5-9ed6-4d38-9f94-4b0d085c93f2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-66efa6-7b84ffc8c9-2dzhp\" (UID: \"f4853ad5-9ed6-4d38-9f94-4b0d085c93f2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66efa6-7b84ffc8c9-2dzhp"
Sep 18 17:33:22.771186 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:22.771168    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4853ad5-9ed6-4d38-9f94-4b0d085c93f2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-66efa6-7b84ffc8c9-2dzhp\" (UID: \"f4853ad5-9ed6-4d38-9f94-4b0d085c93f2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66efa6-7b84ffc8c9-2dzhp"
Sep 18 17:33:22.771253 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:22.771237    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8194a9e1-5e29-4e70-8899-8631ffb2c2d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-66efa6-587f5466b6-vdf5c\" (UID: \"8194a9e1-5e29-4e70-8899-8631ffb2c2d6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66efa6-587f5466b6-vdf5c"
Sep 18 17:33:22.779751 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:22.779724    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl4ck\" (UniqueName: \"kubernetes.io/projected/f4853ad5-9ed6-4d38-9f94-4b0d085c93f2-kube-api-access-xl4ck\") pod \"payload-processing-e2e-shared-b-w2-66efa6-7b84ffc8c9-2dzhp\" (UID: \"f4853ad5-9ed6-4d38-9f94-4b0d085c93f2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66efa6-7b84ffc8c9-2dzhp"
Sep 18 17:33:22.779881 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:22.779808    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvh7v\" (UniqueName: \"kubernetes.io/projected/8194a9e1-5e29-4e70-8899-8631ffb2c2d6-kube-api-access-nvh7v\") pod \"payload-pre-processing-e2e-shared-b-w2-66efa6-587f5466b6-vdf5c\" (UID: \"8194a9e1-5e29-4e70-8899-8631ffb2c2d6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66efa6-587f5466b6-vdf5c"
Sep 18 17:33:22.840359 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:22.840326    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66efa6-587f5466b6-vdf5c"
Sep 18 17:33:22.852618 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:22.852593    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66efa6-7b84ffc8c9-2dzhp"
Sep 18 17:33:23.123391 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:23.123355    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66efa6-587f5466b6-vdf5c"]
Sep 18 17:33:23.124784 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:33:23.124760    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8194a9e1_5e29_4e70_8899_8631ffb2c2d6.slice/crio-ad8fe82df484ec3fafe3d270773b83281db00394e7cd92167233bd73c0faeebe WatchSource:0}: Error finding container ad8fe82df484ec3fafe3d270773b83281db00394e7cd92167233bd73c0faeebe: Status 404 returned error can't find the container with id ad8fe82df484ec3fafe3d270773b83281db00394e7cd92167233bd73c0faeebe
Sep 18 17:33:23.130065 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:23.130036    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-66efa6-7b84ffc8c9-2dzhp"]
Sep 18 17:33:23.131345 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:33:23.131321    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4853ad5_9ed6_4d38_9f94_4b0d085c93f2.slice/crio-db92329bd3663019b1ce93415cc8528a1e354f93ff12cfec0e02023dfb11e7fb WatchSource:0}: Error finding container db92329bd3663019b1ce93415cc8528a1e354f93ff12cfec0e02023dfb11e7fb: Status 404 returned error can't find the container with id db92329bd3663019b1ce93415cc8528a1e354f93ff12cfec0e02023dfb11e7fb
Sep 18 17:33:23.165376 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:23.165352    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66efa6-587f5466b6-vdf5c" event={"ID":"8194a9e1-5e29-4e70-8899-8631ffb2c2d6","Type":"ContainerStarted","Data":"ad8fe82df484ec3fafe3d270773b83281db00394e7cd92167233bd73c0faeebe"}
Sep 18 17:33:23.166497 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:23.166474    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66efa6-7b84ffc8c9-2dzhp" event={"ID":"f4853ad5-9ed6-4d38-9f94-4b0d085c93f2","Type":"ContainerStarted","Data":"db92329bd3663019b1ce93415cc8528a1e354f93ff12cfec0e02023dfb11e7fb"}
Sep 18 17:33:24.173679 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:24.173627    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66efa6-7b84ffc8c9-2dzhp" event={"ID":"f4853ad5-9ed6-4d38-9f94-4b0d085c93f2","Type":"ContainerStarted","Data":"93d6423bc2f458a78481fc54ea604e505f95f791db71766faf9a309ef2decaf8"}
Sep 18 17:33:24.174163 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:24.173707    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66efa6-7b84ffc8c9-2dzhp"
Sep 18 17:33:24.175374 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:24.175346    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66efa6-587f5466b6-vdf5c" event={"ID":"8194a9e1-5e29-4e70-8899-8631ffb2c2d6","Type":"ContainerStarted","Data":"0bf1cba711c885b74bb609a870b06ad913b9d69ef7a5a2a1ac677b9ca563acea"}
Sep 18 17:33:24.175508 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:24.175407    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66efa6-587f5466b6-vdf5c"
Sep 18 17:33:24.191390 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:24.191333    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66efa6-7b84ffc8c9-2dzhp" podStartSLOduration=2.191317165 podStartE2EDuration="2.191317165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:33:24.189992719 +0000 UTC m=+1116.883920247" watchObservedRunningTime="2026-09-18 17:33:24.191317165 +0000 UTC m=+1116.885244692"
Sep 18 17:33:24.206512 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:24.206457    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66efa6-587f5466b6-vdf5c" podStartSLOduration=2.20643988 podStartE2EDuration="2.20643988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:33:24.206092422 +0000 UTC m=+1116.900019979" watchObservedRunningTime="2026-09-18 17:33:24.20643988 +0000 UTC m=+1116.900367408"
Sep 18 17:33:27.235884 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:27.235557    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-636b80-f85c7966c-vp6zj"]
Sep 18 17:33:27.236364 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:27.236201    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-636b80-f85c7966c-vp6zj" podUID="18c90c69-6f7a-4738-a2d3-01ad0cc83d38" containerName="payload-processing" containerID="cri-o://7216a6fb8daea23c8c148f195cdb2f02ba2ddaca25b948cf5f333063025539e9" gracePeriod=30
Sep 18 17:33:27.240664 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:27.240634    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-636b80-94f555b66-ws742"]
Sep 18 17:33:27.241073 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:27.241020    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636b80-94f555b66-ws742" podUID="1b4e16b6-d717-4a39-a879-d34d21f70c67" containerName="payload-pre-processing" containerID="cri-o://f4a5fa741521cdf238deea41b0ab44ccf3e210e5499bc3a58e3e40baf3191519" gracePeriod=30
Sep 18 17:33:27.380779 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:27.380744    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636b80-v4km9"]
Sep 18 17:33:27.383770 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:27.383739    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636b80-v4km9"]
Sep 18 17:33:27.517363 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:27.517340    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-636b80-f85c7966c-vp6zj"
Sep 18 17:33:27.520143 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:27.520123    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636b80-94f555b66-ws742"
Sep 18 17:33:27.620927 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:27.620896    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b4e16b6-d717-4a39-a879-d34d21f70c67-plugins-config-volume\") pod \"1b4e16b6-d717-4a39-a879-d34d21f70c67\" (UID: \"1b4e16b6-d717-4a39-a879-d34d21f70c67\") "
Sep 18 17:33:27.621091 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:27.621010    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18c90c69-6f7a-4738-a2d3-01ad0cc83d38-plugins-config-volume\") pod \"18c90c69-6f7a-4738-a2d3-01ad0cc83d38\" (UID: \"18c90c69-6f7a-4738-a2d3-01ad0cc83d38\") "
Sep 18 17:33:27.621091 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:27.621041    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pz4cr\" (UniqueName: \"kubernetes.io/projected/1b4e16b6-d717-4a39-a879-d34d21f70c67-kube-api-access-pz4cr\") pod \"1b4e16b6-d717-4a39-a879-d34d21f70c67\" (UID: \"1b4e16b6-d717-4a39-a879-d34d21f70c67\") "
Sep 18 17:33:27.621091 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:27.621069    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnp24\" (UniqueName: \"kubernetes.io/projected/18c90c69-6f7a-4738-a2d3-01ad0cc83d38-kube-api-access-lnp24\") pod \"18c90c69-6f7a-4738-a2d3-01ad0cc83d38\" (UID: \"18c90c69-6f7a-4738-a2d3-01ad0cc83d38\") "
Sep 18 17:33:27.621387 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:27.621351    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b4e16b6-d717-4a39-a879-d34d21f70c67-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1b4e16b6-d717-4a39-a879-d34d21f70c67" (UID: "1b4e16b6-d717-4a39-a879-d34d21f70c67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:33:27.621464 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:27.621370    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18c90c69-6f7a-4738-a2d3-01ad0cc83d38-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "18c90c69-6f7a-4738-a2d3-01ad0cc83d38" (UID: "18c90c69-6f7a-4738-a2d3-01ad0cc83d38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:33:27.623172 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:27.623149    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18c90c69-6f7a-4738-a2d3-01ad0cc83d38-kube-api-access-lnp24" (OuterVolumeSpecName: "kube-api-access-lnp24") pod "18c90c69-6f7a-4738-a2d3-01ad0cc83d38" (UID: "18c90c69-6f7a-4738-a2d3-01ad0cc83d38"). InnerVolumeSpecName "kube-api-access-lnp24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:33:27.623256 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:27.623195    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b4e16b6-d717-4a39-a879-d34d21f70c67-kube-api-access-pz4cr" (OuterVolumeSpecName: "kube-api-access-pz4cr") pod "1b4e16b6-d717-4a39-a879-d34d21f70c67" (UID: "1b4e16b6-d717-4a39-a879-d34d21f70c67"). InnerVolumeSpecName "kube-api-access-pz4cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:33:27.722022 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:27.721983    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz4cr\" (UniqueName: \"kubernetes.io/projected/1b4e16b6-d717-4a39-a879-d34d21f70c67-kube-api-access-pz4cr\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:33:27.722022 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:27.722014    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnp24\" (UniqueName: \"kubernetes.io/projected/18c90c69-6f7a-4738-a2d3-01ad0cc83d38-kube-api-access-lnp24\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:33:27.722022 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:27.722023    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b4e16b6-d717-4a39-a879-d34d21f70c67-plugins-config-volume\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:33:27.722022 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:27.722032    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18c90c69-6f7a-4738-a2d3-01ad0cc83d38-plugins-config-volume\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:33:27.764725 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:27.764648    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2319387f-0c5d-4e1c-bcc4-ede05f9956ab" path="/var/lib/kubelet/pods/2319387f-0c5d-4e1c-bcc4-ede05f9956ab/volumes"
Sep 18 17:33:28.199405 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:28.199310    2569 generic.go:356] "Generic (PLEG): container finished" podID="1b4e16b6-d717-4a39-a879-d34d21f70c67" containerID="f4a5fa741521cdf238deea41b0ab44ccf3e210e5499bc3a58e3e40baf3191519" exitCode=0
Sep 18 17:33:28.199405 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:28.199354    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636b80-94f555b66-ws742"
Sep 18 17:33:28.199644 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:28.199401    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636b80-94f555b66-ws742" event={"ID":"1b4e16b6-d717-4a39-a879-d34d21f70c67","Type":"ContainerDied","Data":"f4a5fa741521cdf238deea41b0ab44ccf3e210e5499bc3a58e3e40baf3191519"}
Sep 18 17:33:28.199644 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:28.199459    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636b80-94f555b66-ws742" event={"ID":"1b4e16b6-d717-4a39-a879-d34d21f70c67","Type":"ContainerDied","Data":"a1fbccc4b764a53b502b21c834eef517696a927893d56389314a93ac3322195a"}
Sep 18 17:33:28.199644 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:28.199490    2569 scope.go:117] "RemoveContainer" containerID="f4a5fa741521cdf238deea41b0ab44ccf3e210e5499bc3a58e3e40baf3191519"
Sep 18 17:33:28.200881 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:28.200844    2569 generic.go:356] "Generic (PLEG): container finished" podID="18c90c69-6f7a-4738-a2d3-01ad0cc83d38" containerID="7216a6fb8daea23c8c148f195cdb2f02ba2ddaca25b948cf5f333063025539e9" exitCode=0
Sep 18 17:33:28.200988 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:28.200907    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-636b80-f85c7966c-vp6zj" event={"ID":"18c90c69-6f7a-4738-a2d3-01ad0cc83d38","Type":"ContainerDied","Data":"7216a6fb8daea23c8c148f195cdb2f02ba2ddaca25b948cf5f333063025539e9"}
Sep 18 17:33:28.200988 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:28.200957    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-636b80-f85c7966c-vp6zj" event={"ID":"18c90c69-6f7a-4738-a2d3-01ad0cc83d38","Type":"ContainerDied","Data":"24082803b477e6e571872d0e43c59264ac0a55dbc604277fde674c79b92226eb"}
Sep 18 17:33:28.200988 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:28.200910    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-636b80-f85c7966c-vp6zj"
Sep 18 17:33:28.208854 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:28.208838    2569 scope.go:117] "RemoveContainer" containerID="f4a5fa741521cdf238deea41b0ab44ccf3e210e5499bc3a58e3e40baf3191519"
Sep 18 17:33:28.209149 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:33:28.209130    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4a5fa741521cdf238deea41b0ab44ccf3e210e5499bc3a58e3e40baf3191519\": container with ID starting with f4a5fa741521cdf238deea41b0ab44ccf3e210e5499bc3a58e3e40baf3191519 not found: ID does not exist" containerID="f4a5fa741521cdf238deea41b0ab44ccf3e210e5499bc3a58e3e40baf3191519"
Sep 18 17:33:28.209246 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:28.209159    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4a5fa741521cdf238deea41b0ab44ccf3e210e5499bc3a58e3e40baf3191519"} err="failed to get container status \"f4a5fa741521cdf238deea41b0ab44ccf3e210e5499bc3a58e3e40baf3191519\": rpc error: code = NotFound desc = could not find container \"f4a5fa741521cdf238deea41b0ab44ccf3e210e5499bc3a58e3e40baf3191519\": container with ID starting with f4a5fa741521cdf238deea41b0ab44ccf3e210e5499bc3a58e3e40baf3191519 not found: ID does not exist"
Sep 18 17:33:28.209246 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:28.209182    2569 scope.go:117] "RemoveContainer" containerID="7216a6fb8daea23c8c148f195cdb2f02ba2ddaca25b948cf5f333063025539e9"
Sep 18 17:33:28.217245 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:28.217226    2569 scope.go:117] "RemoveContainer" containerID="7216a6fb8daea23c8c148f195cdb2f02ba2ddaca25b948cf5f333063025539e9"
Sep 18 17:33:28.217499 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:33:28.217481    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7216a6fb8daea23c8c148f195cdb2f02ba2ddaca25b948cf5f333063025539e9\": container with ID starting with 7216a6fb8daea23c8c148f195cdb2f02ba2ddaca25b948cf5f333063025539e9 not found: ID does not exist" containerID="7216a6fb8daea23c8c148f195cdb2f02ba2ddaca25b948cf5f333063025539e9"
Sep 18 17:33:28.217560 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:28.217508    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7216a6fb8daea23c8c148f195cdb2f02ba2ddaca25b948cf5f333063025539e9"} err="failed to get container status \"7216a6fb8daea23c8c148f195cdb2f02ba2ddaca25b948cf5f333063025539e9\": rpc error: code = NotFound desc = could not find container \"7216a6fb8daea23c8c148f195cdb2f02ba2ddaca25b948cf5f333063025539e9\": container with ID starting with 7216a6fb8daea23c8c148f195cdb2f02ba2ddaca25b948cf5f333063025539e9 not found: ID does not exist"
Sep 18 17:33:28.221756 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:28.221734    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-636b80-94f555b66-ws742"]
Sep 18 17:33:28.224904 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:28.224884    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-636b80-94f555b66-ws742"]
Sep 18 17:33:28.234246 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:28.234228    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-636b80-f85c7966c-vp6zj"]
Sep 18 17:33:28.239564 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:28.239546    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-636b80-f85c7966c-vp6zj"]
Sep 18 17:33:29.765884 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:29.765843    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18c90c69-6f7a-4738-a2d3-01ad0cc83d38" path="/var/lib/kubelet/pods/18c90c69-6f7a-4738-a2d3-01ad0cc83d38/volumes"
Sep 18 17:33:29.766266 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:29.766216    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b4e16b6-d717-4a39-a879-d34d21f70c67" path="/var/lib/kubelet/pods/1b4e16b6-d717-4a39-a879-d34d21f70c67/volumes"
Sep 18 17:33:32.400206 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:32.400168    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz"]
Sep 18 17:33:32.400570 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:32.400525    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b4e16b6-d717-4a39-a879-d34d21f70c67" containerName="payload-pre-processing"
Sep 18 17:33:32.400570 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:32.400535    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b4e16b6-d717-4a39-a879-d34d21f70c67" containerName="payload-pre-processing"
Sep 18 17:33:32.400570 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:32.400545    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18c90c69-6f7a-4738-a2d3-01ad0cc83d38" containerName="payload-processing"
Sep 18 17:33:32.400570 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:32.400551    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="18c90c69-6f7a-4738-a2d3-01ad0cc83d38" containerName="payload-processing"
Sep 18 17:33:32.400704 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:32.400619    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b4e16b6-d717-4a39-a879-d34d21f70c67" containerName="payload-pre-processing"
Sep 18 17:33:32.400704 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:32.400627    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="18c90c69-6f7a-4738-a2d3-01ad0cc83d38" containerName="payload-processing"
Sep 18 17:33:32.405302 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:32.405286    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz"
Sep 18 17:33:32.408712 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:32.408692    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-231bbc\"/\"openshift-service-ca.crt\""
Sep 18 17:33:32.408856 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:32.408751    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-231bbc\"/\"kube-root-ca.crt\""
Sep 18 17:33:32.408856 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:32.408805    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-231bbc\"/\"default-dockercfg-ngjg6\""
Sep 18 17:33:32.408856 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:32.408828    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-231bbc\"/\"noten-model-231bbc-kserve-self-signed-certs\""
Sep 18 17:33:32.416334 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:32.416308    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz"]
Sep 18 17:33:32.468000 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:32.467975    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55cb8f4c-9074-4f68-9a4d-7849cea31624-home\") pod \"noten-model-231bbc-kserve-69677879bc-7m2tz\" (UID: \"55cb8f4c-9074-4f68-9a4d-7849cea31624\") " pod="ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz"
Sep 18 17:33:32.468130 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:32.468010    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55cb8f4c-9074-4f68-9a4d-7849cea31624-tmp-dir\") pod \"noten-model-231bbc-kserve-69677879bc-7m2tz\" (UID: \"55cb8f4c-9074-4f68-9a4d-7849cea31624\") " pod="ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz"
Sep 18 17:33:32.468130 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:32.468027    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55cb8f4c-9074-4f68-9a4d-7849cea31624-tls-certs\") pod \"noten-model-231bbc-kserve-69677879bc-7m2tz\" (UID: \"55cb8f4c-9074-4f68-9a4d-7849cea31624\") " pod="ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz"
Sep 18 17:33:32.468203 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:32.468147    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55cb8f4c-9074-4f68-9a4d-7849cea31624-model-cache\") pod \"noten-model-231bbc-kserve-69677879bc-7m2tz\" (UID: \"55cb8f4c-9074-4f68-9a4d-7849cea31624\") " pod="ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz"
Sep 18 17:33:32.468203 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:32.468183    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrbnf\" (UniqueName: \"kubernetes.io/projected/55cb8f4c-9074-4f68-9a4d-7849cea31624-kube-api-access-hrbnf\") pod \"noten-model-231bbc-kserve-69677879bc-7m2tz\" (UID: \"55cb8f4c-9074-4f68-9a4d-7849cea31624\") " pod="ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz"
Sep 18 17:33:32.468273 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:32.468223    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55cb8f4c-9074-4f68-9a4d-7849cea31624-dshm\") pod \"noten-model-231bbc-kserve-69677879bc-7m2tz\" (UID: \"55cb8f4c-9074-4f68-9a4d-7849cea31624\") " pod="ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz"
Sep 18 17:33:32.568773 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:32.568738    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55cb8f4c-9074-4f68-9a4d-7849cea31624-tmp-dir\") pod \"noten-model-231bbc-kserve-69677879bc-7m2tz\" (UID: \"55cb8f4c-9074-4f68-9a4d-7849cea31624\") " pod="ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz"
Sep 18 17:33:32.568773 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:32.568773    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55cb8f4c-9074-4f68-9a4d-7849cea31624-tls-certs\") pod \"noten-model-231bbc-kserve-69677879bc-7m2tz\" (UID: \"55cb8f4c-9074-4f68-9a4d-7849cea31624\") " pod="ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz"
Sep 18 17:33:32.568990 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:32.568829    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55cb8f4c-9074-4f68-9a4d-7849cea31624-model-cache\") pod \"noten-model-231bbc-kserve-69677879bc-7m2tz\" (UID: \"55cb8f4c-9074-4f68-9a4d-7849cea31624\") " pod="ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz"
Sep 18 17:33:32.568990 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:32.568857    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrbnf\" (UniqueName: \"kubernetes.io/projected/55cb8f4c-9074-4f68-9a4d-7849cea31624-kube-api-access-hrbnf\") pod \"noten-model-231bbc-kserve-69677879bc-7m2tz\" (UID: \"55cb8f4c-9074-4f68-9a4d-7849cea31624\") " pod="ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz"
Sep 18 17:33:32.568990 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:32.568924    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55cb8f4c-9074-4f68-9a4d-7849cea31624-dshm\") pod \"noten-model-231bbc-kserve-69677879bc-7m2tz\" (UID: \"55cb8f4c-9074-4f68-9a4d-7849cea31624\") " pod="ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz"
Sep 18 17:33:32.568990 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:32.568955    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55cb8f4c-9074-4f68-9a4d-7849cea31624-home\") pod \"noten-model-231bbc-kserve-69677879bc-7m2tz\" (UID: \"55cb8f4c-9074-4f68-9a4d-7849cea31624\") " pod="ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz"
Sep 18 17:33:32.569200 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:32.569173    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55cb8f4c-9074-4f68-9a4d-7849cea31624-tmp-dir\") pod \"noten-model-231bbc-kserve-69677879bc-7m2tz\" (UID: \"55cb8f4c-9074-4f68-9a4d-7849cea31624\") " pod="ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz"
Sep 18 17:33:32.569265 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:32.569222    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55cb8f4c-9074-4f68-9a4d-7849cea31624-model-cache\") pod \"noten-model-231bbc-kserve-69677879bc-7m2tz\" (UID: \"55cb8f4c-9074-4f68-9a4d-7849cea31624\") " pod="ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz"
Sep 18 17:33:32.569347 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:32.569327    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55cb8f4c-9074-4f68-9a4d-7849cea31624-home\") pod \"noten-model-231bbc-kserve-69677879bc-7m2tz\" (UID: \"55cb8f4c-9074-4f68-9a4d-7849cea31624\") " pod="ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz"
Sep 18 17:33:32.571176 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:32.571154    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55cb8f4c-9074-4f68-9a4d-7849cea31624-dshm\") pod \"noten-model-231bbc-kserve-69677879bc-7m2tz\" (UID: \"55cb8f4c-9074-4f68-9a4d-7849cea31624\") " pod="ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz"
Sep 18 17:33:32.571300 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:32.571283    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55cb8f4c-9074-4f68-9a4d-7849cea31624-tls-certs\") pod \"noten-model-231bbc-kserve-69677879bc-7m2tz\" (UID: \"55cb8f4c-9074-4f68-9a4d-7849cea31624\") " pod="ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz"
Sep 18 17:33:32.579558 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:32.579536    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrbnf\" (UniqueName: \"kubernetes.io/projected/55cb8f4c-9074-4f68-9a4d-7849cea31624-kube-api-access-hrbnf\") pod \"noten-model-231bbc-kserve-69677879bc-7m2tz\" (UID: \"55cb8f4c-9074-4f68-9a4d-7849cea31624\") " pod="ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz"
Sep 18 17:33:32.714736 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:32.714654    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz"
Sep 18 17:33:32.995200 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:32.995169    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz"]
Sep 18 17:33:32.997694 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:33:32.997660    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55cb8f4c_9074_4f68_9a4d_7849cea31624.slice/crio-5c6a2f78b19c6d0e7b67a7dbcf3933509d30c16912d161c3f12b4803395780fd WatchSource:0}: Error finding container 5c6a2f78b19c6d0e7b67a7dbcf3933509d30c16912d161c3f12b4803395780fd: Status 404 returned error can't find the container with id 5c6a2f78b19c6d0e7b67a7dbcf3933509d30c16912d161c3f12b4803395780fd
Sep 18 17:33:33.235557 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:33.235510    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz" event={"ID":"55cb8f4c-9074-4f68-9a4d-7849cea31624","Type":"ContainerStarted","Data":"dacc38d5fdaa9506c01c34b43eb2547b99a0d8f8a7b888b7f8280083007d41de"}
Sep 18 17:33:33.235557 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:33.235559    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz" event={"ID":"55cb8f4c-9074-4f68-9a4d-7849cea31624","Type":"ContainerStarted","Data":"5c6a2f78b19c6d0e7b67a7dbcf3933509d30c16912d161c3f12b4803395780fd"}
Sep 18 17:33:33.254164 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:33.254052    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz" podStartSLOduration=1.254036004 podStartE2EDuration="1.254036004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:33:33.252825581 +0000 UTC m=+1125.946753134" watchObservedRunningTime="2026-09-18 17:33:33.254036004 +0000 UTC m=+1125.947963532"
Sep 18 17:33:35.183747 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:35.183715    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66efa6-7b84ffc8c9-2dzhp"
Sep 18 17:33:35.184219 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:35.183890    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66efa6-587f5466b6-vdf5c"
Sep 18 17:33:42.715775 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:42.715743    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz"
Sep 18 17:33:42.716151 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:42.715788    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz"
Sep 18 17:33:42.728302 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:42.728279    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz"
Sep 18 17:33:43.306272 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:43.306245    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz"
Sep 18 17:33:46.032257 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:46.032213    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aa65a-29829210-b8g9l"]
Sep 18 17:33:46.035160 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:46.035133    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2aa65a-29829210-b8g9l"]
Sep 18 17:33:46.061473 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:46.061443    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2aa65a-59c986568b-bt4sb"]
Sep 18 17:33:46.061690 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:46.061661    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aa65a-59c986568b-bt4sb" podUID="087178e2-6dda-463a-89f8-66c246d90692" containerName="payload-pre-processing" containerID="cri-o://612ff215393dbb06c297ceeb2f254bdfa9cf2e7daf78275f0e698320d1c5c641" gracePeriod=30
Sep 18 17:33:46.068378 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:46.068342    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aa65a-59c986568b-bt4sb" podUID="087178e2-6dda-463a-89f8-66c246d90692" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.48:9004: connect: connection refused"
Sep 18 17:33:46.311753 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:46.311724    2569 generic.go:356] "Generic (PLEG): container finished" podID="087178e2-6dda-463a-89f8-66c246d90692" containerID="612ff215393dbb06c297ceeb2f254bdfa9cf2e7daf78275f0e698320d1c5c641" exitCode=0
Sep 18 17:33:46.311889 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:46.311793    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aa65a-59c986568b-bt4sb" event={"ID":"087178e2-6dda-463a-89f8-66c246d90692","Type":"ContainerDied","Data":"612ff215393dbb06c297ceeb2f254bdfa9cf2e7daf78275f0e698320d1c5c641"}
Sep 18 17:33:46.311889 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:46.311841    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aa65a-59c986568b-bt4sb" event={"ID":"087178e2-6dda-463a-89f8-66c246d90692","Type":"ContainerDied","Data":"0202a8b0417eff249fa8e790517d8c3f842e2be94164ef9951a354786afed7b3"}
Sep 18 17:33:46.311889 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:46.311856    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0202a8b0417eff249fa8e790517d8c3f842e2be94164ef9951a354786afed7b3"
Sep 18 17:33:46.316360 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:46.316343    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aa65a-59c986568b-bt4sb"
Sep 18 17:33:46.393917 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:46.393881    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hv5w\" (UniqueName: \"kubernetes.io/projected/087178e2-6dda-463a-89f8-66c246d90692-kube-api-access-2hv5w\") pod \"087178e2-6dda-463a-89f8-66c246d90692\" (UID: \"087178e2-6dda-463a-89f8-66c246d90692\") "
Sep 18 17:33:46.394085 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:46.393986    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/087178e2-6dda-463a-89f8-66c246d90692-plugins-config-volume\") pod \"087178e2-6dda-463a-89f8-66c246d90692\" (UID: \"087178e2-6dda-463a-89f8-66c246d90692\") "
Sep 18 17:33:46.394335 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:46.394314    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/087178e2-6dda-463a-89f8-66c246d90692-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "087178e2-6dda-463a-89f8-66c246d90692" (UID: "087178e2-6dda-463a-89f8-66c246d90692"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:33:46.395934 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:46.395909    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/087178e2-6dda-463a-89f8-66c246d90692-kube-api-access-2hv5w" (OuterVolumeSpecName: "kube-api-access-2hv5w") pod "087178e2-6dda-463a-89f8-66c246d90692" (UID: "087178e2-6dda-463a-89f8-66c246d90692"). InnerVolumeSpecName "kube-api-access-2hv5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:33:46.495234 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:46.495201    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/087178e2-6dda-463a-89f8-66c246d90692-plugins-config-volume\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:33:46.495234 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:46.495230    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hv5w\" (UniqueName: \"kubernetes.io/projected/087178e2-6dda-463a-89f8-66c246d90692-kube-api-access-2hv5w\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:33:47.317508 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:47.317471    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2aa65a-59c986568b-bt4sb"
Sep 18 17:33:47.339288 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:47.339255    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2aa65a-59c986568b-bt4sb"]
Sep 18 17:33:47.342913 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:47.342886    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2aa65a-59c986568b-bt4sb"]
Sep 18 17:33:47.765798 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:47.765767    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="087178e2-6dda-463a-89f8-66c246d90692" path="/var/lib/kubelet/pods/087178e2-6dda-463a-89f8-66c246d90692/volumes"
Sep 18 17:33:47.766085 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:47.766071    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53add21d-9226-4dab-ae83-80ea86ba930b" path="/var/lib/kubelet/pods/53add21d-9226-4dab-ae83-80ea86ba930b/volumes"
Sep 18 17:33:47.853728 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:47.853699    2569 scope.go:117] "RemoveContainer" containerID="612ff215393dbb06c297ceeb2f254bdfa9cf2e7daf78275f0e698320d1c5c641"
Sep 18 17:33:47.862471 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:33:47.862451    2569 scope.go:117] "RemoveContainer" containerID="075eda9ebcc4a20464c3057a4303d70138078855e10b31b077084c250a90c170"
Sep 18 17:34:04.531751 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:04.531711    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b68d6946-66b4c6b97-7f4jv"]
Sep 18 17:34:04.532269 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:04.532086    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="087178e2-6dda-463a-89f8-66c246d90692" containerName="payload-pre-processing"
Sep 18 17:34:04.532269 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:04.532099    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="087178e2-6dda-463a-89f8-66c246d90692" containerName="payload-pre-processing"
Sep 18 17:34:04.532269 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:04.532176    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="087178e2-6dda-463a-89f8-66c246d90692" containerName="payload-pre-processing"
Sep 18 17:34:04.536879 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:04.536843    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b68d6946-66b4c6b97-7f4jv"
Sep 18 17:34:04.540647 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:04.540622    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-b68d6946-dockercfg-zsddt\""
Sep 18 17:34:04.541024 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:04.540703    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b68d6946\""
Sep 18 17:34:04.543912 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:04.543888    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b68d6946-66b4c6b97-7f4jv"]
Sep 18 17:34:04.672009 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:04.671972    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmr87\" (UniqueName: \"kubernetes.io/projected/2048ef0c-bc30-4d73-a677-b71149532417-kube-api-access-nmr87\") pod \"payload-processing-e2e-mt-b68d6946-66b4c6b97-7f4jv\" (UID: \"2048ef0c-bc30-4d73-a677-b71149532417\") " pod="openshift-ingress/payload-processing-e2e-mt-b68d6946-66b4c6b97-7f4jv"
Sep 18 17:34:04.672176 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:04.672037    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2048ef0c-bc30-4d73-a677-b71149532417-plugins-config-volume\") pod \"payload-processing-e2e-mt-b68d6946-66b4c6b97-7f4jv\" (UID: \"2048ef0c-bc30-4d73-a677-b71149532417\") " pod="openshift-ingress/payload-processing-e2e-mt-b68d6946-66b4c6b97-7f4jv"
Sep 18 17:34:04.773134 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:04.773098    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmr87\" (UniqueName: \"kubernetes.io/projected/2048ef0c-bc30-4d73-a677-b71149532417-kube-api-access-nmr87\") pod \"payload-processing-e2e-mt-b68d6946-66b4c6b97-7f4jv\" (UID: \"2048ef0c-bc30-4d73-a677-b71149532417\") " pod="openshift-ingress/payload-processing-e2e-mt-b68d6946-66b4c6b97-7f4jv"
Sep 18 17:34:04.773306 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:04.773158    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2048ef0c-bc30-4d73-a677-b71149532417-plugins-config-volume\") pod \"payload-processing-e2e-mt-b68d6946-66b4c6b97-7f4jv\" (UID: \"2048ef0c-bc30-4d73-a677-b71149532417\") " pod="openshift-ingress/payload-processing-e2e-mt-b68d6946-66b4c6b97-7f4jv"
Sep 18 17:34:04.773830 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:04.773813    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2048ef0c-bc30-4d73-a677-b71149532417-plugins-config-volume\") pod \"payload-processing-e2e-mt-b68d6946-66b4c6b97-7f4jv\" (UID: \"2048ef0c-bc30-4d73-a677-b71149532417\") " pod="openshift-ingress/payload-processing-e2e-mt-b68d6946-66b4c6b97-7f4jv"
Sep 18 17:34:04.788760 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:04.788694    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmr87\" (UniqueName: \"kubernetes.io/projected/2048ef0c-bc30-4d73-a677-b71149532417-kube-api-access-nmr87\") pod \"payload-processing-e2e-mt-b68d6946-66b4c6b97-7f4jv\" (UID: \"2048ef0c-bc30-4d73-a677-b71149532417\") " pod="openshift-ingress/payload-processing-e2e-mt-b68d6946-66b4c6b97-7f4jv"
Sep 18 17:34:04.848249 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:04.848213    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b68d6946-66b4c6b97-7f4jv"
Sep 18 17:34:05.123338 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:05.123207    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b68d6946-66b4c6b97-7f4jv"]
Sep 18 17:34:05.126177 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:34:05.126142    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2048ef0c_bc30_4d73_a677_b71149532417.slice/crio-76b8a5734666c8dc1a59ce77e272e3ee02ea237645ac75b1b34acb6c81c71807 WatchSource:0}: Error finding container 76b8a5734666c8dc1a59ce77e272e3ee02ea237645ac75b1b34acb6c81c71807: Status 404 returned error can't find the container with id 76b8a5734666c8dc1a59ce77e272e3ee02ea237645ac75b1b34acb6c81c71807
Sep 18 17:34:05.431214 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:05.431116    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b68d6946-66b4c6b97-7f4jv" event={"ID":"2048ef0c-bc30-4d73-a677-b71149532417","Type":"ContainerStarted","Data":"1de479b67844fc2e65d028918dc04e7f944560a444fec4e5fe2a9f0de511a615"}
Sep 18 17:34:05.431214 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:05.431200    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b68d6946-66b4c6b97-7f4jv" event={"ID":"2048ef0c-bc30-4d73-a677-b71149532417","Type":"ContainerStarted","Data":"76b8a5734666c8dc1a59ce77e272e3ee02ea237645ac75b1b34acb6c81c71807"}
Sep 18 17:34:05.431441 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:05.431345    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-b68d6946-66b4c6b97-7f4jv"
Sep 18 17:34:05.449727 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:05.449677    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-b68d6946-66b4c6b97-7f4jv" podStartSLOduration=1.449664565 podStartE2EDuration="1.449664565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:34:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:34:05.447193129 +0000 UTC m=+1158.141120675" watchObservedRunningTime="2026-09-18 17:34:05.449664565 +0000 UTC m=+1158.143592141"
Sep 18 17:34:05.597941 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:05.597904    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-eeab6e-645b4665b7-h78xd"]
Sep 18 17:34:05.601456 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:05.601432    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eeab6e-645b4665b7-h78xd"
Sep 18 17:34:05.604002 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:05.603978    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-eeab6e-dockercfg-rddmm\""
Sep 18 17:34:05.604151 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:05.604020    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-eeab6e\""
Sep 18 17:34:05.611845 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:05.611817    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-eeab6e-645b4665b7-h78xd"]
Sep 18 17:34:05.683127 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:05.683036    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5fd32a3-79f9-4b0a-9b59-a8f6866838c5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-eeab6e-645b4665b7-h78xd\" (UID: \"e5fd32a3-79f9-4b0a-9b59-a8f6866838c5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eeab6e-645b4665b7-h78xd"
Sep 18 17:34:05.683127 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:05.683073    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7trxg\" (UniqueName: \"kubernetes.io/projected/e5fd32a3-79f9-4b0a-9b59-a8f6866838c5-kube-api-access-7trxg\") pod \"payload-processing-e2e-shared-a-w3-eeab6e-645b4665b7-h78xd\" (UID: \"e5fd32a3-79f9-4b0a-9b59-a8f6866838c5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eeab6e-645b4665b7-h78xd"
Sep 18 17:34:05.783929 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:05.783892    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5fd32a3-79f9-4b0a-9b59-a8f6866838c5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-eeab6e-645b4665b7-h78xd\" (UID: \"e5fd32a3-79f9-4b0a-9b59-a8f6866838c5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eeab6e-645b4665b7-h78xd"
Sep 18 17:34:05.783929 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:05.783930    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7trxg\" (UniqueName: \"kubernetes.io/projected/e5fd32a3-79f9-4b0a-9b59-a8f6866838c5-kube-api-access-7trxg\") pod \"payload-processing-e2e-shared-a-w3-eeab6e-645b4665b7-h78xd\" (UID: \"e5fd32a3-79f9-4b0a-9b59-a8f6866838c5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eeab6e-645b4665b7-h78xd"
Sep 18 17:34:05.784574 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:05.784553    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5fd32a3-79f9-4b0a-9b59-a8f6866838c5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-eeab6e-645b4665b7-h78xd\" (UID: \"e5fd32a3-79f9-4b0a-9b59-a8f6866838c5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eeab6e-645b4665b7-h78xd"
Sep 18 17:34:05.792328 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:05.792294    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7trxg\" (UniqueName: \"kubernetes.io/projected/e5fd32a3-79f9-4b0a-9b59-a8f6866838c5-kube-api-access-7trxg\") pod \"payload-processing-e2e-shared-a-w3-eeab6e-645b4665b7-h78xd\" (UID: \"e5fd32a3-79f9-4b0a-9b59-a8f6866838c5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eeab6e-645b4665b7-h78xd"
Sep 18 17:34:05.913343 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:05.913311    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eeab6e-645b4665b7-h78xd"
Sep 18 17:34:06.189190 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:06.189148    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-eeab6e-645b4665b7-h78xd"]
Sep 18 17:34:06.191934 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:34:06.191907    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5fd32a3_79f9_4b0a_9b59_a8f6866838c5.slice/crio-88cc3905ee24853dc74c53afb66fd54793f823c1eaee57e8f2be6ab1361e3a45 WatchSource:0}: Error finding container 88cc3905ee24853dc74c53afb66fd54793f823c1eaee57e8f2be6ab1361e3a45: Status 404 returned error can't find the container with id 88cc3905ee24853dc74c53afb66fd54793f823c1eaee57e8f2be6ab1361e3a45
Sep 18 17:34:06.438808 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:06.438766    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eeab6e-645b4665b7-h78xd" event={"ID":"e5fd32a3-79f9-4b0a-9b59-a8f6866838c5","Type":"ContainerStarted","Data":"ef2a7bfdc4112f009ec8aa4c945dcb14488cda1229571f53394eb7c997d651ea"}
Sep 18 17:34:06.439012 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:06.438829    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eeab6e-645b4665b7-h78xd" event={"ID":"e5fd32a3-79f9-4b0a-9b59-a8f6866838c5","Type":"ContainerStarted","Data":"88cc3905ee24853dc74c53afb66fd54793f823c1eaee57e8f2be6ab1361e3a45"}
Sep 18 17:34:06.439012 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:06.438991    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eeab6e-645b4665b7-h78xd"
Sep 18 17:34:06.455438 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:06.455374    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eeab6e-645b4665b7-h78xd" podStartSLOduration=1.455354813 podStartE2EDuration="1.455354813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:34:06.454766881 +0000 UTC m=+1159.148694435" watchObservedRunningTime="2026-09-18 17:34:06.455354813 +0000 UTC m=+1159.149282342"
Sep 18 17:34:10.951010 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:10.950931    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz"]
Sep 18 17:34:16.441948 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:16.441915    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-b68d6946-66b4c6b97-7f4jv"
Sep 18 17:34:17.446673 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:17.446641    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eeab6e-645b4665b7-h78xd"
Sep 18 17:34:25.970295 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:25.970256    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz" podUID="55cb8f4c-9074-4f68-9a4d-7849cea31624" containerName="main" containerID="cri-o://dacc38d5fdaa9506c01c34b43eb2547b99a0d8f8a7b888b7f8280083007d41de" gracePeriod=45
Sep 18 17:34:26.214374 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:26.214350    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz"
Sep 18 17:34:26.279372 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:26.279342    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55cb8f4c-9074-4f68-9a4d-7849cea31624-dshm\") pod \"55cb8f4c-9074-4f68-9a4d-7849cea31624\" (UID: \"55cb8f4c-9074-4f68-9a4d-7849cea31624\") "
Sep 18 17:34:26.279555 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:26.279388    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55cb8f4c-9074-4f68-9a4d-7849cea31624-tmp-dir\") pod \"55cb8f4c-9074-4f68-9a4d-7849cea31624\" (UID: \"55cb8f4c-9074-4f68-9a4d-7849cea31624\") "
Sep 18 17:34:26.279555 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:26.279458    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55cb8f4c-9074-4f68-9a4d-7849cea31624-tls-certs\") pod \"55cb8f4c-9074-4f68-9a4d-7849cea31624\" (UID: \"55cb8f4c-9074-4f68-9a4d-7849cea31624\") "
Sep 18 17:34:26.279555 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:26.279479    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55cb8f4c-9074-4f68-9a4d-7849cea31624-home\") pod \"55cb8f4c-9074-4f68-9a4d-7849cea31624\" (UID: \"55cb8f4c-9074-4f68-9a4d-7849cea31624\") "
Sep 18 17:34:26.279555 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:26.279523    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55cb8f4c-9074-4f68-9a4d-7849cea31624-model-cache\") pod \"55cb8f4c-9074-4f68-9a4d-7849cea31624\" (UID: \"55cb8f4c-9074-4f68-9a4d-7849cea31624\") "
Sep 18 17:34:26.279555 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:26.279555    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrbnf\" (UniqueName: \"kubernetes.io/projected/55cb8f4c-9074-4f68-9a4d-7849cea31624-kube-api-access-hrbnf\") pod \"55cb8f4c-9074-4f68-9a4d-7849cea31624\" (UID: \"55cb8f4c-9074-4f68-9a4d-7849cea31624\") "
Sep 18 17:34:26.279829 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:26.279711    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55cb8f4c-9074-4f68-9a4d-7849cea31624-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "55cb8f4c-9074-4f68-9a4d-7849cea31624" (UID: "55cb8f4c-9074-4f68-9a4d-7849cea31624"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:34:26.279829 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:26.279811    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55cb8f4c-9074-4f68-9a4d-7849cea31624-home" (OuterVolumeSpecName: "home") pod "55cb8f4c-9074-4f68-9a4d-7849cea31624" (UID: "55cb8f4c-9074-4f68-9a4d-7849cea31624"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:34:26.279973 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:26.279820    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55cb8f4c-9074-4f68-9a4d-7849cea31624-model-cache" (OuterVolumeSpecName: "model-cache") pod "55cb8f4c-9074-4f68-9a4d-7849cea31624" (UID: "55cb8f4c-9074-4f68-9a4d-7849cea31624"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:34:26.279973 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:26.279939    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55cb8f4c-9074-4f68-9a4d-7849cea31624-model-cache\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:34:26.279973 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:26.279957    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55cb8f4c-9074-4f68-9a4d-7849cea31624-tmp-dir\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:34:26.279973 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:26.279965    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55cb8f4c-9074-4f68-9a4d-7849cea31624-home\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:34:26.281687 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:26.281661    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55cb8f4c-9074-4f68-9a4d-7849cea31624-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55cb8f4c-9074-4f68-9a4d-7849cea31624" (UID: "55cb8f4c-9074-4f68-9a4d-7849cea31624"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:34:26.281792 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:26.281664    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55cb8f4c-9074-4f68-9a4d-7849cea31624-kube-api-access-hrbnf" (OuterVolumeSpecName: "kube-api-access-hrbnf") pod "55cb8f4c-9074-4f68-9a4d-7849cea31624" (UID: "55cb8f4c-9074-4f68-9a4d-7849cea31624"). InnerVolumeSpecName "kube-api-access-hrbnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:34:26.281856 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:26.281839    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55cb8f4c-9074-4f68-9a4d-7849cea31624-dshm" (OuterVolumeSpecName: "dshm") pod "55cb8f4c-9074-4f68-9a4d-7849cea31624" (UID: "55cb8f4c-9074-4f68-9a4d-7849cea31624"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:34:26.381038 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:26.381004    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55cb8f4c-9074-4f68-9a4d-7849cea31624-tls-certs\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:34:26.381038 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:26.381033    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrbnf\" (UniqueName: \"kubernetes.io/projected/55cb8f4c-9074-4f68-9a4d-7849cea31624-kube-api-access-hrbnf\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:34:26.381038 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:26.381044    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55cb8f4c-9074-4f68-9a4d-7849cea31624-dshm\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:34:26.546275 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:26.546198    2569 generic.go:356] "Generic (PLEG): container finished" podID="55cb8f4c-9074-4f68-9a4d-7849cea31624" containerID="dacc38d5fdaa9506c01c34b43eb2547b99a0d8f8a7b888b7f8280083007d41de" exitCode=0
Sep 18 17:34:26.546275 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:26.546240    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz"
Sep 18 17:34:26.546445 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:26.546278    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz" event={"ID":"55cb8f4c-9074-4f68-9a4d-7849cea31624","Type":"ContainerDied","Data":"dacc38d5fdaa9506c01c34b43eb2547b99a0d8f8a7b888b7f8280083007d41de"}
Sep 18 17:34:26.546445 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:26.546317    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz" event={"ID":"55cb8f4c-9074-4f68-9a4d-7849cea31624","Type":"ContainerDied","Data":"5c6a2f78b19c6d0e7b67a7dbcf3933509d30c16912d161c3f12b4803395780fd"}
Sep 18 17:34:26.546445 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:26.546336    2569 scope.go:117] "RemoveContainer" containerID="dacc38d5fdaa9506c01c34b43eb2547b99a0d8f8a7b888b7f8280083007d41de"
Sep 18 17:34:26.556197 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:26.556180    2569 scope.go:117] "RemoveContainer" containerID="dacc38d5fdaa9506c01c34b43eb2547b99a0d8f8a7b888b7f8280083007d41de"
Sep 18 17:34:26.556467 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:34:26.556448    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dacc38d5fdaa9506c01c34b43eb2547b99a0d8f8a7b888b7f8280083007d41de\": container with ID starting with dacc38d5fdaa9506c01c34b43eb2547b99a0d8f8a7b888b7f8280083007d41de not found: ID does not exist" containerID="dacc38d5fdaa9506c01c34b43eb2547b99a0d8f8a7b888b7f8280083007d41de"
Sep 18 17:34:26.556537 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:26.556474    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dacc38d5fdaa9506c01c34b43eb2547b99a0d8f8a7b888b7f8280083007d41de"} err="failed to get container status \"dacc38d5fdaa9506c01c34b43eb2547b99a0d8f8a7b888b7f8280083007d41de\": rpc error: code = NotFound desc = could not find container \"dacc38d5fdaa9506c01c34b43eb2547b99a0d8f8a7b888b7f8280083007d41de\": container with ID starting with dacc38d5fdaa9506c01c34b43eb2547b99a0d8f8a7b888b7f8280083007d41de not found: ID does not exist"
Sep 18 17:34:26.569287 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:26.569258    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz"]
Sep 18 17:34:26.573158 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:26.573136    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-231bbc/noten-model-231bbc-kserve-69677879bc-7m2tz"]
Sep 18 17:34:27.768110 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:27.768074    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55cb8f4c-9074-4f68-9a4d-7849cea31624" path="/var/lib/kubelet/pods/55cb8f4c-9074-4f68-9a4d-7849cea31624/volumes"
Sep 18 17:34:47.900178 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:34:47.900152    2569 scope.go:117] "RemoveContainer" containerID="27591c77892acce9c82a63ce6933ef7749920ca7fdd693c1ccda782c689b94e7"
Sep 18 17:35:05.398597 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:05.398563    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5"]
Sep 18 17:35:05.400758 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:05.398962    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55cb8f4c-9074-4f68-9a4d-7849cea31624" containerName="main"
Sep 18 17:35:05.400758 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:05.398974    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="55cb8f4c-9074-4f68-9a4d-7849cea31624" containerName="main"
Sep 18 17:35:05.400758 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:05.399042    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="55cb8f4c-9074-4f68-9a4d-7849cea31624" containerName="main"
Sep 18 17:35:05.401747 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:05.401724    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5"
Sep 18 17:35:05.404134 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:05.404112    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0a2e13\"/\"distinct-2-w0-0a2e13-kserve-self-signed-certs\""
Sep 18 17:35:05.411835 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:05.411813    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5"]
Sep 18 17:35:05.447055 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:05.447022    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldl9l\" (UniqueName: \"kubernetes.io/projected/6c161326-9a1b-4736-ab01-5280f4073af0-kube-api-access-ldl9l\") pod \"distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5\" (UID: \"6c161326-9a1b-4736-ab01-5280f4073af0\") " pod="e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5"
Sep 18 17:35:05.447229 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:05.447071    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c161326-9a1b-4736-ab01-5280f4073af0-model-cache\") pod \"distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5\" (UID: \"6c161326-9a1b-4736-ab01-5280f4073af0\") " pod="e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5"
Sep 18 17:35:05.447229 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:05.447092    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c161326-9a1b-4736-ab01-5280f4073af0-dshm\") pod \"distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5\" (UID: \"6c161326-9a1b-4736-ab01-5280f4073af0\") " pod="e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5"
Sep 18 17:35:05.447229 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:05.447191    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c161326-9a1b-4736-ab01-5280f4073af0-tls-certs\") pod \"distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5\" (UID: \"6c161326-9a1b-4736-ab01-5280f4073af0\") " pod="e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5"
Sep 18 17:35:05.447402 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:05.447232    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c161326-9a1b-4736-ab01-5280f4073af0-home\") pod \"distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5\" (UID: \"6c161326-9a1b-4736-ab01-5280f4073af0\") " pod="e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5"
Sep 18 17:35:05.447402 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:05.447258    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c161326-9a1b-4736-ab01-5280f4073af0-tmp-dir\") pod \"distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5\" (UID: \"6c161326-9a1b-4736-ab01-5280f4073af0\") " pod="e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5"
Sep 18 17:35:05.548559 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:05.548516    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldl9l\" (UniqueName: \"kubernetes.io/projected/6c161326-9a1b-4736-ab01-5280f4073af0-kube-api-access-ldl9l\") pod \"distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5\" (UID: \"6c161326-9a1b-4736-ab01-5280f4073af0\") " pod="e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5"
Sep 18 17:35:05.548737 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:05.548590    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c161326-9a1b-4736-ab01-5280f4073af0-model-cache\") pod \"distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5\" (UID: \"6c161326-9a1b-4736-ab01-5280f4073af0\") " pod="e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5"
Sep 18 17:35:05.548737 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:05.548623    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c161326-9a1b-4736-ab01-5280f4073af0-dshm\") pod \"distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5\" (UID: \"6c161326-9a1b-4736-ab01-5280f4073af0\") " pod="e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5"
Sep 18 17:35:05.548737 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:05.548676    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c161326-9a1b-4736-ab01-5280f4073af0-tls-certs\") pod \"distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5\" (UID: \"6c161326-9a1b-4736-ab01-5280f4073af0\") " pod="e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5"
Sep 18 17:35:05.548737 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:05.548705    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c161326-9a1b-4736-ab01-5280f4073af0-home\") pod \"distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5\" (UID: \"6c161326-9a1b-4736-ab01-5280f4073af0\") " pod="e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5"
Sep 18 17:35:05.548986 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:05.548853    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c161326-9a1b-4736-ab01-5280f4073af0-tmp-dir\") pod \"distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5\" (UID: \"6c161326-9a1b-4736-ab01-5280f4073af0\") " pod="e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5"
Sep 18 17:35:05.549064 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:05.549042    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c161326-9a1b-4736-ab01-5280f4073af0-model-cache\") pod \"distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5\" (UID: \"6c161326-9a1b-4736-ab01-5280f4073af0\") " pod="e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5"
Sep 18 17:35:05.549118 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:05.549093    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c161326-9a1b-4736-ab01-5280f4073af0-home\") pod \"distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5\" (UID: \"6c161326-9a1b-4736-ab01-5280f4073af0\") " pod="e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5"
Sep 18 17:35:05.549196 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:05.549180    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c161326-9a1b-4736-ab01-5280f4073af0-tmp-dir\") pod \"distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5\" (UID: \"6c161326-9a1b-4736-ab01-5280f4073af0\") " pod="e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5"
Sep 18 17:35:05.550982 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:05.550963    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c161326-9a1b-4736-ab01-5280f4073af0-dshm\") pod \"distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5\" (UID: \"6c161326-9a1b-4736-ab01-5280f4073af0\") " pod="e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5"
Sep 18 17:35:05.551154 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:05.551137    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c161326-9a1b-4736-ab01-5280f4073af0-tls-certs\") pod \"distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5\" (UID: \"6c161326-9a1b-4736-ab01-5280f4073af0\") " pod="e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5"
Sep 18 17:35:05.556166 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:05.556144    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldl9l\" (UniqueName: \"kubernetes.io/projected/6c161326-9a1b-4736-ab01-5280f4073af0-kube-api-access-ldl9l\") pod \"distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5\" (UID: \"6c161326-9a1b-4736-ab01-5280f4073af0\") " pod="e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5"
Sep 18 17:35:05.712235 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:05.712127    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5"
Sep 18 17:35:05.985907 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:05.985880    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5"]
Sep 18 17:35:05.987606 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:35:05.987578    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c161326_9a1b_4736_ab01_5280f4073af0.slice/crio-a0baa00b15c351449f0be150ed826ae09e40e1fbc8a05e84694b3b97bc378396 WatchSource:0}: Error finding container a0baa00b15c351449f0be150ed826ae09e40e1fbc8a05e84694b3b97bc378396: Status 404 returned error can't find the container with id a0baa00b15c351449f0be150ed826ae09e40e1fbc8a05e84694b3b97bc378396
Sep 18 17:35:06.777087 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:06.777053    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5" event={"ID":"6c161326-9a1b-4736-ab01-5280f4073af0","Type":"ContainerStarted","Data":"32490aa28dc5c1514ba6bc4dbbf9336f8c9f623da5ecd9af3d39237c2276afe8"}
Sep 18 17:35:06.777087 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:06.777094    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5" event={"ID":"6c161326-9a1b-4736-ab01-5280f4073af0","Type":"ContainerStarted","Data":"a0baa00b15c351449f0be150ed826ae09e40e1fbc8a05e84694b3b97bc378396"}
Sep 18 17:35:06.796933 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:06.796887    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5" podStartSLOduration=1.796853998 podStartE2EDuration="1.796853998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:35:06.794669904 +0000 UTC m=+1219.488597425" watchObservedRunningTime="2026-09-18 17:35:06.796853998 +0000 UTC m=+1219.490781525"
Sep 18 17:35:15.712464 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:15.712425    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5"
Sep 18 17:35:15.712464 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:15.712473    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5"
Sep 18 17:35:15.725591 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:15.725563    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5"
Sep 18 17:35:15.835019 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:15.834985    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5"
Sep 18 17:35:37.307460 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:37.307421    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c"]
Sep 18 17:35:37.311364 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:37.311346    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c"
Sep 18 17:35:37.316723 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:37.316702    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b68d6946\"/\"openshift-service-ca.crt\""
Sep 18 17:35:37.316853 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:37.316800    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b68d6946\"/\"default-dockercfg-jx978\""
Sep 18 17:35:37.317958 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:37.317943    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b68d6946\"/\"kube-root-ca.crt\""
Sep 18 17:35:37.322065 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:37.322050    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b68d6946\"/\"e2e-lifecycle-model-b68d6946-kserve-self-signed-certs\""
Sep 18 17:35:37.329352 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:37.329318    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c"]
Sep 18 17:35:37.464337 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:37.464297    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-model-cache\") pod \"e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c\" (UID: \"3f831579-9b52-4d32-aa4f-ffa5b40f84e7\") " pod="ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c"
Sep 18 17:35:37.464516 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:37.464350    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-home\") pod \"e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c\" (UID: \"3f831579-9b52-4d32-aa4f-ffa5b40f84e7\") " pod="ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c"
Sep 18 17:35:37.464516 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:37.464412    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-tmp-dir\") pod \"e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c\" (UID: \"3f831579-9b52-4d32-aa4f-ffa5b40f84e7\") " pod="ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c"
Sep 18 17:35:37.464516 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:37.464465    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-tls-certs\") pod \"e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c\" (UID: \"3f831579-9b52-4d32-aa4f-ffa5b40f84e7\") " pod="ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c"
Sep 18 17:35:37.464632 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:37.464524    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c5t7\" (UniqueName: \"kubernetes.io/projected/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-kube-api-access-4c5t7\") pod \"e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c\" (UID: \"3f831579-9b52-4d32-aa4f-ffa5b40f84e7\") " pod="ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c"
Sep 18 17:35:37.464632 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:37.464588    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-dshm\") pod \"e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c\" (UID: \"3f831579-9b52-4d32-aa4f-ffa5b40f84e7\") " pod="ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c"
Sep 18 17:35:37.565978 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:37.565852    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-dshm\") pod \"e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c\" (UID: \"3f831579-9b52-4d32-aa4f-ffa5b40f84e7\") " pod="ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c"
Sep 18 17:35:37.566111 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:37.565996    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-model-cache\") pod \"e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c\" (UID: \"3f831579-9b52-4d32-aa4f-ffa5b40f84e7\") " pod="ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c"
Sep 18 17:35:37.566111 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:37.566042    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-home\") pod \"e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c\" (UID: \"3f831579-9b52-4d32-aa4f-ffa5b40f84e7\") " pod="ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c"
Sep 18 17:35:37.566194 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:37.566163    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-tmp-dir\") pod \"e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c\" (UID: \"3f831579-9b52-4d32-aa4f-ffa5b40f84e7\") " pod="ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c"
Sep 18 17:35:37.566247 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:37.566212    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-tls-certs\") pod \"e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c\" (UID: \"3f831579-9b52-4d32-aa4f-ffa5b40f84e7\") " pod="ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c"
Sep 18 17:35:37.566300 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:37.566255    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c5t7\" (UniqueName: \"kubernetes.io/projected/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-kube-api-access-4c5t7\") pod \"e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c\" (UID: \"3f831579-9b52-4d32-aa4f-ffa5b40f84e7\") " pod="ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c"
Sep 18 17:35:37.566401 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:37.566376    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-model-cache\") pod \"e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c\" (UID: \"3f831579-9b52-4d32-aa4f-ffa5b40f84e7\") " pod="ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c"
Sep 18 17:35:37.566457 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:37.566393    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-home\") pod \"e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c\" (UID: \"3f831579-9b52-4d32-aa4f-ffa5b40f84e7\") " pod="ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c"
Sep 18 17:35:37.566621 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:37.566582    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-tmp-dir\") pod \"e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c\" (UID: \"3f831579-9b52-4d32-aa4f-ffa5b40f84e7\") " pod="ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c"
Sep 18 17:35:37.568217 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:37.568193    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-dshm\") pod \"e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c\" (UID: \"3f831579-9b52-4d32-aa4f-ffa5b40f84e7\") " pod="ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c"
Sep 18 17:35:37.568512 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:37.568492    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-tls-certs\") pod \"e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c\" (UID: \"3f831579-9b52-4d32-aa4f-ffa5b40f84e7\") " pod="ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c"
Sep 18 17:35:37.591371 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:37.591340    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c5t7\" (UniqueName: \"kubernetes.io/projected/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-kube-api-access-4c5t7\") pod \"e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c\" (UID: \"3f831579-9b52-4d32-aa4f-ffa5b40f84e7\") " pod="ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c"
Sep 18 17:35:37.621235 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:37.621212    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c"
Sep 18 17:35:37.899950 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:37.899924    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c"]
Sep 18 17:35:37.901900 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:35:37.901842    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f831579_9b52_4d32_aa4f_ffa5b40f84e7.slice/crio-cb867a5d5432b93ae6466b4efabe370051c3d1e5212683bd296d4dc32c42ddca WatchSource:0}: Error finding container cb867a5d5432b93ae6466b4efabe370051c3d1e5212683bd296d4dc32c42ddca: Status 404 returned error can't find the container with id cb867a5d5432b93ae6466b4efabe370051c3d1e5212683bd296d4dc32c42ddca
Sep 18 17:35:37.944835 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:37.944814    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c" event={"ID":"3f831579-9b52-4d32-aa4f-ffa5b40f84e7","Type":"ContainerStarted","Data":"cb867a5d5432b93ae6466b4efabe370051c3d1e5212683bd296d4dc32c42ddca"}
Sep 18 17:35:38.952231 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:38.952195    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c" event={"ID":"3f831579-9b52-4d32-aa4f-ffa5b40f84e7","Type":"ContainerStarted","Data":"40d86c84c0891e248b178832beb6879b7e9597ecabe1bd3934d193db33ab83d1"}
Sep 18 17:35:39.038582 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:39.038528    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c" podStartSLOduration=2.038511878 podStartE2EDuration="2.038511878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:35:39.034824462 +0000 UTC m=+1251.728751989" watchObservedRunningTime="2026-09-18 17:35:39.038511878 +0000 UTC m=+1251.732439405"
Sep 18 17:35:43.620337 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:43.620252    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5"]
Sep 18 17:35:43.624011 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:43.623991    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5"
Sep 18 17:35:43.627291 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:43.627268    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-eeab6e\"/\"default-dockercfg-sqhzj\""
Sep 18 17:35:43.628178 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:43.628158    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-eeab6e\"/\"kube-root-ca.crt\""
Sep 18 17:35:43.628312 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:43.628209    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-eeab6e\"/\"openshift-service-ca.crt\""
Sep 18 17:35:43.628312 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:43.628209    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-eeab6e\"/\"auth-test-model-eeab6e-kserve-self-signed-certs\""
Sep 18 17:35:43.638188 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:43.638162    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5"]
Sep 18 17:35:43.719747 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:43.719697    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-model-cache\") pod \"auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5\" (UID: \"ffa59d70-a4b8-4e96-9f66-9319ac5c48a6\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5"
Sep 18 17:35:43.719994 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:43.719777    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-tls-certs\") pod \"auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5\" (UID: \"ffa59d70-a4b8-4e96-9f66-9319ac5c48a6\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5"
Sep 18 17:35:43.719994 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:43.719814    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fvrx\" (UniqueName: \"kubernetes.io/projected/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-kube-api-access-8fvrx\") pod \"auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5\" (UID: \"ffa59d70-a4b8-4e96-9f66-9319ac5c48a6\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5"
Sep 18 17:35:43.719994 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:43.719842    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-home\") pod \"auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5\" (UID: \"ffa59d70-a4b8-4e96-9f66-9319ac5c48a6\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5"
Sep 18 17:35:43.720176 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:43.720044    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-dshm\") pod \"auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5\" (UID: \"ffa59d70-a4b8-4e96-9f66-9319ac5c48a6\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5"
Sep 18 17:35:43.720176 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:43.720072    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-tmp-dir\") pod \"auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5\" (UID: \"ffa59d70-a4b8-4e96-9f66-9319ac5c48a6\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5"
Sep 18 17:35:43.820991 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:43.820952    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-tls-certs\") pod \"auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5\" (UID: \"ffa59d70-a4b8-4e96-9f66-9319ac5c48a6\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5"
Sep 18 17:35:43.821189 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:43.821009    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fvrx\" (UniqueName: \"kubernetes.io/projected/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-kube-api-access-8fvrx\") pod \"auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5\" (UID: \"ffa59d70-a4b8-4e96-9f66-9319ac5c48a6\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5"
Sep 18 17:35:43.821189 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:43.821042    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-home\") pod \"auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5\" (UID: \"ffa59d70-a4b8-4e96-9f66-9319ac5c48a6\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5"
Sep 18 17:35:43.821189 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:43.821142    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-dshm\") pod \"auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5\" (UID: \"ffa59d70-a4b8-4e96-9f66-9319ac5c48a6\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5"
Sep 18 17:35:43.821189 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:43.821163    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-tmp-dir\") pod \"auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5\" (UID: \"ffa59d70-a4b8-4e96-9f66-9319ac5c48a6\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5"
Sep 18 17:35:43.821405 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:43.821219    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-model-cache\") pod \"auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5\" (UID: \"ffa59d70-a4b8-4e96-9f66-9319ac5c48a6\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5"
Sep 18 17:35:43.821583 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:43.821559    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-home\") pod \"auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5\" (UID: \"ffa59d70-a4b8-4e96-9f66-9319ac5c48a6\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5"
Sep 18 17:35:43.821651 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:43.821585    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-tmp-dir\") pod \"auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5\" (UID: \"ffa59d70-a4b8-4e96-9f66-9319ac5c48a6\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5"
Sep 18 17:35:43.821651 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:43.821629    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-model-cache\") pod \"auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5\" (UID: \"ffa59d70-a4b8-4e96-9f66-9319ac5c48a6\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5"
Sep 18 17:35:43.823387 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:43.823370    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-dshm\") pod \"auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5\" (UID: \"ffa59d70-a4b8-4e96-9f66-9319ac5c48a6\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5"
Sep 18 17:35:43.823573 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:43.823557    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-tls-certs\") pod \"auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5\" (UID: \"ffa59d70-a4b8-4e96-9f66-9319ac5c48a6\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5"
Sep 18 17:35:43.831391 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:43.831361    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fvrx\" (UniqueName: \"kubernetes.io/projected/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-kube-api-access-8fvrx\") pod \"auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5\" (UID: \"ffa59d70-a4b8-4e96-9f66-9319ac5c48a6\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5"
Sep 18 17:35:43.934873 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:43.934767    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5"
Sep 18 17:35:44.216375 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:44.216348    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5"]
Sep 18 17:35:44.218033 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:35:44.217996    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podffa59d70_a4b8_4e96_9f66_9319ac5c48a6.slice/crio-5b3f82ef6420559429491a42990406441fed8be22dc5a73cc78bf87263a4be78 WatchSource:0}: Error finding container 5b3f82ef6420559429491a42990406441fed8be22dc5a73cc78bf87263a4be78: Status 404 returned error can't find the container with id 5b3f82ef6420559429491a42990406441fed8be22dc5a73cc78bf87263a4be78
Sep 18 17:35:44.987068 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:44.987030    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5" event={"ID":"ffa59d70-a4b8-4e96-9f66-9319ac5c48a6","Type":"ContainerStarted","Data":"4b1e1b8a28104195d99611b519c8cccc8167c08497fe9dcc1f5c5577b74e55a7"}
Sep 18 17:35:44.987442 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:44.987079    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5" event={"ID":"ffa59d70-a4b8-4e96-9f66-9319ac5c48a6","Type":"ContainerStarted","Data":"5b3f82ef6420559429491a42990406441fed8be22dc5a73cc78bf87263a4be78"}
Sep 18 17:35:45.007942 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:45.007880    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5" podStartSLOduration=2.00785249 podStartE2EDuration="2.00785249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:35:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:35:45.006125177 +0000 UTC m=+1257.700052705" watchObservedRunningTime="2026-09-18 17:35:45.00785249 +0000 UTC m=+1257.701780015"
Sep 18 17:35:47.622335 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:47.622290    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c"
Sep 18 17:35:47.622748 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:47.622348    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c"
Sep 18 17:35:47.635103 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:47.635078    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c"
Sep 18 17:35:47.925290 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:47.925202    2569 scope.go:117] "RemoveContainer" containerID="50e8a5907e7c22255b7b99ec5a2a1c7f827b8245d6eef9061aed9f9e13190e3f"
Sep 18 17:35:47.934048 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:47.934023    2569 scope.go:117] "RemoveContainer" containerID="0d1ed69721bcf66c2f6f39b2d04ae4a25982a8eaf50be09b64817d8273c79661"
Sep 18 17:35:48.016068 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:48.016043    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c"
Sep 18 17:35:53.935156 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:53.935114    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5"
Sep 18 17:35:53.935639 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:53.935172    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5"
Sep 18 17:35:53.947746 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:53.947720    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5"
Sep 18 17:35:54.044519 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:35:54.044488    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5"
Sep 18 17:36:36.486360 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:36.486325    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b68d6946-pxfgs"]
Sep 18 17:36:36.489900 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:36.489881    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b68d6946-pxfgs"
Sep 18 17:36:36.492578 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:36.492559    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qt978\""
Sep 18 17:36:36.502278 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:36.502254    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b68d6946-pxfgs"]
Sep 18 17:36:36.546244 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:36.546210    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13089db8-e7d7-45fa-a345-2794c6f3b165-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b68d6946-pxfgs\" (UID: \"13089db8-e7d7-45fa-a345-2794c6f3b165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b68d6946-pxfgs"
Sep 18 17:36:36.647804 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:36.647771    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13089db8-e7d7-45fa-a345-2794c6f3b165-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b68d6946-pxfgs\" (UID: \"13089db8-e7d7-45fa-a345-2794c6f3b165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b68d6946-pxfgs"
Sep 18 17:36:36.648448 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:36.648423    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13089db8-e7d7-45fa-a345-2794c6f3b165-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b68d6946-pxfgs\" (UID: \"13089db8-e7d7-45fa-a345-2794c6f3b165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b68d6946-pxfgs"
Sep 18 17:36:36.800336 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:36.800243    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b68d6946-pxfgs"
Sep 18 17:36:37.077996 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:37.077966    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b68d6946-pxfgs"]
Sep 18 17:36:37.079614 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:36:37.079586    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13089db8_e7d7_45fa_a345_2794c6f3b165.slice/crio-fad520b91dc53778c2dfd2938bce6feb7ab801341c5d9b6c89aa8f1866d80ade WatchSource:0}: Error finding container fad520b91dc53778c2dfd2938bce6feb7ab801341c5d9b6c89aa8f1866d80ade: Status 404 returned error can't find the container with id fad520b91dc53778c2dfd2938bce6feb7ab801341c5d9b6c89aa8f1866d80ade
Sep 18 17:36:37.276078 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:37.276047    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b68d6946-pxfgs" event={"ID":"13089db8-e7d7-45fa-a345-2794c6f3b165","Type":"ContainerStarted","Data":"fad520b91dc53778c2dfd2938bce6feb7ab801341c5d9b6c89aa8f1866d80ade"}
Sep 18 17:36:38.282912 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:38.282855    2569 generic.go:356] "Generic (PLEG): container finished" podID="13089db8-e7d7-45fa-a345-2794c6f3b165" containerID="885b832cafc5efe9ee0e06f00cfc3b4d6b1fc69238e96c535d6dcae20d2ff136" exitCode=0
Sep 18 17:36:38.283310 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:38.282937    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b68d6946-pxfgs" event={"ID":"13089db8-e7d7-45fa-a345-2794c6f3b165","Type":"ContainerDied","Data":"885b832cafc5efe9ee0e06f00cfc3b4d6b1fc69238e96c535d6dcae20d2ff136"}
Sep 18 17:36:39.426315 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:39.426290    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b68d6946-pxfgs"
Sep 18 17:36:39.579424 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:39.579337    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13089db8-e7d7-45fa-a345-2794c6f3b165-maas-api-service-ca\") pod \"13089db8-e7d7-45fa-a345-2794c6f3b165\" (UID: \"13089db8-e7d7-45fa-a345-2794c6f3b165\") "
Sep 18 17:36:39.579702 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:39.579678    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13089db8-e7d7-45fa-a345-2794c6f3b165-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "13089db8-e7d7-45fa-a345-2794c6f3b165" (UID: "13089db8-e7d7-45fa-a345-2794c6f3b165"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:36:39.681303 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:39.681263    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13089db8-e7d7-45fa-a345-2794c6f3b165-maas-api-service-ca\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:36:40.294673 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:40.294645    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b68d6946-pxfgs"
Sep 18 17:36:40.294857 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:40.294678    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b68d6946-pxfgs" event={"ID":"13089db8-e7d7-45fa-a345-2794c6f3b165","Type":"ContainerDied","Data":"fad520b91dc53778c2dfd2938bce6feb7ab801341c5d9b6c89aa8f1866d80ade"}
Sep 18 17:36:40.294857 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:40.294733    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fad520b91dc53778c2dfd2938bce6feb7ab801341c5d9b6c89aa8f1866d80ade"
Sep 18 17:36:47.959005 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:47.958974    2569 scope.go:117] "RemoveContainer" containerID="034258b763cb0612896b9ea4ba64c7a329a20e4203142b089784adc473f0e490"
Sep 18 17:36:52.046226 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:52.046183    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b68d6946-66b4c6b97-7f4jv"]
Sep 18 17:36:52.046822 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:52.046439    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-b68d6946-66b4c6b97-7f4jv" podUID="2048ef0c-bc30-4d73-a677-b71149532417" containerName="payload-processing" containerID="cri-o://1de479b67844fc2e65d028918dc04e7f944560a444fec4e5fe2a9f0de511a615" gracePeriod=30
Sep 18 17:36:52.273366 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:52.273317    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b68d6946-pxfgs"]
Sep 18 17:36:52.275629 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:52.275606    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b68d6946-pxfgs"]
Sep 18 17:36:52.296139 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:52.296115    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b68d6946-66b4c6b97-7f4jv"
Sep 18 17:36:52.306652 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:52.306589    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2048ef0c-bc30-4d73-a677-b71149532417-plugins-config-volume\") pod \"2048ef0c-bc30-4d73-a677-b71149532417\" (UID: \"2048ef0c-bc30-4d73-a677-b71149532417\") "
Sep 18 17:36:52.306764 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:52.306692    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmr87\" (UniqueName: \"kubernetes.io/projected/2048ef0c-bc30-4d73-a677-b71149532417-kube-api-access-nmr87\") pod \"2048ef0c-bc30-4d73-a677-b71149532417\" (UID: \"2048ef0c-bc30-4d73-a677-b71149532417\") "
Sep 18 17:36:52.306956 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:52.306935    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2048ef0c-bc30-4d73-a677-b71149532417-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2048ef0c-bc30-4d73-a677-b71149532417" (UID: "2048ef0c-bc30-4d73-a677-b71149532417"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:36:52.307124 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:52.307105    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2048ef0c-bc30-4d73-a677-b71149532417-plugins-config-volume\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:36:52.308723 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:52.308700    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2048ef0c-bc30-4d73-a677-b71149532417-kube-api-access-nmr87" (OuterVolumeSpecName: "kube-api-access-nmr87") pod "2048ef0c-bc30-4d73-a677-b71149532417" (UID: "2048ef0c-bc30-4d73-a677-b71149532417"). InnerVolumeSpecName "kube-api-access-nmr87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:36:52.361328 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:52.361298    2569 generic.go:356] "Generic (PLEG): container finished" podID="2048ef0c-bc30-4d73-a677-b71149532417" containerID="1de479b67844fc2e65d028918dc04e7f944560a444fec4e5fe2a9f0de511a615" exitCode=0
Sep 18 17:36:52.361515 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:52.361349    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b68d6946-66b4c6b97-7f4jv"
Sep 18 17:36:52.361515 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:52.361365    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b68d6946-66b4c6b97-7f4jv" event={"ID":"2048ef0c-bc30-4d73-a677-b71149532417","Type":"ContainerDied","Data":"1de479b67844fc2e65d028918dc04e7f944560a444fec4e5fe2a9f0de511a615"}
Sep 18 17:36:52.361515 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:52.361405    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b68d6946-66b4c6b97-7f4jv" event={"ID":"2048ef0c-bc30-4d73-a677-b71149532417","Type":"ContainerDied","Data":"76b8a5734666c8dc1a59ce77e272e3ee02ea237645ac75b1b34acb6c81c71807"}
Sep 18 17:36:52.361515 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:52.361427    2569 scope.go:117] "RemoveContainer" containerID="1de479b67844fc2e65d028918dc04e7f944560a444fec4e5fe2a9f0de511a615"
Sep 18 17:36:52.370831 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:52.370813    2569 scope.go:117] "RemoveContainer" containerID="1de479b67844fc2e65d028918dc04e7f944560a444fec4e5fe2a9f0de511a615"
Sep 18 17:36:52.371143 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:36:52.371124    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1de479b67844fc2e65d028918dc04e7f944560a444fec4e5fe2a9f0de511a615\": container with ID starting with 1de479b67844fc2e65d028918dc04e7f944560a444fec4e5fe2a9f0de511a615 not found: ID does not exist" containerID="1de479b67844fc2e65d028918dc04e7f944560a444fec4e5fe2a9f0de511a615"
Sep 18 17:36:52.371206 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:52.371151    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1de479b67844fc2e65d028918dc04e7f944560a444fec4e5fe2a9f0de511a615"} err="failed to get container status \"1de479b67844fc2e65d028918dc04e7f944560a444fec4e5fe2a9f0de511a615\": rpc error: code = NotFound desc = could not find container \"1de479b67844fc2e65d028918dc04e7f944560a444fec4e5fe2a9f0de511a615\": container with ID starting with 1de479b67844fc2e65d028918dc04e7f944560a444fec4e5fe2a9f0de511a615 not found: ID does not exist"
Sep 18 17:36:52.384395 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:52.384364    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b68d6946-66b4c6b97-7f4jv"]
Sep 18 17:36:52.390323 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:52.390302    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b68d6946-66b4c6b97-7f4jv"]
Sep 18 17:36:52.408567 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:52.408539    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmr87\" (UniqueName: \"kubernetes.io/projected/2048ef0c-bc30-4d73-a677-b71149532417-kube-api-access-nmr87\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:36:53.768785 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:53.768752    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13089db8-e7d7-45fa-a345-2794c6f3b165" path="/var/lib/kubelet/pods/13089db8-e7d7-45fa-a345-2794c6f3b165/volumes"
Sep 18 17:36:53.769225 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:36:53.769208    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2048ef0c-bc30-4d73-a677-b71149532417" path="/var/lib/kubelet/pods/2048ef0c-bc30-4d73-a677-b71149532417/volumes"
Sep 18 17:37:00.307888 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:00.307824    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c"]
Sep 18 17:37:11.008768 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:11.008682    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh"]
Sep 18 17:37:11.009155 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:11.009095    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2048ef0c-bc30-4d73-a677-b71149532417" containerName="payload-processing"
Sep 18 17:37:11.009155 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:11.009111    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2048ef0c-bc30-4d73-a677-b71149532417" containerName="payload-processing"
Sep 18 17:37:11.009155 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:11.009124    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13089db8-e7d7-45fa-a345-2794c6f3b165" containerName="revoke-keys"
Sep 18 17:37:11.009155 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:11.009130    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="13089db8-e7d7-45fa-a345-2794c6f3b165" containerName="revoke-keys"
Sep 18 17:37:11.009297 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:11.009234    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2048ef0c-bc30-4d73-a677-b71149532417" containerName="payload-processing"
Sep 18 17:37:11.009297 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:11.009245    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="13089db8-e7d7-45fa-a345-2794c6f3b165" containerName="revoke-keys"
Sep 18 17:37:11.013781 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:11.013763    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh"
Sep 18 17:37:11.017255 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:11.017232    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-808470\"/\"auth-test-model-808470-kserve-self-signed-certs\""
Sep 18 17:37:11.017379 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:11.017329    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-808470\"/\"kube-root-ca.crt\""
Sep 18 17:37:11.017379 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:11.017339    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-808470\"/\"default-dockercfg-j42zw\""
Sep 18 17:37:11.017510 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:11.017338    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-808470\"/\"openshift-service-ca.crt\""
Sep 18 17:37:11.023810 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:11.023787    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh"]
Sep 18 17:37:11.102800 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:11.102758    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7q6h\" (UniqueName: \"kubernetes.io/projected/76083dd6-57b2-4c34-96d5-17b73d8c8654-kube-api-access-f7q6h\") pod \"auth-test-model-808470-kserve-54757fc7b7-n9dnh\" (UID: \"76083dd6-57b2-4c34-96d5-17b73d8c8654\") " pod="ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh"
Sep 18 17:37:11.103023 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:11.102907    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76083dd6-57b2-4c34-96d5-17b73d8c8654-model-cache\") pod \"auth-test-model-808470-kserve-54757fc7b7-n9dnh\" (UID: \"76083dd6-57b2-4c34-96d5-17b73d8c8654\") " pod="ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh"
Sep 18 17:37:11.103023 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:11.102984    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76083dd6-57b2-4c34-96d5-17b73d8c8654-home\") pod \"auth-test-model-808470-kserve-54757fc7b7-n9dnh\" (UID: \"76083dd6-57b2-4c34-96d5-17b73d8c8654\") " pod="ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh"
Sep 18 17:37:11.103152 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:11.103024    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76083dd6-57b2-4c34-96d5-17b73d8c8654-tls-certs\") pod \"auth-test-model-808470-kserve-54757fc7b7-n9dnh\" (UID: \"76083dd6-57b2-4c34-96d5-17b73d8c8654\") " pod="ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh"
Sep 18 17:37:11.103152 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:11.103097    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76083dd6-57b2-4c34-96d5-17b73d8c8654-tmp-dir\") pod \"auth-test-model-808470-kserve-54757fc7b7-n9dnh\" (UID: \"76083dd6-57b2-4c34-96d5-17b73d8c8654\") " pod="ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh"
Sep 18 17:37:11.103152 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:11.103136    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76083dd6-57b2-4c34-96d5-17b73d8c8654-dshm\") pod \"auth-test-model-808470-kserve-54757fc7b7-n9dnh\" (UID: \"76083dd6-57b2-4c34-96d5-17b73d8c8654\") " pod="ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh"
Sep 18 17:37:11.204485 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:11.204447    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76083dd6-57b2-4c34-96d5-17b73d8c8654-tmp-dir\") pod \"auth-test-model-808470-kserve-54757fc7b7-n9dnh\" (UID: \"76083dd6-57b2-4c34-96d5-17b73d8c8654\") " pod="ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh"
Sep 18 17:37:11.204485 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:11.204489    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76083dd6-57b2-4c34-96d5-17b73d8c8654-dshm\") pod \"auth-test-model-808470-kserve-54757fc7b7-n9dnh\" (UID: \"76083dd6-57b2-4c34-96d5-17b73d8c8654\") " pod="ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh"
Sep 18 17:37:11.204691 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:11.204524    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7q6h\" (UniqueName: \"kubernetes.io/projected/76083dd6-57b2-4c34-96d5-17b73d8c8654-kube-api-access-f7q6h\") pod \"auth-test-model-808470-kserve-54757fc7b7-n9dnh\" (UID: \"76083dd6-57b2-4c34-96d5-17b73d8c8654\") " pod="ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh"
Sep 18 17:37:11.204691 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:11.204566    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76083dd6-57b2-4c34-96d5-17b73d8c8654-model-cache\") pod \"auth-test-model-808470-kserve-54757fc7b7-n9dnh\" (UID: \"76083dd6-57b2-4c34-96d5-17b73d8c8654\") " pod="ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh"
Sep 18 17:37:11.204691 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:11.204611    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76083dd6-57b2-4c34-96d5-17b73d8c8654-home\") pod \"auth-test-model-808470-kserve-54757fc7b7-n9dnh\" (UID: \"76083dd6-57b2-4c34-96d5-17b73d8c8654\") " pod="ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh"
Sep 18 17:37:11.204691 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:11.204641    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76083dd6-57b2-4c34-96d5-17b73d8c8654-tls-certs\") pod \"auth-test-model-808470-kserve-54757fc7b7-n9dnh\" (UID: \"76083dd6-57b2-4c34-96d5-17b73d8c8654\") " pod="ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh"
Sep 18 17:37:11.204971 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:11.204941    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76083dd6-57b2-4c34-96d5-17b73d8c8654-tmp-dir\") pod \"auth-test-model-808470-kserve-54757fc7b7-n9dnh\" (UID: \"76083dd6-57b2-4c34-96d5-17b73d8c8654\") " pod="ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh"
Sep 18 17:37:11.205036 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:11.205007    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76083dd6-57b2-4c34-96d5-17b73d8c8654-model-cache\") pod \"auth-test-model-808470-kserve-54757fc7b7-n9dnh\" (UID: \"76083dd6-57b2-4c34-96d5-17b73d8c8654\") " pod="ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh"
Sep 18 17:37:11.205091 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:11.205048    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76083dd6-57b2-4c34-96d5-17b73d8c8654-home\") pod \"auth-test-model-808470-kserve-54757fc7b7-n9dnh\" (UID: \"76083dd6-57b2-4c34-96d5-17b73d8c8654\") " pod="ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh"
Sep 18 17:37:11.206780 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:11.206761    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76083dd6-57b2-4c34-96d5-17b73d8c8654-dshm\") pod \"auth-test-model-808470-kserve-54757fc7b7-n9dnh\" (UID: \"76083dd6-57b2-4c34-96d5-17b73d8c8654\") " pod="ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh"
Sep 18 17:37:11.207093 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:11.207075    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76083dd6-57b2-4c34-96d5-17b73d8c8654-tls-certs\") pod \"auth-test-model-808470-kserve-54757fc7b7-n9dnh\" (UID: \"76083dd6-57b2-4c34-96d5-17b73d8c8654\") " pod="ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh"
Sep 18 17:37:11.213590 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:11.213569    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7q6h\" (UniqueName: \"kubernetes.io/projected/76083dd6-57b2-4c34-96d5-17b73d8c8654-kube-api-access-f7q6h\") pod \"auth-test-model-808470-kserve-54757fc7b7-n9dnh\" (UID: \"76083dd6-57b2-4c34-96d5-17b73d8c8654\") " pod="ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh"
Sep 18 17:37:11.324474 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:11.324396    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh"
Sep 18 17:37:11.598741 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:11.598703    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh"]
Sep 18 17:37:11.600510 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:37:11.600482    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76083dd6_57b2_4c34_96d5_17b73d8c8654.slice/crio-9da1a17dccf4866bf3443970abc41de8a5b08242d25c8e1333cb6aea7852e76e WatchSource:0}: Error finding container 9da1a17dccf4866bf3443970abc41de8a5b08242d25c8e1333cb6aea7852e76e: Status 404 returned error can't find the container with id 9da1a17dccf4866bf3443970abc41de8a5b08242d25c8e1333cb6aea7852e76e
Sep 18 17:37:12.472626 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:12.472591    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh" event={"ID":"76083dd6-57b2-4c34-96d5-17b73d8c8654","Type":"ContainerStarted","Data":"ea94513aa202645cbe777e877e829f6abb7d219da7e475ae68f30b958213a726"}
Sep 18 17:37:12.473034 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:12.472639    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh" event={"ID":"76083dd6-57b2-4c34-96d5-17b73d8c8654","Type":"ContainerStarted","Data":"9da1a17dccf4866bf3443970abc41de8a5b08242d25c8e1333cb6aea7852e76e"}
Sep 18 17:37:12.503194 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:12.503137    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh" podStartSLOduration=2.503122223 podStartE2EDuration="2.503122223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:37:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:37:12.501402707 +0000 UTC m=+1345.195330234" watchObservedRunningTime="2026-09-18 17:37:12.503122223 +0000 UTC m=+1345.197049751"
Sep 18 17:37:15.327252 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:15.327203    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c" podUID="3f831579-9b52-4d32-aa4f-ffa5b40f84e7" containerName="main" containerID="cri-o://40d86c84c0891e248b178832beb6879b7e9597ecabe1bd3934d193db33ab83d1" gracePeriod=45
Sep 18 17:37:15.492224 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:15.492193    2569 generic.go:356] "Generic (PLEG): container finished" podID="3f831579-9b52-4d32-aa4f-ffa5b40f84e7" containerID="40d86c84c0891e248b178832beb6879b7e9597ecabe1bd3934d193db33ab83d1" exitCode=0
Sep 18 17:37:15.492424 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:15.492270    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c" event={"ID":"3f831579-9b52-4d32-aa4f-ffa5b40f84e7","Type":"ContainerDied","Data":"40d86c84c0891e248b178832beb6879b7e9597ecabe1bd3934d193db33ab83d1"}
Sep 18 17:37:15.587379 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:15.587320    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c"
Sep 18 17:37:15.748954 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:15.748915    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-home\") pod \"3f831579-9b52-4d32-aa4f-ffa5b40f84e7\" (UID: \"3f831579-9b52-4d32-aa4f-ffa5b40f84e7\") "
Sep 18 17:37:15.749140 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:15.748965    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-tmp-dir\") pod \"3f831579-9b52-4d32-aa4f-ffa5b40f84e7\" (UID: \"3f831579-9b52-4d32-aa4f-ffa5b40f84e7\") "
Sep 18 17:37:15.749140 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:15.749016    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4c5t7\" (UniqueName: \"kubernetes.io/projected/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-kube-api-access-4c5t7\") pod \"3f831579-9b52-4d32-aa4f-ffa5b40f84e7\" (UID: \"3f831579-9b52-4d32-aa4f-ffa5b40f84e7\") "
Sep 18 17:37:15.749140 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:15.749090    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-model-cache\") pod \"3f831579-9b52-4d32-aa4f-ffa5b40f84e7\" (UID: \"3f831579-9b52-4d32-aa4f-ffa5b40f84e7\") "
Sep 18 17:37:15.749140 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:15.749124    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-tls-certs\") pod \"3f831579-9b52-4d32-aa4f-ffa5b40f84e7\" (UID: \"3f831579-9b52-4d32-aa4f-ffa5b40f84e7\") "
Sep 18 17:37:15.749360 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:15.749205    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-home" (OuterVolumeSpecName: "home") pod "3f831579-9b52-4d32-aa4f-ffa5b40f84e7" (UID: "3f831579-9b52-4d32-aa4f-ffa5b40f84e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:37:15.749360 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:15.749228    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3f831579-9b52-4d32-aa4f-ffa5b40f84e7" (UID: "3f831579-9b52-4d32-aa4f-ffa5b40f84e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:37:15.749360 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:15.749237    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-dshm\") pod \"3f831579-9b52-4d32-aa4f-ffa5b40f84e7\" (UID: \"3f831579-9b52-4d32-aa4f-ffa5b40f84e7\") "
Sep 18 17:37:15.749360 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:15.749344    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-model-cache" (OuterVolumeSpecName: "model-cache") pod "3f831579-9b52-4d32-aa4f-ffa5b40f84e7" (UID: "3f831579-9b52-4d32-aa4f-ffa5b40f84e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:37:15.749599 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:15.749578    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-model-cache\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:37:15.749659 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:15.749602    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-home\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:37:15.749659 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:15.749611    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-tmp-dir\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:37:15.751418 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:15.751391    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-dshm" (OuterVolumeSpecName: "dshm") pod "3f831579-9b52-4d32-aa4f-ffa5b40f84e7" (UID: "3f831579-9b52-4d32-aa4f-ffa5b40f84e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:37:15.751544 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:15.751412    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-kube-api-access-4c5t7" (OuterVolumeSpecName: "kube-api-access-4c5t7") pod "3f831579-9b52-4d32-aa4f-ffa5b40f84e7" (UID: "3f831579-9b52-4d32-aa4f-ffa5b40f84e7"). InnerVolumeSpecName "kube-api-access-4c5t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:37:15.751613 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:15.751593    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f831579-9b52-4d32-aa4f-ffa5b40f84e7" (UID: "3f831579-9b52-4d32-aa4f-ffa5b40f84e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:37:15.850532 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:15.850443    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c5t7\" (UniqueName: \"kubernetes.io/projected/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-kube-api-access-4c5t7\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:37:15.850532 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:15.850478    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-tls-certs\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:37:15.850532 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:15.850488    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f831579-9b52-4d32-aa4f-ffa5b40f84e7-dshm\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:37:16.499333 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:16.499304    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c"
Sep 18 17:37:16.499800 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:16.499330    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c" event={"ID":"3f831579-9b52-4d32-aa4f-ffa5b40f84e7","Type":"ContainerDied","Data":"cb867a5d5432b93ae6466b4efabe370051c3d1e5212683bd296d4dc32c42ddca"}
Sep 18 17:37:16.499800 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:16.499383    2569 scope.go:117] "RemoveContainer" containerID="40d86c84c0891e248b178832beb6879b7e9597ecabe1bd3934d193db33ab83d1"
Sep 18 17:37:16.516934 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:16.516906    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c"]
Sep 18 17:37:16.518990 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:16.518970    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-b68d6946/e2e-lifecycle-model-b68d6946-kserve-6d86c44b79-l229c"]
Sep 18 17:37:17.765468 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:17.765436    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f831579-9b52-4d32-aa4f-ffa5b40f84e7" path="/var/lib/kubelet/pods/3f831579-9b52-4d32-aa4f-ffa5b40f84e7/volumes"
Sep 18 17:37:21.325431 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:21.325395    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh"
Sep 18 17:37:21.325834 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:21.325473    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh"
Sep 18 17:37:21.338189 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:21.338167    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh"
Sep 18 17:37:21.537161 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:37:21.537134    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh"
Sep 18 17:38:47.999156 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:38:47.999079    2569 scope.go:117] "RemoveContainer" containerID="cc4b2a885d1f76aa21a6334aef6d2ed1523aed477f79300e93b7aeb2297b6f8a"
Sep 18 17:38:48.008369 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:38:48.008352    2569 scope.go:117] "RemoveContainer" containerID="d6d61dbc8fe39315137ef5e87735ffe9a1e1944ebf5ecc40c1ed336b6a15aab0"
Sep 18 17:39:29.194173 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:29.194138    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f4a54f-5dc4c7ff86-5kljr"]
Sep 18 17:39:29.194609 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:29.194527    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f831579-9b52-4d32-aa4f-ffa5b40f84e7" containerName="main"
Sep 18 17:39:29.194609 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:29.194539    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f831579-9b52-4d32-aa4f-ffa5b40f84e7" containerName="main"
Sep 18 17:39:29.194687 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:29.194618    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f831579-9b52-4d32-aa4f-ffa5b40f84e7" containerName="main"
Sep 18 17:39:29.197643 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:29.197627    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f4a54f-5dc4c7ff86-5kljr"
Sep 18 17:39:29.200179 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:29.200157    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f4a54f\""
Sep 18 17:39:29.200589 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:29.200574    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f4a54f-dockercfg-sqfc6\""
Sep 18 17:39:29.212339 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:29.212316    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f4a54f-5dc4c7ff86-5kljr"]
Sep 18 17:39:29.214613 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:29.214587    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f4a54f-5fd8946565-dfltf"]
Sep 18 17:39:29.218269 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:29.218251    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f4a54f-5fd8946565-dfltf"
Sep 18 17:39:29.227933 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:29.227908    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f4a54f-5fd8946565-dfltf"]
Sep 18 17:39:29.251751 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:29.251724    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n6nc\" (UniqueName: \"kubernetes.io/projected/24308a53-fe4b-47b4-9ca6-5baf7c47ff1f-kube-api-access-6n6nc\") pod \"payload-processing-e2e-api-b-f4a54f-5fd8946565-dfltf\" (UID: \"24308a53-fe4b-47b4-9ca6-5baf7c47ff1f\") " pod="openshift-ingress/payload-processing-e2e-api-b-f4a54f-5fd8946565-dfltf"
Sep 18 17:39:29.251913 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:29.251770    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jbcw\" (UniqueName: \"kubernetes.io/projected/ca4533b5-8527-43e2-87f2-0e6e7fb7d065-kube-api-access-8jbcw\") pod \"payload-pre-processing-e2e-api-b-f4a54f-5dc4c7ff86-5kljr\" (UID: \"ca4533b5-8527-43e2-87f2-0e6e7fb7d065\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f4a54f-5dc4c7ff86-5kljr"
Sep 18 17:39:29.251913 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:29.251884    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24308a53-fe4b-47b4-9ca6-5baf7c47ff1f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f4a54f-5fd8946565-dfltf\" (UID: \"24308a53-fe4b-47b4-9ca6-5baf7c47ff1f\") " pod="openshift-ingress/payload-processing-e2e-api-b-f4a54f-5fd8946565-dfltf"
Sep 18 17:39:29.252005 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:29.251922    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca4533b5-8527-43e2-87f2-0e6e7fb7d065-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f4a54f-5dc4c7ff86-5kljr\" (UID: \"ca4533b5-8527-43e2-87f2-0e6e7fb7d065\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f4a54f-5dc4c7ff86-5kljr"
Sep 18 17:39:29.353101 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:29.353063    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jbcw\" (UniqueName: \"kubernetes.io/projected/ca4533b5-8527-43e2-87f2-0e6e7fb7d065-kube-api-access-8jbcw\") pod \"payload-pre-processing-e2e-api-b-f4a54f-5dc4c7ff86-5kljr\" (UID: \"ca4533b5-8527-43e2-87f2-0e6e7fb7d065\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f4a54f-5dc4c7ff86-5kljr"
Sep 18 17:39:29.353276 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:29.353140    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24308a53-fe4b-47b4-9ca6-5baf7c47ff1f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f4a54f-5fd8946565-dfltf\" (UID: \"24308a53-fe4b-47b4-9ca6-5baf7c47ff1f\") " pod="openshift-ingress/payload-processing-e2e-api-b-f4a54f-5fd8946565-dfltf"
Sep 18 17:39:29.353276 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:29.353172    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca4533b5-8527-43e2-87f2-0e6e7fb7d065-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f4a54f-5dc4c7ff86-5kljr\" (UID: \"ca4533b5-8527-43e2-87f2-0e6e7fb7d065\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f4a54f-5dc4c7ff86-5kljr"
Sep 18 17:39:29.353276 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:29.353227    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n6nc\" (UniqueName: \"kubernetes.io/projected/24308a53-fe4b-47b4-9ca6-5baf7c47ff1f-kube-api-access-6n6nc\") pod \"payload-processing-e2e-api-b-f4a54f-5fd8946565-dfltf\" (UID: \"24308a53-fe4b-47b4-9ca6-5baf7c47ff1f\") " pod="openshift-ingress/payload-processing-e2e-api-b-f4a54f-5fd8946565-dfltf"
Sep 18 17:39:29.353912 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:29.353858    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24308a53-fe4b-47b4-9ca6-5baf7c47ff1f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f4a54f-5fd8946565-dfltf\" (UID: \"24308a53-fe4b-47b4-9ca6-5baf7c47ff1f\") " pod="openshift-ingress/payload-processing-e2e-api-b-f4a54f-5fd8946565-dfltf"
Sep 18 17:39:29.353958 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:29.353926    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca4533b5-8527-43e2-87f2-0e6e7fb7d065-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f4a54f-5dc4c7ff86-5kljr\" (UID: \"ca4533b5-8527-43e2-87f2-0e6e7fb7d065\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f4a54f-5dc4c7ff86-5kljr"
Sep 18 17:39:29.361211 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:29.361175    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jbcw\" (UniqueName: \"kubernetes.io/projected/ca4533b5-8527-43e2-87f2-0e6e7fb7d065-kube-api-access-8jbcw\") pod \"payload-pre-processing-e2e-api-b-f4a54f-5dc4c7ff86-5kljr\" (UID: \"ca4533b5-8527-43e2-87f2-0e6e7fb7d065\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f4a54f-5dc4c7ff86-5kljr"
Sep 18 17:39:29.361346 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:29.361228    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n6nc\" (UniqueName: \"kubernetes.io/projected/24308a53-fe4b-47b4-9ca6-5baf7c47ff1f-kube-api-access-6n6nc\") pod \"payload-processing-e2e-api-b-f4a54f-5fd8946565-dfltf\" (UID: \"24308a53-fe4b-47b4-9ca6-5baf7c47ff1f\") " pod="openshift-ingress/payload-processing-e2e-api-b-f4a54f-5fd8946565-dfltf"
Sep 18 17:39:29.508205 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:29.508173    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f4a54f-5dc4c7ff86-5kljr"
Sep 18 17:39:29.530652 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:29.530619    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f4a54f-5fd8946565-dfltf"
Sep 18 17:39:29.787215 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:29.787190    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f4a54f-5dc4c7ff86-5kljr"]
Sep 18 17:39:29.789084 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:39:29.789058    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca4533b5_8527_43e2_87f2_0e6e7fb7d065.slice/crio-7b2ffb7fa7a9be74fdfa476d5efda1f883d80aa1619185dc080c0bb8bc320089 WatchSource:0}: Error finding container 7b2ffb7fa7a9be74fdfa476d5efda1f883d80aa1619185dc080c0bb8bc320089: Status 404 returned error can't find the container with id 7b2ffb7fa7a9be74fdfa476d5efda1f883d80aa1619185dc080c0bb8bc320089
Sep 18 17:39:29.814311 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:29.814279    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f4a54f-5fd8946565-dfltf"]
Sep 18 17:39:29.822044 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:39:29.822015    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24308a53_fe4b_47b4_9ca6_5baf7c47ff1f.slice/crio-2e9766a6cb95520b7f3620fa652579bf9e1257ed40d02acfdced5ea9965b3312 WatchSource:0}: Error finding container 2e9766a6cb95520b7f3620fa652579bf9e1257ed40d02acfdced5ea9965b3312: Status 404 returned error can't find the container with id 2e9766a6cb95520b7f3620fa652579bf9e1257ed40d02acfdced5ea9965b3312
Sep 18 17:39:30.229895 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:30.229780    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f4a54f-5dc4c7ff86-5kljr" event={"ID":"ca4533b5-8527-43e2-87f2-0e6e7fb7d065","Type":"ContainerStarted","Data":"a25c8dc4b9153b31e64c1b56fac370bb2000ea690ee25bee20a99cef44b95ffb"}
Sep 18 17:39:30.229895 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:30.229845    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f4a54f-5dc4c7ff86-5kljr" event={"ID":"ca4533b5-8527-43e2-87f2-0e6e7fb7d065","Type":"ContainerStarted","Data":"7b2ffb7fa7a9be74fdfa476d5efda1f883d80aa1619185dc080c0bb8bc320089"}
Sep 18 17:39:30.229895 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:30.229885    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f4a54f-5dc4c7ff86-5kljr"
Sep 18 17:39:30.231521 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:30.231497    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f4a54f-5fd8946565-dfltf" event={"ID":"24308a53-fe4b-47b4-9ca6-5baf7c47ff1f","Type":"ContainerStarted","Data":"2695023dbe232561f99890cb96c94d16a093d56afa97a7fe2411bc9d93599005"}
Sep 18 17:39:30.231629 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:30.231535    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f4a54f-5fd8946565-dfltf" event={"ID":"24308a53-fe4b-47b4-9ca6-5baf7c47ff1f","Type":"ContainerStarted","Data":"2e9766a6cb95520b7f3620fa652579bf9e1257ed40d02acfdced5ea9965b3312"}
Sep 18 17:39:30.231681 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:30.231625    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f4a54f-5fd8946565-dfltf"
Sep 18 17:39:30.247916 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:30.247852    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f4a54f-5dc4c7ff86-5kljr" podStartSLOduration=1.247838746 podStartE2EDuration="1.247838746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:39:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:39:30.246323104 +0000 UTC m=+1482.940250645" watchObservedRunningTime="2026-09-18 17:39:30.247838746 +0000 UTC m=+1482.941766324"
Sep 18 17:39:30.265391 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:30.265345    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f4a54f-5fd8946565-dfltf" podStartSLOduration=1.265329113 podStartE2EDuration="1.265329113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:39:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:39:30.263131222 +0000 UTC m=+1482.957058752" watchObservedRunningTime="2026-09-18 17:39:30.265329113 +0000 UTC m=+1482.959256647"
Sep 18 17:39:39.306260 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:39.306218    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a2e13-kqp7t"]
Sep 18 17:39:39.310144 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:39.310122    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a2e13-kqp7t"
Sep 18 17:39:39.312892 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:39.312852    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qt978\""
Sep 18 17:39:39.317731 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:39.317703    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a2e13-kqp7t"]
Sep 18 17:39:39.450413 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:39.450374    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/065001ba-2b57-42dd-af12-11ebc8def128-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-0a2e13-kqp7t\" (UID: \"065001ba-2b57-42dd-af12-11ebc8def128\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a2e13-kqp7t"
Sep 18 17:39:39.552127 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:39.552081    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/065001ba-2b57-42dd-af12-11ebc8def128-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-0a2e13-kqp7t\" (UID: \"065001ba-2b57-42dd-af12-11ebc8def128\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a2e13-kqp7t"
Sep 18 17:39:39.552697 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:39.552677    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/065001ba-2b57-42dd-af12-11ebc8def128-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-0a2e13-kqp7t\" (UID: \"065001ba-2b57-42dd-af12-11ebc8def128\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a2e13-kqp7t"
Sep 18 17:39:39.622749 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:39.622665    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a2e13-kqp7t"
Sep 18 17:39:39.893225 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:39.893199    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a2e13-kqp7t"]
Sep 18 17:39:39.894565 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:39:39.894538    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod065001ba_2b57_42dd_af12_11ebc8def128.slice/crio-de2a9fdc60310f7c8a52a8ec2c8fb7ff2cbbf2ff17b6d95152b147250ebb6c43 WatchSource:0}: Error finding container de2a9fdc60310f7c8a52a8ec2c8fb7ff2cbbf2ff17b6d95152b147250ebb6c43: Status 404 returned error can't find the container with id de2a9fdc60310f7c8a52a8ec2c8fb7ff2cbbf2ff17b6d95152b147250ebb6c43
Sep 18 17:39:40.287941 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:40.287898    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a2e13-kqp7t" event={"ID":"065001ba-2b57-42dd-af12-11ebc8def128","Type":"ContainerStarted","Data":"0d346758e1ef961f7ca714d389203a33755c80d7035f53cd89c79f8d2c2909ac"}
Sep 18 17:39:40.288139 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:40.287985    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a2e13-kqp7t" event={"ID":"065001ba-2b57-42dd-af12-11ebc8def128","Type":"ContainerStarted","Data":"de2a9fdc60310f7c8a52a8ec2c8fb7ff2cbbf2ff17b6d95152b147250ebb6c43"}
Sep 18 17:39:40.305699 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:40.305647    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a2e13-kqp7t" podStartSLOduration=1.305634162 podStartE2EDuration="1.305634162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:39:40.304923784 +0000 UTC m=+1492.998851339" watchObservedRunningTime="2026-09-18 17:39:40.305634162 +0000 UTC m=+1492.999561691"
Sep 18 17:39:41.240611 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:41.240582    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f4a54f-5dc4c7ff86-5kljr"
Sep 18 17:39:41.241070 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:41.240757    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f4a54f-5fd8946565-dfltf"
Sep 18 17:39:41.297294 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:41.297253    2569 generic.go:356] "Generic (PLEG): container finished" podID="065001ba-2b57-42dd-af12-11ebc8def128" containerID="0d346758e1ef961f7ca714d389203a33755c80d7035f53cd89c79f8d2c2909ac" exitCode=0
Sep 18 17:39:41.297446 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:41.297329    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a2e13-kqp7t" event={"ID":"065001ba-2b57-42dd-af12-11ebc8def128","Type":"ContainerDied","Data":"0d346758e1ef961f7ca714d389203a33755c80d7035f53cd89c79f8d2c2909ac"}
Sep 18 17:39:42.432818 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:42.432794    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a2e13-kqp7t"
Sep 18 17:39:42.579417 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:42.579332    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/065001ba-2b57-42dd-af12-11ebc8def128-maas-api-service-ca\") pod \"065001ba-2b57-42dd-af12-11ebc8def128\" (UID: \"065001ba-2b57-42dd-af12-11ebc8def128\") "
Sep 18 17:39:42.579689 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:42.579661    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/065001ba-2b57-42dd-af12-11ebc8def128-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "065001ba-2b57-42dd-af12-11ebc8def128" (UID: "065001ba-2b57-42dd-af12-11ebc8def128"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:39:42.680692 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:42.680660    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/065001ba-2b57-42dd-af12-11ebc8def128-maas-api-service-ca\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:39:43.308932 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:43.308894    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a2e13-kqp7t"
Sep 18 17:39:43.308932 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:43.308909    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a2e13-kqp7t" event={"ID":"065001ba-2b57-42dd-af12-11ebc8def128","Type":"ContainerDied","Data":"de2a9fdc60310f7c8a52a8ec2c8fb7ff2cbbf2ff17b6d95152b147250ebb6c43"}
Sep 18 17:39:43.309150 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:43.308962    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de2a9fdc60310f7c8a52a8ec2c8fb7ff2cbbf2ff17b6d95152b147250ebb6c43"
Sep 18 17:39:44.188316 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:44.188282    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5"]
Sep 18 17:39:44.215696 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:44.215657    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l"]
Sep 18 17:39:44.223298 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:44.223271    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj"]
Sep 18 17:39:45.523469 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:45.523429    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c0254-zgspc"]
Sep 18 17:39:45.524376 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:45.524354    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="065001ba-2b57-42dd-af12-11ebc8def128" containerName="revoke-keys"
Sep 18 17:39:45.524376 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:45.524378    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="065001ba-2b57-42dd-af12-11ebc8def128" containerName="revoke-keys"
Sep 18 17:39:45.524606 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:45.524589    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="065001ba-2b57-42dd-af12-11ebc8def128" containerName="revoke-keys"
Sep 18 17:39:45.528319 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:45.528296    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c0254-zgspc"
Sep 18 17:39:45.532275 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:45.532254    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qt978\""
Sep 18 17:39:45.534802 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:45.534770    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c0254-zgspc"]
Sep 18 17:39:45.607805 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:45.607767    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/819c1de6-0c65-42f9-a929-68e1e9c23811-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9c0254-zgspc\" (UID: \"819c1de6-0c65-42f9-a929-68e1e9c23811\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c0254-zgspc"
Sep 18 17:39:45.709251 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:45.709210    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/819c1de6-0c65-42f9-a929-68e1e9c23811-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9c0254-zgspc\" (UID: \"819c1de6-0c65-42f9-a929-68e1e9c23811\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c0254-zgspc"
Sep 18 17:39:45.709856 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:45.709834    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/819c1de6-0c65-42f9-a929-68e1e9c23811-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9c0254-zgspc\" (UID: \"819c1de6-0c65-42f9-a929-68e1e9c23811\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c0254-zgspc"
Sep 18 17:39:45.841367 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:45.841287    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c0254-zgspc"
Sep 18 17:39:46.114769 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:46.114742    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c0254-zgspc"]
Sep 18 17:39:46.117300 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:39:46.117273    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod819c1de6_0c65_42f9_a929_68e1e9c23811.slice/crio-11c1335d83e34a2f54d63956e01c7d9224d5f04a3610572ce46a82d4587d99c4 WatchSource:0}: Error finding container 11c1335d83e34a2f54d63956e01c7d9224d5f04a3610572ce46a82d4587d99c4: Status 404 returned error can't find the container with id 11c1335d83e34a2f54d63956e01c7d9224d5f04a3610572ce46a82d4587d99c4
Sep 18 17:39:46.325937 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:46.325902    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c0254-zgspc" event={"ID":"819c1de6-0c65-42f9-a929-68e1e9c23811","Type":"ContainerStarted","Data":"11c1335d83e34a2f54d63956e01c7d9224d5f04a3610572ce46a82d4587d99c4"}
Sep 18 17:39:47.333179 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:47.333145    2569 generic.go:356] "Generic (PLEG): container finished" podID="819c1de6-0c65-42f9-a929-68e1e9c23811" containerID="13694fa5f0d8aa26359d65d308346f428ca84a76b8c2f8691f27e325a0beda1b" exitCode=0
Sep 18 17:39:47.333563 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:47.333196    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c0254-zgspc" event={"ID":"819c1de6-0c65-42f9-a929-68e1e9c23811","Type":"ContainerDied","Data":"13694fa5f0d8aa26359d65d308346f428ca84a76b8c2f8691f27e325a0beda1b"}
Sep 18 17:39:48.032914 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:48.032857    2569 scope.go:117] "RemoveContainer" containerID="26751883c6c307e3d48f9399a19c9b4d5f19072a7c1f8c864c2ec8782c7a04fd"
Sep 18 17:39:48.474330 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:48.474305    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c0254-zgspc"
Sep 18 17:39:48.635831 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:48.635742    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/819c1de6-0c65-42f9-a929-68e1e9c23811-maas-api-service-ca\") pod \"819c1de6-0c65-42f9-a929-68e1e9c23811\" (UID: \"819c1de6-0c65-42f9-a929-68e1e9c23811\") "
Sep 18 17:39:48.636147 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:48.636125    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/819c1de6-0c65-42f9-a929-68e1e9c23811-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "819c1de6-0c65-42f9-a929-68e1e9c23811" (UID: "819c1de6-0c65-42f9-a929-68e1e9c23811"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:39:48.736980 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:48.736945    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/819c1de6-0c65-42f9-a929-68e1e9c23811-maas-api-service-ca\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:39:49.344975 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:49.344939    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c0254-zgspc" event={"ID":"819c1de6-0c65-42f9-a929-68e1e9c23811","Type":"ContainerDied","Data":"11c1335d83e34a2f54d63956e01c7d9224d5f04a3610572ce46a82d4587d99c4"}
Sep 18 17:39:49.344975 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:49.344970    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c0254-zgspc"
Sep 18 17:39:49.345188 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:49.345013    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11c1335d83e34a2f54d63956e01c7d9224d5f04a3610572ce46a82d4587d99c4"
Sep 18 17:39:59.208138 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.208075    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5" podUID="6c161326-9a1b-4736-ab01-5280f4073af0" containerName="main" containerID="cri-o://32490aa28dc5c1514ba6bc4dbbf9336f8c9f623da5ecd9af3d39237c2276afe8" gracePeriod=45
Sep 18 17:39:59.234438 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.234401    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l" podUID="0b29ad1f-339b-48f8-bc20-8e51ba37f30c" containerName="main" containerID="cri-o://2f07db007f1fa6f9befdd2f740a9cd7b1678e09a59f5c1dca41e8a0c3867938b" gracePeriod=45
Sep 18 17:39:59.241461 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.241428    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj" podUID="399fd9db-af7b-4e20-9c6b-0d34c703dead" containerName="main" containerID="cri-o://0c6875848524d9efd329a1e7b6b6c12dd4d414f78c5734b2c88c749b8dab7c4f" gracePeriod=45
Sep 18 17:39:59.405348 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.405282    2569 generic.go:356] "Generic (PLEG): container finished" podID="399fd9db-af7b-4e20-9c6b-0d34c703dead" containerID="0c6875848524d9efd329a1e7b6b6c12dd4d414f78c5734b2c88c749b8dab7c4f" exitCode=0
Sep 18 17:39:59.405546 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.405416    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj" event={"ID":"399fd9db-af7b-4e20-9c6b-0d34c703dead","Type":"ContainerDied","Data":"0c6875848524d9efd329a1e7b6b6c12dd4d414f78c5734b2c88c749b8dab7c4f"}
Sep 18 17:39:59.408041 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.408002    2569 generic.go:356] "Generic (PLEG): container finished" podID="0b29ad1f-339b-48f8-bc20-8e51ba37f30c" containerID="2f07db007f1fa6f9befdd2f740a9cd7b1678e09a59f5c1dca41e8a0c3867938b" exitCode=0
Sep 18 17:39:59.408263 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.408211    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l" event={"ID":"0b29ad1f-339b-48f8-bc20-8e51ba37f30c","Type":"ContainerDied","Data":"2f07db007f1fa6f9befdd2f740a9cd7b1678e09a59f5c1dca41e8a0c3867938b"}
Sep 18 17:39:59.410413 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.410383    2569 generic.go:356] "Generic (PLEG): container finished" podID="6c161326-9a1b-4736-ab01-5280f4073af0" containerID="32490aa28dc5c1514ba6bc4dbbf9336f8c9f623da5ecd9af3d39237c2276afe8" exitCode=0
Sep 18 17:39:59.410532 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.410442    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5" event={"ID":"6c161326-9a1b-4736-ab01-5280f4073af0","Type":"ContainerDied","Data":"32490aa28dc5c1514ba6bc4dbbf9336f8c9f623da5ecd9af3d39237c2276afe8"}
Sep 18 17:39:59.420505 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.420482    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a2e13-29829210-fpjfn"]
Sep 18 17:39:59.423966 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.423934    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0a2e13-29829210-fpjfn"]
Sep 18 17:39:59.452482 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.452389    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0a2e13-5bdc574c94-r2lmr"]
Sep 18 17:39:59.452698 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.452610    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a2e13-5bdc574c94-r2lmr" podUID="fa4e3db2-2e1e-42e0-9e05-107039d7ed3e" containerName="payload-pre-processing" containerID="cri-o://911af9e2b551a5008fa5e881bd7b60c056affb9c7d17120242b5d04bc38b2d71" gracePeriod=30
Sep 18 17:39:59.570242 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.570220    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj"
Sep 18 17:39:59.574146 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.573932    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5"
Sep 18 17:39:59.581099 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.581073    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l"
Sep 18 17:39:59.635395 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.635362    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a2e13-kqp7t"]
Sep 18 17:39:59.639164 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.639140    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0a2e13-kqp7t"]
Sep 18 17:39:59.641817 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.641790    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/399fd9db-af7b-4e20-9c6b-0d34c703dead-home\") pod \"399fd9db-af7b-4e20-9c6b-0d34c703dead\" (UID: \"399fd9db-af7b-4e20-9c6b-0d34c703dead\") "
Sep 18 17:39:59.641966 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.641842    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwpm8\" (UniqueName: \"kubernetes.io/projected/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-kube-api-access-zwpm8\") pod \"0b29ad1f-339b-48f8-bc20-8e51ba37f30c\" (UID: \"0b29ad1f-339b-48f8-bc20-8e51ba37f30c\") "
Sep 18 17:39:59.641966 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.641899    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/399fd9db-af7b-4e20-9c6b-0d34c703dead-tls-certs\") pod \"399fd9db-af7b-4e20-9c6b-0d34c703dead\" (UID: \"399fd9db-af7b-4e20-9c6b-0d34c703dead\") "
Sep 18 17:39:59.641966 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.641923    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-model-cache\") pod \"0b29ad1f-339b-48f8-bc20-8e51ba37f30c\" (UID: \"0b29ad1f-339b-48f8-bc20-8e51ba37f30c\") "
Sep 18 17:39:59.641966 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.641949    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-home\") pod \"0b29ad1f-339b-48f8-bc20-8e51ba37f30c\" (UID: \"0b29ad1f-339b-48f8-bc20-8e51ba37f30c\") "
Sep 18 17:39:59.642188 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.641976    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c161326-9a1b-4736-ab01-5280f4073af0-tmp-dir\") pod \"6c161326-9a1b-4736-ab01-5280f4073af0\" (UID: \"6c161326-9a1b-4736-ab01-5280f4073af0\") "
Sep 18 17:39:59.642188 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.642012    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-dshm\") pod \"0b29ad1f-339b-48f8-bc20-8e51ba37f30c\" (UID: \"0b29ad1f-339b-48f8-bc20-8e51ba37f30c\") "
Sep 18 17:39:59.642188 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.642041    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c161326-9a1b-4736-ab01-5280f4073af0-tls-certs\") pod \"6c161326-9a1b-4736-ab01-5280f4073af0\" (UID: \"6c161326-9a1b-4736-ab01-5280f4073af0\") "
Sep 18 17:39:59.642188 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.642064    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c161326-9a1b-4736-ab01-5280f4073af0-home\") pod \"6c161326-9a1b-4736-ab01-5280f4073af0\" (UID: \"6c161326-9a1b-4736-ab01-5280f4073af0\") "
Sep 18 17:39:59.642188 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.642096    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c161326-9a1b-4736-ab01-5280f4073af0-model-cache\") pod \"6c161326-9a1b-4736-ab01-5280f4073af0\" (UID: \"6c161326-9a1b-4736-ab01-5280f4073af0\") "
Sep 18 17:39:59.642188 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.642093    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/399fd9db-af7b-4e20-9c6b-0d34c703dead-home" (OuterVolumeSpecName: "home") pod "399fd9db-af7b-4e20-9c6b-0d34c703dead" (UID: "399fd9db-af7b-4e20-9c6b-0d34c703dead"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:39:59.642188 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.642126    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-model-cache" (OuterVolumeSpecName: "model-cache") pod "0b29ad1f-339b-48f8-bc20-8e51ba37f30c" (UID: "0b29ad1f-339b-48f8-bc20-8e51ba37f30c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:39:59.642188 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.642134    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9scct\" (UniqueName: \"kubernetes.io/projected/399fd9db-af7b-4e20-9c6b-0d34c703dead-kube-api-access-9scct\") pod \"399fd9db-af7b-4e20-9c6b-0d34c703dead\" (UID: \"399fd9db-af7b-4e20-9c6b-0d34c703dead\") "
Sep 18 17:39:59.642188 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.642177    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-tmp-dir\") pod \"0b29ad1f-339b-48f8-bc20-8e51ba37f30c\" (UID: \"0b29ad1f-339b-48f8-bc20-8e51ba37f30c\") "
Sep 18 17:39:59.642631 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.642224    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/399fd9db-af7b-4e20-9c6b-0d34c703dead-dshm\") pod \"399fd9db-af7b-4e20-9c6b-0d34c703dead\" (UID: \"399fd9db-af7b-4e20-9c6b-0d34c703dead\") "
Sep 18 17:39:59.642631 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.642260    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldl9l\" (UniqueName: \"kubernetes.io/projected/6c161326-9a1b-4736-ab01-5280f4073af0-kube-api-access-ldl9l\") pod \"6c161326-9a1b-4736-ab01-5280f4073af0\" (UID: \"6c161326-9a1b-4736-ab01-5280f4073af0\") "
Sep 18 17:39:59.642631 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.642283    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/399fd9db-af7b-4e20-9c6b-0d34c703dead-tmp-dir\") pod \"399fd9db-af7b-4e20-9c6b-0d34c703dead\" (UID: \"399fd9db-af7b-4e20-9c6b-0d34c703dead\") "
Sep 18 17:39:59.642631 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.642319    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-tls-certs\") pod \"0b29ad1f-339b-48f8-bc20-8e51ba37f30c\" (UID: \"0b29ad1f-339b-48f8-bc20-8e51ba37f30c\") "
Sep 18 17:39:59.642631 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.642384    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/399fd9db-af7b-4e20-9c6b-0d34c703dead-model-cache\") pod \"399fd9db-af7b-4e20-9c6b-0d34c703dead\" (UID: \"399fd9db-af7b-4e20-9c6b-0d34c703dead\") "
Sep 18 17:39:59.642631 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.642450    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c161326-9a1b-4736-ab01-5280f4073af0-dshm\") pod \"6c161326-9a1b-4736-ab01-5280f4073af0\" (UID: \"6c161326-9a1b-4736-ab01-5280f4073af0\") "
Sep 18 17:39:59.642968 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.642814    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/399fd9db-af7b-4e20-9c6b-0d34c703dead-home\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:39:59.642968 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.642832    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-model-cache\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:39:59.643108 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.642283    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c161326-9a1b-4736-ab01-5280f4073af0-home" (OuterVolumeSpecName: "home") pod "6c161326-9a1b-4736-ab01-5280f4073af0" (UID: "6c161326-9a1b-4736-ab01-5280f4073af0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:39:59.643194 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.642499    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-home" (OuterVolumeSpecName: "home") pod "0b29ad1f-339b-48f8-bc20-8e51ba37f30c" (UID: "0b29ad1f-339b-48f8-bc20-8e51ba37f30c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:39:59.643534 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.643253    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/399fd9db-af7b-4e20-9c6b-0d34c703dead-model-cache" (OuterVolumeSpecName: "model-cache") pod "399fd9db-af7b-4e20-9c6b-0d34c703dead" (UID: "399fd9db-af7b-4e20-9c6b-0d34c703dead"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:39:59.643534 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.643029    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/399fd9db-af7b-4e20-9c6b-0d34c703dead-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "399fd9db-af7b-4e20-9c6b-0d34c703dead" (UID: "399fd9db-af7b-4e20-9c6b-0d34c703dead"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:39:59.643700 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.643676    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0b29ad1f-339b-48f8-bc20-8e51ba37f30c" (UID: "0b29ad1f-339b-48f8-bc20-8e51ba37f30c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:39:59.644048 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.644001    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c161326-9a1b-4736-ab01-5280f4073af0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6c161326-9a1b-4736-ab01-5280f4073af0" (UID: "6c161326-9a1b-4736-ab01-5280f4073af0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:39:59.644261 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.644223    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c161326-9a1b-4736-ab01-5280f4073af0-model-cache" (OuterVolumeSpecName: "model-cache") pod "6c161326-9a1b-4736-ab01-5280f4073af0" (UID: "6c161326-9a1b-4736-ab01-5280f4073af0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:39:59.646099 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.646060    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-dshm" (OuterVolumeSpecName: "dshm") pod "0b29ad1f-339b-48f8-bc20-8e51ba37f30c" (UID: "0b29ad1f-339b-48f8-bc20-8e51ba37f30c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:39:59.646099 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.646089    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/399fd9db-af7b-4e20-9c6b-0d34c703dead-dshm" (OuterVolumeSpecName: "dshm") pod "399fd9db-af7b-4e20-9c6b-0d34c703dead" (UID: "399fd9db-af7b-4e20-9c6b-0d34c703dead"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:39:59.646275 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.646112    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-kube-api-access-zwpm8" (OuterVolumeSpecName: "kube-api-access-zwpm8") pod "0b29ad1f-339b-48f8-bc20-8e51ba37f30c" (UID: "0b29ad1f-339b-48f8-bc20-8e51ba37f30c"). InnerVolumeSpecName "kube-api-access-zwpm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:39:59.646429 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.646411    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c161326-9a1b-4736-ab01-5280f4073af0-dshm" (OuterVolumeSpecName: "dshm") pod "6c161326-9a1b-4736-ab01-5280f4073af0" (UID: "6c161326-9a1b-4736-ab01-5280f4073af0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:39:59.646501 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.646399    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/399fd9db-af7b-4e20-9c6b-0d34c703dead-kube-api-access-9scct" (OuterVolumeSpecName: "kube-api-access-9scct") pod "399fd9db-af7b-4e20-9c6b-0d34c703dead" (UID: "399fd9db-af7b-4e20-9c6b-0d34c703dead"). InnerVolumeSpecName "kube-api-access-9scct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:39:59.647820 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.647778    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0b29ad1f-339b-48f8-bc20-8e51ba37f30c" (UID: "0b29ad1f-339b-48f8-bc20-8e51ba37f30c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:39:59.647943 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.647830    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c161326-9a1b-4736-ab01-5280f4073af0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6c161326-9a1b-4736-ab01-5280f4073af0" (UID: "6c161326-9a1b-4736-ab01-5280f4073af0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:39:59.647943 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.647905    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/399fd9db-af7b-4e20-9c6b-0d34c703dead-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "399fd9db-af7b-4e20-9c6b-0d34c703dead" (UID: "399fd9db-af7b-4e20-9c6b-0d34c703dead"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:39:59.648066 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.648046    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c161326-9a1b-4736-ab01-5280f4073af0-kube-api-access-ldl9l" (OuterVolumeSpecName: "kube-api-access-ldl9l") pod "6c161326-9a1b-4736-ab01-5280f4073af0" (UID: "6c161326-9a1b-4736-ab01-5280f4073af0"). InnerVolumeSpecName "kube-api-access-ldl9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:39:59.719919 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.719844    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a2e13-5bdc574c94-r2lmr"
Sep 18 17:39:59.743785 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.743759    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72m2p\" (UniqueName: \"kubernetes.io/projected/fa4e3db2-2e1e-42e0-9e05-107039d7ed3e-kube-api-access-72m2p\") pod \"fa4e3db2-2e1e-42e0-9e05-107039d7ed3e\" (UID: \"fa4e3db2-2e1e-42e0-9e05-107039d7ed3e\") "
Sep 18 17:39:59.743943 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.743904    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4e3db2-2e1e-42e0-9e05-107039d7ed3e-plugins-config-volume\") pod \"fa4e3db2-2e1e-42e0-9e05-107039d7ed3e\" (UID: \"fa4e3db2-2e1e-42e0-9e05-107039d7ed3e\") "
Sep 18 17:39:59.744199 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.744184    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/399fd9db-af7b-4e20-9c6b-0d34c703dead-model-cache\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:39:59.744284 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.744207    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c161326-9a1b-4736-ab01-5280f4073af0-dshm\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:39:59.744284 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.744221    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwpm8\" (UniqueName: \"kubernetes.io/projected/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-kube-api-access-zwpm8\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:39:59.744284 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.744233    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/399fd9db-af7b-4e20-9c6b-0d34c703dead-tls-certs\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:39:59.744284 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.744246    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-home\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:39:59.744284 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.744259    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c161326-9a1b-4736-ab01-5280f4073af0-tmp-dir\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:39:59.744284 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.744241    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa4e3db2-2e1e-42e0-9e05-107039d7ed3e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fa4e3db2-2e1e-42e0-9e05-107039d7ed3e" (UID: "fa4e3db2-2e1e-42e0-9e05-107039d7ed3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:39:59.744284 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.744272    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-dshm\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:39:59.744519 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.744307    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c161326-9a1b-4736-ab01-5280f4073af0-tls-certs\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:39:59.744519 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.744317    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c161326-9a1b-4736-ab01-5280f4073af0-home\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:39:59.744519 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.744327    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c161326-9a1b-4736-ab01-5280f4073af0-model-cache\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:39:59.744519 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.744336    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9scct\" (UniqueName: \"kubernetes.io/projected/399fd9db-af7b-4e20-9c6b-0d34c703dead-kube-api-access-9scct\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:39:59.744519 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.744345    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-tmp-dir\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:39:59.744519 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.744354    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/399fd9db-af7b-4e20-9c6b-0d34c703dead-dshm\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:39:59.744519 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.744363    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldl9l\" (UniqueName: \"kubernetes.io/projected/6c161326-9a1b-4736-ab01-5280f4073af0-kube-api-access-ldl9l\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:39:59.744519 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.744372    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/399fd9db-af7b-4e20-9c6b-0d34c703dead-tmp-dir\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:39:59.744519 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.744381    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b29ad1f-339b-48f8-bc20-8e51ba37f30c-tls-certs\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:39:59.745702 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.745685    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa4e3db2-2e1e-42e0-9e05-107039d7ed3e-kube-api-access-72m2p" (OuterVolumeSpecName: "kube-api-access-72m2p") pod "fa4e3db2-2e1e-42e0-9e05-107039d7ed3e" (UID: "fa4e3db2-2e1e-42e0-9e05-107039d7ed3e"). InnerVolumeSpecName "kube-api-access-72m2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:39:59.768946 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.768913    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="065001ba-2b57-42dd-af12-11ebc8def128" path="/var/lib/kubelet/pods/065001ba-2b57-42dd-af12-11ebc8def128/volumes"
Sep 18 17:39:59.769220 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.769208    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63d51e20-2f48-4fbb-ac4e-5ad2377ad376" path="/var/lib/kubelet/pods/63d51e20-2f48-4fbb-ac4e-5ad2377ad376/volumes"
Sep 18 17:39:59.845651 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.845622    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4e3db2-2e1e-42e0-9e05-107039d7ed3e-plugins-config-volume\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:39:59.845651 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:39:59.845645    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72m2p\" (UniqueName: \"kubernetes.io/projected/fa4e3db2-2e1e-42e0-9e05-107039d7ed3e-kube-api-access-72m2p\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:40:00.416880 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:00.416829    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l"
Sep 18 17:40:00.417349 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:00.416893    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l" event={"ID":"0b29ad1f-339b-48f8-bc20-8e51ba37f30c","Type":"ContainerDied","Data":"254a79edce1904920931dcf484ca4e2a1e865b90ff2793121fdcd683daa32988"}
Sep 18 17:40:00.417349 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:00.416943    2569 scope.go:117] "RemoveContainer" containerID="2f07db007f1fa6f9befdd2f740a9cd7b1678e09a59f5c1dca41e8a0c3867938b"
Sep 18 17:40:00.418501 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:00.418474    2569 generic.go:356] "Generic (PLEG): container finished" podID="fa4e3db2-2e1e-42e0-9e05-107039d7ed3e" containerID="911af9e2b551a5008fa5e881bd7b60c056affb9c7d17120242b5d04bc38b2d71" exitCode=0
Sep 18 17:40:00.418669 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:00.418524    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a2e13-5bdc574c94-r2lmr"
Sep 18 17:40:00.418669 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:00.418551    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a2e13-5bdc574c94-r2lmr" event={"ID":"fa4e3db2-2e1e-42e0-9e05-107039d7ed3e","Type":"ContainerDied","Data":"911af9e2b551a5008fa5e881bd7b60c056affb9c7d17120242b5d04bc38b2d71"}
Sep 18 17:40:00.418669 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:00.418614    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0a2e13-5bdc574c94-r2lmr" event={"ID":"fa4e3db2-2e1e-42e0-9e05-107039d7ed3e","Type":"ContainerDied","Data":"a82e572254e629c6be5a95f0bb7b89087bc3870748ff752e4695102c4e2936f2"}
Sep 18 17:40:00.420281 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:00.420252    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5" event={"ID":"6c161326-9a1b-4736-ab01-5280f4073af0","Type":"ContainerDied","Data":"a0baa00b15c351449f0be150ed826ae09e40e1fbc8a05e84694b3b97bc378396"}
Sep 18 17:40:00.420412 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:00.420340    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5"
Sep 18 17:40:00.421991 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:00.421970    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj" event={"ID":"399fd9db-af7b-4e20-9c6b-0d34c703dead","Type":"ContainerDied","Data":"7c75f7552b0daef57a4fa0d35ff4f1556880cbb4d2deb3e427b44d875fec8990"}
Sep 18 17:40:00.422069 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:00.422006    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj"
Sep 18 17:40:00.427645 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:00.427629    2569 scope.go:117] "RemoveContainer" containerID="911af9e2b551a5008fa5e881bd7b60c056affb9c7d17120242b5d04bc38b2d71"
Sep 18 17:40:00.436180 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:00.436162    2569 scope.go:117] "RemoveContainer" containerID="911af9e2b551a5008fa5e881bd7b60c056affb9c7d17120242b5d04bc38b2d71"
Sep 18 17:40:00.436444 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:40:00.436427    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"911af9e2b551a5008fa5e881bd7b60c056affb9c7d17120242b5d04bc38b2d71\": container with ID starting with 911af9e2b551a5008fa5e881bd7b60c056affb9c7d17120242b5d04bc38b2d71 not found: ID does not exist" containerID="911af9e2b551a5008fa5e881bd7b60c056affb9c7d17120242b5d04bc38b2d71"
Sep 18 17:40:00.436490 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:00.436453    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"911af9e2b551a5008fa5e881bd7b60c056affb9c7d17120242b5d04bc38b2d71"} err="failed to get container status \"911af9e2b551a5008fa5e881bd7b60c056affb9c7d17120242b5d04bc38b2d71\": rpc error: code = NotFound desc = could not find container \"911af9e2b551a5008fa5e881bd7b60c056affb9c7d17120242b5d04bc38b2d71\": container with ID starting with 911af9e2b551a5008fa5e881bd7b60c056affb9c7d17120242b5d04bc38b2d71 not found: ID does not exist"
Sep 18 17:40:00.436490 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:00.436474    2569 scope.go:117] "RemoveContainer" containerID="32490aa28dc5c1514ba6bc4dbbf9336f8c9f623da5ecd9af3d39237c2276afe8"
Sep 18 17:40:00.441344 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:00.441325    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l"]
Sep 18 17:40:00.447324 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:00.447301    2569 scope.go:117] "RemoveContainer" containerID="0c6875848524d9efd329a1e7b6b6c12dd4d414f78c5734b2c88c749b8dab7c4f"
Sep 18 17:40:00.448037 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:00.447942    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-0a2e13/premium-w0-0a2e13-kserve-58b9c99c94-8xr5l"]
Sep 18 17:40:00.457499 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:00.457481    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0a2e13-5bdc574c94-r2lmr"]
Sep 18 17:40:00.461458 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:00.461437    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0a2e13-5bdc574c94-r2lmr"]
Sep 18 17:40:00.470876 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:00.470805    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5"]
Sep 18 17:40:00.475590 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:00.475567    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-0a2e13/distinct-2-w0-0a2e13-kserve-5744f5577d-48pn5"]
Sep 18 17:40:00.484963 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:00.484939    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj"]
Sep 18 17:40:00.488677 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:00.488655    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-0a2e13/simulator-w0-0a2e13-kserve-6d85849f99-446rj"]
Sep 18 17:40:01.765471 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:01.765442    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b29ad1f-339b-48f8-bc20-8e51ba37f30c" path="/var/lib/kubelet/pods/0b29ad1f-339b-48f8-bc20-8e51ba37f30c/volumes"
Sep 18 17:40:01.765839 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:01.765832    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="399fd9db-af7b-4e20-9c6b-0d34c703dead" path="/var/lib/kubelet/pods/399fd9db-af7b-4e20-9c6b-0d34c703dead/volumes"
Sep 18 17:40:01.766242 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:01.766228    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c161326-9a1b-4736-ab01-5280f4073af0" path="/var/lib/kubelet/pods/6c161326-9a1b-4736-ab01-5280f4073af0/volumes"
Sep 18 17:40:01.766871 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:01.766843    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa4e3db2-2e1e-42e0-9e05-107039d7ed3e" path="/var/lib/kubelet/pods/fa4e3db2-2e1e-42e0-9e05-107039d7ed3e/volumes"
Sep 18 17:40:04.080393 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:04.080348    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c0254-zgspc"]
Sep 18 17:40:04.082380 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:04.082359    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c0254-zgspc"]
Sep 18 17:40:05.766099 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:05.766063    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="819c1de6-0c65-42f9-a929-68e1e9c23811" path="/var/lib/kubelet/pods/819c1de6-0c65-42f9-a929-68e1e9c23811/volumes"
Sep 18 17:40:11.550666 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:11.550584    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f4a54f-nll9z"]
Sep 18 17:40:11.551043 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:11.550968    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="399fd9db-af7b-4e20-9c6b-0d34c703dead" containerName="main"
Sep 18 17:40:11.551043 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:11.550980    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="399fd9db-af7b-4e20-9c6b-0d34c703dead" containerName="main"
Sep 18 17:40:11.551043 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:11.550990    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="819c1de6-0c65-42f9-a929-68e1e9c23811" containerName="revoke-keys"
Sep 18 17:40:11.551043 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:11.550997    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="819c1de6-0c65-42f9-a929-68e1e9c23811" containerName="revoke-keys"
Sep 18 17:40:11.551043 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:11.551018    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c161326-9a1b-4736-ab01-5280f4073af0" containerName="main"
Sep 18 17:40:11.551043 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:11.551024    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c161326-9a1b-4736-ab01-5280f4073af0" containerName="main"
Sep 18 17:40:11.551043 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:11.551031    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa4e3db2-2e1e-42e0-9e05-107039d7ed3e" containerName="payload-pre-processing"
Sep 18 17:40:11.551043 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:11.551036    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa4e3db2-2e1e-42e0-9e05-107039d7ed3e" containerName="payload-pre-processing"
Sep 18 17:40:11.551043 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:11.551044    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b29ad1f-339b-48f8-bc20-8e51ba37f30c" containerName="main"
Sep 18 17:40:11.551328 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:11.551051    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b29ad1f-339b-48f8-bc20-8e51ba37f30c" containerName="main"
Sep 18 17:40:11.551328 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:11.551106    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="399fd9db-af7b-4e20-9c6b-0d34c703dead" containerName="main"
Sep 18 17:40:11.551328 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:11.551114    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b29ad1f-339b-48f8-bc20-8e51ba37f30c" containerName="main"
Sep 18 17:40:11.551328 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:11.551119    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="819c1de6-0c65-42f9-a929-68e1e9c23811" containerName="revoke-keys"
Sep 18 17:40:11.551328 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:11.551126    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c161326-9a1b-4736-ab01-5280f4073af0" containerName="main"
Sep 18 17:40:11.551328 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:11.551131    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa4e3db2-2e1e-42e0-9e05-107039d7ed3e" containerName="payload-pre-processing"
Sep 18 17:40:11.554249 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:11.554234    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f4a54f-nll9z"
Sep 18 17:40:11.556448 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:11.556426    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qt978\""
Sep 18 17:40:11.562216 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:11.562191    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f4a54f-nll9z"]
Sep 18 17:40:11.652962 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:11.652931    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b68d15f-a84b-4a54-8a4d-ad4e6b581bfe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f4a54f-nll9z\" (UID: \"5b68d15f-a84b-4a54-8a4d-ad4e6b581bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f4a54f-nll9z"
Sep 18 17:40:11.753992 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:11.753952    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b68d15f-a84b-4a54-8a4d-ad4e6b581bfe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f4a54f-nll9z\" (UID: \"5b68d15f-a84b-4a54-8a4d-ad4e6b581bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f4a54f-nll9z"
Sep 18 17:40:11.754831 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:11.754811    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b68d15f-a84b-4a54-8a4d-ad4e6b581bfe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f4a54f-nll9z\" (UID: \"5b68d15f-a84b-4a54-8a4d-ad4e6b581bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f4a54f-nll9z"
Sep 18 17:40:11.865960 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:11.865853    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f4a54f-nll9z"
Sep 18 17:40:12.153359 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:12.153327    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f4a54f-nll9z"]
Sep 18 17:40:12.154835 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:40:12.154806    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b68d15f_a84b_4a54_8a4d_ad4e6b581bfe.slice/crio-9b66f4165e0c1636b57739453c994f44e926e208b31b3cb28ad97917a7717650 WatchSource:0}: Error finding container 9b66f4165e0c1636b57739453c994f44e926e208b31b3cb28ad97917a7717650: Status 404 returned error can't find the container with id 9b66f4165e0c1636b57739453c994f44e926e208b31b3cb28ad97917a7717650
Sep 18 17:40:12.490996 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:12.490961    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f4a54f-nll9z" event={"ID":"5b68d15f-a84b-4a54-8a4d-ad4e6b581bfe","Type":"ContainerStarted","Data":"c2cdfbc54911028276eaf4e5057fa8b9f867131f71d9d56318fdabab9f606338"}
Sep 18 17:40:12.491143 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:12.491020    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f4a54f-nll9z" event={"ID":"5b68d15f-a84b-4a54-8a4d-ad4e6b581bfe","Type":"ContainerStarted","Data":"9b66f4165e0c1636b57739453c994f44e926e208b31b3cb28ad97917a7717650"}
Sep 18 17:40:13.497921 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:13.497889    2569 generic.go:356] "Generic (PLEG): container finished" podID="5b68d15f-a84b-4a54-8a4d-ad4e6b581bfe" containerID="c2cdfbc54911028276eaf4e5057fa8b9f867131f71d9d56318fdabab9f606338" exitCode=0
Sep 18 17:40:13.498375 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:13.497951    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f4a54f-nll9z" event={"ID":"5b68d15f-a84b-4a54-8a4d-ad4e6b581bfe","Type":"ContainerDied","Data":"c2cdfbc54911028276eaf4e5057fa8b9f867131f71d9d56318fdabab9f606338"}
Sep 18 17:40:14.631427 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:14.631404    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f4a54f-nll9z"
Sep 18 17:40:14.681496 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:14.681465    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b68d15f-a84b-4a54-8a4d-ad4e6b581bfe-maas-api-service-ca\") pod \"5b68d15f-a84b-4a54-8a4d-ad4e6b581bfe\" (UID: \"5b68d15f-a84b-4a54-8a4d-ad4e6b581bfe\") "
Sep 18 17:40:14.681843 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:14.681818    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b68d15f-a84b-4a54-8a4d-ad4e6b581bfe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5b68d15f-a84b-4a54-8a4d-ad4e6b581bfe" (UID: "5b68d15f-a84b-4a54-8a4d-ad4e6b581bfe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:40:14.783144 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:14.783065    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b68d15f-a84b-4a54-8a4d-ad4e6b581bfe-maas-api-service-ca\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:40:15.509280 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:15.509247    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f4a54f-nll9z"
Sep 18 17:40:15.509461 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:15.509286    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f4a54f-nll9z" event={"ID":"5b68d15f-a84b-4a54-8a4d-ad4e6b581bfe","Type":"ContainerDied","Data":"9b66f4165e0c1636b57739453c994f44e926e208b31b3cb28ad97917a7717650"}
Sep 18 17:40:15.509461 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:15.509334    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b66f4165e0c1636b57739453c994f44e926e208b31b3cb28ad97917a7717650"
Sep 18 17:40:27.381828 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:27.381793    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f4a54f-5fd8946565-dfltf"]
Sep 18 17:40:27.382354 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:27.382018    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f4a54f-5fd8946565-dfltf" podUID="24308a53-fe4b-47b4-9ca6-5baf7c47ff1f" containerName="payload-processing" containerID="cri-o://2695023dbe232561f99890cb96c94d16a093d56afa97a7fe2411bc9d93599005" gracePeriod=30
Sep 18 17:40:27.388651 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:27.388620    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f4a54f-5dc4c7ff86-5kljr"]
Sep 18 17:40:27.388914 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:27.388888    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f4a54f-5dc4c7ff86-5kljr" podUID="ca4533b5-8527-43e2-87f2-0e6e7fb7d065" containerName="payload-pre-processing" containerID="cri-o://a25c8dc4b9153b31e64c1b56fac370bb2000ea690ee25bee20a99cef44b95ffb" gracePeriod=30
Sep 18 17:40:27.535432 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:27.535363    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f4a54f-nll9z"]
Sep 18 17:40:27.537505 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:27.537481    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f4a54f-nll9z"]
Sep 18 17:40:27.576419 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:27.576386    2569 generic.go:356] "Generic (PLEG): container finished" podID="ca4533b5-8527-43e2-87f2-0e6e7fb7d065" containerID="a25c8dc4b9153b31e64c1b56fac370bb2000ea690ee25bee20a99cef44b95ffb" exitCode=0
Sep 18 17:40:27.576589 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:27.576463    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f4a54f-5dc4c7ff86-5kljr" event={"ID":"ca4533b5-8527-43e2-87f2-0e6e7fb7d065","Type":"ContainerDied","Data":"a25c8dc4b9153b31e64c1b56fac370bb2000ea690ee25bee20a99cef44b95ffb"}
Sep 18 17:40:27.578200 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:27.578179    2569 generic.go:356] "Generic (PLEG): container finished" podID="24308a53-fe4b-47b4-9ca6-5baf7c47ff1f" containerID="2695023dbe232561f99890cb96c94d16a093d56afa97a7fe2411bc9d93599005" exitCode=0
Sep 18 17:40:27.578308 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:27.578215    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f4a54f-5fd8946565-dfltf" event={"ID":"24308a53-fe4b-47b4-9ca6-5baf7c47ff1f","Type":"ContainerDied","Data":"2695023dbe232561f99890cb96c94d16a093d56afa97a7fe2411bc9d93599005"}
Sep 18 17:40:27.660063 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:27.660038    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f4a54f-5fd8946565-dfltf"
Sep 18 17:40:27.662708 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:27.662690    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f4a54f-5dc4c7ff86-5kljr"
Sep 18 17:40:27.765537 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:27.765499    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b68d15f-a84b-4a54-8a4d-ad4e6b581bfe" path="/var/lib/kubelet/pods/5b68d15f-a84b-4a54-8a4d-ad4e6b581bfe/volumes"
Sep 18 17:40:27.813983 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:27.813954    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6n6nc\" (UniqueName: \"kubernetes.io/projected/24308a53-fe4b-47b4-9ca6-5baf7c47ff1f-kube-api-access-6n6nc\") pod \"24308a53-fe4b-47b4-9ca6-5baf7c47ff1f\" (UID: \"24308a53-fe4b-47b4-9ca6-5baf7c47ff1f\") "
Sep 18 17:40:27.814158 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:27.814007    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jbcw\" (UniqueName: \"kubernetes.io/projected/ca4533b5-8527-43e2-87f2-0e6e7fb7d065-kube-api-access-8jbcw\") pod \"ca4533b5-8527-43e2-87f2-0e6e7fb7d065\" (UID: \"ca4533b5-8527-43e2-87f2-0e6e7fb7d065\") "
Sep 18 17:40:27.814158 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:27.814106    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca4533b5-8527-43e2-87f2-0e6e7fb7d065-plugins-config-volume\") pod \"ca4533b5-8527-43e2-87f2-0e6e7fb7d065\" (UID: \"ca4533b5-8527-43e2-87f2-0e6e7fb7d065\") "
Sep 18 17:40:27.814158 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:27.814138    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24308a53-fe4b-47b4-9ca6-5baf7c47ff1f-plugins-config-volume\") pod \"24308a53-fe4b-47b4-9ca6-5baf7c47ff1f\" (UID: \"24308a53-fe4b-47b4-9ca6-5baf7c47ff1f\") "
Sep 18 17:40:27.814545 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:27.814512    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca4533b5-8527-43e2-87f2-0e6e7fb7d065-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ca4533b5-8527-43e2-87f2-0e6e7fb7d065" (UID: "ca4533b5-8527-43e2-87f2-0e6e7fb7d065"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:40:27.814650 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:27.814566    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24308a53-fe4b-47b4-9ca6-5baf7c47ff1f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "24308a53-fe4b-47b4-9ca6-5baf7c47ff1f" (UID: "24308a53-fe4b-47b4-9ca6-5baf7c47ff1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:40:27.816214 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:27.816195    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24308a53-fe4b-47b4-9ca6-5baf7c47ff1f-kube-api-access-6n6nc" (OuterVolumeSpecName: "kube-api-access-6n6nc") pod "24308a53-fe4b-47b4-9ca6-5baf7c47ff1f" (UID: "24308a53-fe4b-47b4-9ca6-5baf7c47ff1f"). InnerVolumeSpecName "kube-api-access-6n6nc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:40:27.816313 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:27.816221    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca4533b5-8527-43e2-87f2-0e6e7fb7d065-kube-api-access-8jbcw" (OuterVolumeSpecName: "kube-api-access-8jbcw") pod "ca4533b5-8527-43e2-87f2-0e6e7fb7d065" (UID: "ca4533b5-8527-43e2-87f2-0e6e7fb7d065"). InnerVolumeSpecName "kube-api-access-8jbcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:40:27.915423 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:27.915345    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca4533b5-8527-43e2-87f2-0e6e7fb7d065-plugins-config-volume\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:40:27.915423 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:27.915373    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24308a53-fe4b-47b4-9ca6-5baf7c47ff1f-plugins-config-volume\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:40:27.915423 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:27.915384    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n6nc\" (UniqueName: \"kubernetes.io/projected/24308a53-fe4b-47b4-9ca6-5baf7c47ff1f-kube-api-access-6n6nc\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:40:27.915423 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:27.915394    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jbcw\" (UniqueName: \"kubernetes.io/projected/ca4533b5-8527-43e2-87f2-0e6e7fb7d065-kube-api-access-8jbcw\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:40:28.585812 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:28.585779    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f4a54f-5fd8946565-dfltf"
Sep 18 17:40:28.586269 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:28.585817    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f4a54f-5fd8946565-dfltf" event={"ID":"24308a53-fe4b-47b4-9ca6-5baf7c47ff1f","Type":"ContainerDied","Data":"2e9766a6cb95520b7f3620fa652579bf9e1257ed40d02acfdced5ea9965b3312"}
Sep 18 17:40:28.586269 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:28.585912    2569 scope.go:117] "RemoveContainer" containerID="2695023dbe232561f99890cb96c94d16a093d56afa97a7fe2411bc9d93599005"
Sep 18 17:40:28.587332 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:28.587207    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f4a54f-5dc4c7ff86-5kljr"
Sep 18 17:40:28.587332 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:28.587274    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f4a54f-5dc4c7ff86-5kljr" event={"ID":"ca4533b5-8527-43e2-87f2-0e6e7fb7d065","Type":"ContainerDied","Data":"7b2ffb7fa7a9be74fdfa476d5efda1f883d80aa1619185dc080c0bb8bc320089"}
Sep 18 17:40:28.597565 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:28.597541    2569 scope.go:117] "RemoveContainer" containerID="a25c8dc4b9153b31e64c1b56fac370bb2000ea690ee25bee20a99cef44b95ffb"
Sep 18 17:40:28.613644 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:28.613620    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f4a54f-5fd8946565-dfltf"]
Sep 18 17:40:28.617959 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:28.617936    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f4a54f-5fd8946565-dfltf"]
Sep 18 17:40:28.628305 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:28.628287    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f4a54f-5dc4c7ff86-5kljr"]
Sep 18 17:40:28.632318 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:28.632300    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f4a54f-5dc4c7ff86-5kljr"]
Sep 18 17:40:29.765533 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:29.765503    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24308a53-fe4b-47b4-9ca6-5baf7c47ff1f" path="/var/lib/kubelet/pods/24308a53-fe4b-47b4-9ca6-5baf7c47ff1f/volumes"
Sep 18 17:40:29.765993 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:29.765972    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca4533b5-8527-43e2-87f2-0e6e7fb7d065" path="/var/lib/kubelet/pods/ca4533b5-8527-43e2-87f2-0e6e7fb7d065/volumes"
Sep 18 17:40:34.933109 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:34.933068    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dd235c-c9l44"]
Sep 18 17:40:34.933607 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:34.933585    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca4533b5-8527-43e2-87f2-0e6e7fb7d065" containerName="payload-pre-processing"
Sep 18 17:40:34.933686 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:34.933611    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca4533b5-8527-43e2-87f2-0e6e7fb7d065" containerName="payload-pre-processing"
Sep 18 17:40:34.933686 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:34.933627    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b68d15f-a84b-4a54-8a4d-ad4e6b581bfe" containerName="revoke-keys"
Sep 18 17:40:34.933686 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:34.933637    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b68d15f-a84b-4a54-8a4d-ad4e6b581bfe" containerName="revoke-keys"
Sep 18 17:40:34.933686 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:34.933673    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24308a53-fe4b-47b4-9ca6-5baf7c47ff1f" containerName="payload-processing"
Sep 18 17:40:34.933686 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:34.933681    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="24308a53-fe4b-47b4-9ca6-5baf7c47ff1f" containerName="payload-processing"
Sep 18 17:40:34.933973 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:34.933783    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca4533b5-8527-43e2-87f2-0e6e7fb7d065" containerName="payload-pre-processing"
Sep 18 17:40:34.933973 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:34.933796    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="24308a53-fe4b-47b4-9ca6-5baf7c47ff1f" containerName="payload-processing"
Sep 18 17:40:34.933973 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:34.933809    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b68d15f-a84b-4a54-8a4d-ad4e6b581bfe" containerName="revoke-keys"
Sep 18 17:40:34.938306 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:34.938284    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dd235c-c9l44"
Sep 18 17:40:34.940781 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:34.940756    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qt978\""
Sep 18 17:40:34.945129 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:34.945105    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dd235c-c9l44"]
Sep 18 17:40:35.081723 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:35.081678    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97443689-98b6-47f8-8ad2-7131536c262a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-dd235c-c9l44\" (UID: \"97443689-98b6-47f8-8ad2-7131536c262a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dd235c-c9l44"
Sep 18 17:40:35.183386 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:35.183303    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97443689-98b6-47f8-8ad2-7131536c262a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-dd235c-c9l44\" (UID: \"97443689-98b6-47f8-8ad2-7131536c262a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dd235c-c9l44"
Sep 18 17:40:35.183945 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:35.183925    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97443689-98b6-47f8-8ad2-7131536c262a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-dd235c-c9l44\" (UID: \"97443689-98b6-47f8-8ad2-7131536c262a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dd235c-c9l44"
Sep 18 17:40:35.250404 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:35.250375    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dd235c-c9l44"
Sep 18 17:40:35.524582 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:35.524555    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dd235c-c9l44"]
Sep 18 17:40:35.527289 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:40:35.527260    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97443689_98b6_47f8_8ad2_7131536c262a.slice/crio-f6241291268e985817f7ede54756a69e7ec0afb9e630820944f6cec3162d2be6 WatchSource:0}: Error finding container f6241291268e985817f7ede54756a69e7ec0afb9e630820944f6cec3162d2be6: Status 404 returned error can't find the container with id f6241291268e985817f7ede54756a69e7ec0afb9e630820944f6cec3162d2be6
Sep 18 17:40:35.628314 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:35.628280    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dd235c-c9l44" event={"ID":"97443689-98b6-47f8-8ad2-7131536c262a","Type":"ContainerStarted","Data":"f6241291268e985817f7ede54756a69e7ec0afb9e630820944f6cec3162d2be6"}
Sep 18 17:40:36.635494 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:36.635403    2569 generic.go:356] "Generic (PLEG): container finished" podID="97443689-98b6-47f8-8ad2-7131536c262a" containerID="69399f7edca4cb83e019a4960ff145634cac37532ef6bf1eb7ec1d1f5ad3407d" exitCode=0
Sep 18 17:40:36.635852 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:36.635493    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dd235c-c9l44" event={"ID":"97443689-98b6-47f8-8ad2-7131536c262a","Type":"ContainerDied","Data":"69399f7edca4cb83e019a4960ff145634cac37532ef6bf1eb7ec1d1f5ad3407d"}
Sep 18 17:40:37.773762 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:37.773718    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dd235c-c9l44"
Sep 18 17:40:37.911915 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:37.911804    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97443689-98b6-47f8-8ad2-7131536c262a-maas-api-service-ca\") pod \"97443689-98b6-47f8-8ad2-7131536c262a\" (UID: \"97443689-98b6-47f8-8ad2-7131536c262a\") "
Sep 18 17:40:37.912167 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:37.912143    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97443689-98b6-47f8-8ad2-7131536c262a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "97443689-98b6-47f8-8ad2-7131536c262a" (UID: "97443689-98b6-47f8-8ad2-7131536c262a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:40:38.012605 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:38.012574    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97443689-98b6-47f8-8ad2-7131536c262a-maas-api-service-ca\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:40:38.647156 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:38.647122    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dd235c-c9l44"
Sep 18 17:40:38.647156 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:38.647139    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dd235c-c9l44" event={"ID":"97443689-98b6-47f8-8ad2-7131536c262a","Type":"ContainerDied","Data":"f6241291268e985817f7ede54756a69e7ec0afb9e630820944f6cec3162d2be6"}
Sep 18 17:40:38.647372 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:38.647183    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6241291268e985817f7ede54756a69e7ec0afb9e630820944f6cec3162d2be6"
Sep 18 17:40:45.402760 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.402727    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl"]
Sep 18 17:40:45.403259 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.402995    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl" podUID="87dc18db-0ad6-4e73-9c6d-8be180920810" containerName="maas-api" containerID="cri-o://97f88de0a52477c8fefe09506ac6b9d623ef6242843cd2a822b7a44a7fbd94d4" gracePeriod=30
Sep 18 17:40:45.455668 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.455636    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dd235c-6cf55ffd4c-k4gv9"]
Sep 18 17:40:45.455925 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.455900    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dd235c-6cf55ffd4c-k4gv9" podUID="73c3ef72-b53a-41af-899b-e3c9b7aed1d6" containerName="payload-processing" containerID="cri-o://9e2fae374220f7034ea25bf4c86fa9e46231a57ede4b836621507fd3427b4910" gracePeriod=30
Sep 18 17:40:45.465439 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.465410    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dd235c-575594b894-wvv67"]
Sep 18 17:40:45.465684 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.465640    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dd235c-575594b894-wvv67" podUID="986f5132-20cd-4154-b364-b7d352e5324b" containerName="payload-pre-processing" containerID="cri-o://c3649d8a019a83942a706bb90c59efec831022576433a33ff024e13e28409722" gracePeriod=30
Sep 18 17:40:45.638700 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.638667    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dd235c-c9l44"]
Sep 18 17:40:45.644559 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.644531    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dd235c-c9l44"]
Sep 18 17:40:45.688364 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.688259    2569 generic.go:356] "Generic (PLEG): container finished" podID="87dc18db-0ad6-4e73-9c6d-8be180920810" containerID="97f88de0a52477c8fefe09506ac6b9d623ef6242843cd2a822b7a44a7fbd94d4" exitCode=0
Sep 18 17:40:45.688517 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.688410    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl" event={"ID":"87dc18db-0ad6-4e73-9c6d-8be180920810","Type":"ContainerDied","Data":"97f88de0a52477c8fefe09506ac6b9d623ef6242843cd2a822b7a44a7fbd94d4"}
Sep 18 17:40:45.690321 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.690296    2569 generic.go:356] "Generic (PLEG): container finished" podID="986f5132-20cd-4154-b364-b7d352e5324b" containerID="c3649d8a019a83942a706bb90c59efec831022576433a33ff024e13e28409722" exitCode=0
Sep 18 17:40:45.690443 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.690336    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dd235c-575594b894-wvv67" event={"ID":"986f5132-20cd-4154-b364-b7d352e5324b","Type":"ContainerDied","Data":"c3649d8a019a83942a706bb90c59efec831022576433a33ff024e13e28409722"}
Sep 18 17:40:45.692671 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.692651    2569 generic.go:356] "Generic (PLEG): container finished" podID="73c3ef72-b53a-41af-899b-e3c9b7aed1d6" containerID="9e2fae374220f7034ea25bf4c86fa9e46231a57ede4b836621507fd3427b4910" exitCode=0
Sep 18 17:40:45.692754 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.692697    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dd235c-6cf55ffd4c-k4gv9" event={"ID":"73c3ef72-b53a-41af-899b-e3c9b7aed1d6","Type":"ContainerDied","Data":"9e2fae374220f7034ea25bf4c86fa9e46231a57ede4b836621507fd3427b4910"}
Sep 18 17:40:45.740511 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.740487    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dd235c-6cf55ffd4c-k4gv9"
Sep 18 17:40:45.753044 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.753022    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl"
Sep 18 17:40:45.756967 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.756948    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dd235c-575594b894-wvv67"
Sep 18 17:40:45.766260 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.766234    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97443689-98b6-47f8-8ad2-7131536c262a" path="/var/lib/kubelet/pods/97443689-98b6-47f8-8ad2-7131536c262a/volumes"
Sep 18 17:40:45.781568 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.781540    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/986f5132-20cd-4154-b364-b7d352e5324b-plugins-config-volume\") pod \"986f5132-20cd-4154-b364-b7d352e5324b\" (UID: \"986f5132-20cd-4154-b364-b7d352e5324b\") "
Sep 18 17:40:45.782562 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.781602    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twwh7\" (UniqueName: \"kubernetes.io/projected/986f5132-20cd-4154-b364-b7d352e5324b-kube-api-access-twwh7\") pod \"986f5132-20cd-4154-b364-b7d352e5324b\" (UID: \"986f5132-20cd-4154-b364-b7d352e5324b\") "
Sep 18 17:40:45.782562 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.781631    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c3ef72-b53a-41af-899b-e3c9b7aed1d6-plugins-config-volume\") pod \"73c3ef72-b53a-41af-899b-e3c9b7aed1d6\" (UID: \"73c3ef72-b53a-41af-899b-e3c9b7aed1d6\") "
Sep 18 17:40:45.782562 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.781668    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87dc18db-0ad6-4e73-9c6d-8be180920810-maas-api-tls\") pod \"87dc18db-0ad6-4e73-9c6d-8be180920810\" (UID: \"87dc18db-0ad6-4e73-9c6d-8be180920810\") "
Sep 18 17:40:45.782562 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.781732    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fqzv\" (UniqueName: \"kubernetes.io/projected/87dc18db-0ad6-4e73-9c6d-8be180920810-kube-api-access-6fqzv\") pod \"87dc18db-0ad6-4e73-9c6d-8be180920810\" (UID: \"87dc18db-0ad6-4e73-9c6d-8be180920810\") "
Sep 18 17:40:45.782562 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.781786    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87dc18db-0ad6-4e73-9c6d-8be180920810-metrics-certs\") pod \"87dc18db-0ad6-4e73-9c6d-8be180920810\" (UID: \"87dc18db-0ad6-4e73-9c6d-8be180920810\") "
Sep 18 17:40:45.782562 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.781837    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92ng9\" (UniqueName: \"kubernetes.io/projected/73c3ef72-b53a-41af-899b-e3c9b7aed1d6-kube-api-access-92ng9\") pod \"73c3ef72-b53a-41af-899b-e3c9b7aed1d6\" (UID: \"73c3ef72-b53a-41af-899b-e3c9b7aed1d6\") "
Sep 18 17:40:45.782562 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.781899    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87dc18db-0ad6-4e73-9c6d-8be180920810-openshift-service-ca\") pod \"87dc18db-0ad6-4e73-9c6d-8be180920810\" (UID: \"87dc18db-0ad6-4e73-9c6d-8be180920810\") "
Sep 18 17:40:45.782562 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.781972    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/986f5132-20cd-4154-b364-b7d352e5324b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "986f5132-20cd-4154-b364-b7d352e5324b" (UID: "986f5132-20cd-4154-b364-b7d352e5324b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:40:45.782562 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.782200    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/986f5132-20cd-4154-b364-b7d352e5324b-plugins-config-volume\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:40:45.782562 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.782509    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87dc18db-0ad6-4e73-9c6d-8be180920810-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "87dc18db-0ad6-4e73-9c6d-8be180920810" (UID: "87dc18db-0ad6-4e73-9c6d-8be180920810"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:40:45.784095 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.784052    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73c3ef72-b53a-41af-899b-e3c9b7aed1d6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "73c3ef72-b53a-41af-899b-e3c9b7aed1d6" (UID: "73c3ef72-b53a-41af-899b-e3c9b7aed1d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:40:45.784683 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.784620    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87dc18db-0ad6-4e73-9c6d-8be180920810-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "87dc18db-0ad6-4e73-9c6d-8be180920810" (UID: "87dc18db-0ad6-4e73-9c6d-8be180920810"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:40:45.784993 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.784960    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c3ef72-b53a-41af-899b-e3c9b7aed1d6-kube-api-access-92ng9" (OuterVolumeSpecName: "kube-api-access-92ng9") pod "73c3ef72-b53a-41af-899b-e3c9b7aed1d6" (UID: "73c3ef72-b53a-41af-899b-e3c9b7aed1d6"). InnerVolumeSpecName "kube-api-access-92ng9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:40:45.785662 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.785617    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87dc18db-0ad6-4e73-9c6d-8be180920810-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "87dc18db-0ad6-4e73-9c6d-8be180920810" (UID: "87dc18db-0ad6-4e73-9c6d-8be180920810"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:40:45.785831 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.785778    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/986f5132-20cd-4154-b364-b7d352e5324b-kube-api-access-twwh7" (OuterVolumeSpecName: "kube-api-access-twwh7") pod "986f5132-20cd-4154-b364-b7d352e5324b" (UID: "986f5132-20cd-4154-b364-b7d352e5324b"). InnerVolumeSpecName "kube-api-access-twwh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:40:45.786329 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.786302    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87dc18db-0ad6-4e73-9c6d-8be180920810-kube-api-access-6fqzv" (OuterVolumeSpecName: "kube-api-access-6fqzv") pod "87dc18db-0ad6-4e73-9c6d-8be180920810" (UID: "87dc18db-0ad6-4e73-9c6d-8be180920810"). InnerVolumeSpecName "kube-api-access-6fqzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:40:45.883516 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.883477    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twwh7\" (UniqueName: \"kubernetes.io/projected/986f5132-20cd-4154-b364-b7d352e5324b-kube-api-access-twwh7\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:40:45.883516 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.883508    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c3ef72-b53a-41af-899b-e3c9b7aed1d6-plugins-config-volume\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:40:45.883516 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.883520    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87dc18db-0ad6-4e73-9c6d-8be180920810-maas-api-tls\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:40:45.883744 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.883530    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fqzv\" (UniqueName: \"kubernetes.io/projected/87dc18db-0ad6-4e73-9c6d-8be180920810-kube-api-access-6fqzv\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:40:45.883744 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.883539    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87dc18db-0ad6-4e73-9c6d-8be180920810-metrics-certs\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:40:45.883744 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.883547    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92ng9\" (UniqueName: \"kubernetes.io/projected/73c3ef72-b53a-41af-899b-e3c9b7aed1d6-kube-api-access-92ng9\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:40:45.883744 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:45.883556    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87dc18db-0ad6-4e73-9c6d-8be180920810-openshift-service-ca\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:40:46.699019 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:46.698927    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dd235c-575594b894-wvv67"
Sep 18 17:40:46.699019 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:46.698972    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dd235c-575594b894-wvv67" event={"ID":"986f5132-20cd-4154-b364-b7d352e5324b","Type":"ContainerDied","Data":"df97cccebbf31e6ad0f01d0c6258e55055fac24cffd41bf73acb6cb27e245e45"}
Sep 18 17:40:46.699545 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:46.699042    2569 scope.go:117] "RemoveContainer" containerID="c3649d8a019a83942a706bb90c59efec831022576433a33ff024e13e28409722"
Sep 18 17:40:46.700499 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:46.700485    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dd235c-6cf55ffd4c-k4gv9"
Sep 18 17:40:46.700569 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:46.700501    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dd235c-6cf55ffd4c-k4gv9" event={"ID":"73c3ef72-b53a-41af-899b-e3c9b7aed1d6","Type":"ContainerDied","Data":"13526d3aace981831b457463d47421aa6fcf91715c77e0c037e781eab394772d"}
Sep 18 17:40:46.702032 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:46.702008    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl" event={"ID":"87dc18db-0ad6-4e73-9c6d-8be180920810","Type":"ContainerDied","Data":"7afb66d7754960f1131d7d5d5fa045c888cf48dcfaf738297d5502694b923034"}
Sep 18 17:40:46.702141 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:46.702010    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl"
Sep 18 17:40:46.710167 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:46.710147    2569 scope.go:117] "RemoveContainer" containerID="9e2fae374220f7034ea25bf4c86fa9e46231a57ede4b836621507fd3427b4910"
Sep 18 17:40:46.718647 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:46.718631    2569 scope.go:117] "RemoveContainer" containerID="97f88de0a52477c8fefe09506ac6b9d623ef6242843cd2a822b7a44a7fbd94d4"
Sep 18 17:40:46.741759 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:46.741733    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dd235c-575594b894-wvv67"]
Sep 18 17:40:46.759134 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:46.759112    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dd235c-575594b894-wvv67"]
Sep 18 17:40:46.783830 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:46.783802    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dd235c-6cf55ffd4c-k4gv9"]
Sep 18 17:40:46.791330 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:46.791304    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dd235c-6cf55ffd4c-k4gv9"]
Sep 18 17:40:46.811223 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:46.811194    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl"]
Sep 18 17:40:46.817333 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:46.817309    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dd235c-6948c6dd-7d8tl"]
Sep 18 17:40:47.765287 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:47.765255    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73c3ef72-b53a-41af-899b-e3c9b7aed1d6" path="/var/lib/kubelet/pods/73c3ef72-b53a-41af-899b-e3c9b7aed1d6/volumes"
Sep 18 17:40:47.765748 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:47.765733    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87dc18db-0ad6-4e73-9c6d-8be180920810" path="/var/lib/kubelet/pods/87dc18db-0ad6-4e73-9c6d-8be180920810/volumes"
Sep 18 17:40:47.766128 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:47.766115    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="986f5132-20cd-4154-b364-b7d352e5324b" path="/var/lib/kubelet/pods/986f5132-20cd-4154-b364-b7d352e5324b/volumes"
Sep 18 17:40:48.053059 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:48.052973    2569 scope.go:117] "RemoveContainer" containerID="604aab1db04837e4b458e53c9fe65f96500da32ea4347f3109dcd88144dd0a84"
Sep 18 17:40:53.215788 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:53.215745    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-66efa6-d6kqw"]
Sep 18 17:40:53.216186 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:53.216121    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73c3ef72-b53a-41af-899b-e3c9b7aed1d6" containerName="payload-processing"
Sep 18 17:40:53.216186 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:53.216134    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="73c3ef72-b53a-41af-899b-e3c9b7aed1d6" containerName="payload-processing"
Sep 18 17:40:53.216186 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:53.216148    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="986f5132-20cd-4154-b364-b7d352e5324b" containerName="payload-pre-processing"
Sep 18 17:40:53.216186 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:53.216156    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="986f5132-20cd-4154-b364-b7d352e5324b" containerName="payload-pre-processing"
Sep 18 17:40:53.216186 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:53.216171    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87dc18db-0ad6-4e73-9c6d-8be180920810" containerName="maas-api"
Sep 18 17:40:53.216186 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:53.216177    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="87dc18db-0ad6-4e73-9c6d-8be180920810" containerName="maas-api"
Sep 18 17:40:53.216376 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:53.216192    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97443689-98b6-47f8-8ad2-7131536c262a" containerName="revoke-keys"
Sep 18 17:40:53.216376 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:53.216197    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="97443689-98b6-47f8-8ad2-7131536c262a" containerName="revoke-keys"
Sep 18 17:40:53.216376 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:53.216271    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="986f5132-20cd-4154-b364-b7d352e5324b" containerName="payload-pre-processing"
Sep 18 17:40:53.216376 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:53.216280    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="87dc18db-0ad6-4e73-9c6d-8be180920810" containerName="maas-api"
Sep 18 17:40:53.216376 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:53.216287    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="97443689-98b6-47f8-8ad2-7131536c262a" containerName="revoke-keys"
Sep 18 17:40:53.216376 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:53.216294    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="73c3ef72-b53a-41af-899b-e3c9b7aed1d6" containerName="payload-processing"
Sep 18 17:40:53.219511 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:53.219492    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-66efa6-d6kqw"
Sep 18 17:40:53.222167 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:53.222139    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qt978\""
Sep 18 17:40:53.222167 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:53.222145    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 17:40:53.227207 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:53.226813    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-66efa6-d6kqw"]
Sep 18 17:40:53.249415 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:53.249381    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd28558-6da2-429b-8b82-0fbb0abad22b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-66efa6-d6kqw\" (UID: \"ddd28558-6da2-429b-8b82-0fbb0abad22b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-66efa6-d6kqw"
Sep 18 17:40:53.350231 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:53.350193    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd28558-6da2-429b-8b82-0fbb0abad22b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-66efa6-d6kqw\" (UID: \"ddd28558-6da2-429b-8b82-0fbb0abad22b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-66efa6-d6kqw"
Sep 18 17:40:53.350843 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:53.350821    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd28558-6da2-429b-8b82-0fbb0abad22b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-66efa6-d6kqw\" (UID: \"ddd28558-6da2-429b-8b82-0fbb0abad22b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-66efa6-d6kqw"
Sep 18 17:40:53.531707 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:53.531667    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-66efa6-d6kqw"
Sep 18 17:40:53.799686 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:53.799659    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-66efa6-d6kqw"]
Sep 18 17:40:53.802380 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:40:53.802349    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd28558_6da2_429b_8b82_0fbb0abad22b.slice/crio-23b21d51e2e2b48daeddbc049860b71d2d59ce23c290b26561c47638739d9fa3 WatchSource:0}: Error finding container 23b21d51e2e2b48daeddbc049860b71d2d59ce23c290b26561c47638739d9fa3: Status 404 returned error can't find the container with id 23b21d51e2e2b48daeddbc049860b71d2d59ce23c290b26561c47638739d9fa3
Sep 18 17:40:54.746390 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:54.746354    2569 generic.go:356] "Generic (PLEG): container finished" podID="ddd28558-6da2-429b-8b82-0fbb0abad22b" containerID="e5619660975970fb29420ecf651f31b5bd7eb80472200716c88dd6cc65464833" exitCode=0
Sep 18 17:40:54.746847 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:54.746426    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-66efa6-d6kqw" event={"ID":"ddd28558-6da2-429b-8b82-0fbb0abad22b","Type":"ContainerDied","Data":"e5619660975970fb29420ecf651f31b5bd7eb80472200716c88dd6cc65464833"}
Sep 18 17:40:54.746847 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:54.746484    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-66efa6-d6kqw" event={"ID":"ddd28558-6da2-429b-8b82-0fbb0abad22b","Type":"ContainerStarted","Data":"23b21d51e2e2b48daeddbc049860b71d2d59ce23c290b26561c47638739d9fa3"}
Sep 18 17:40:55.884133 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:55.884108    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-66efa6-d6kqw"
Sep 18 17:40:55.972642 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:55.972603    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd28558-6da2-429b-8b82-0fbb0abad22b-maas-api-service-ca\") pod \"ddd28558-6da2-429b-8b82-0fbb0abad22b\" (UID: \"ddd28558-6da2-429b-8b82-0fbb0abad22b\") "
Sep 18 17:40:55.972966 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:55.972944    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddd28558-6da2-429b-8b82-0fbb0abad22b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ddd28558-6da2-429b-8b82-0fbb0abad22b" (UID: "ddd28558-6da2-429b-8b82-0fbb0abad22b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:40:56.073514 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:56.073438    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd28558-6da2-429b-8b82-0fbb0abad22b-maas-api-service-ca\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:40:56.759829 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:56.759801    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-66efa6-d6kqw"
Sep 18 17:40:56.759995 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:56.759847    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-66efa6-d6kqw" event={"ID":"ddd28558-6da2-429b-8b82-0fbb0abad22b","Type":"ContainerDied","Data":"23b21d51e2e2b48daeddbc049860b71d2d59ce23c290b26561c47638739d9fa3"}
Sep 18 17:40:56.759995 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:40:56.759902    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23b21d51e2e2b48daeddbc049860b71d2d59ce23c290b26561c47638739d9fa3"
Sep 18 17:41:03.833000 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:03.832965    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-66efa6-7b84ffc8c9-2dzhp"]
Sep 18 17:41:03.833585 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:03.833218    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66efa6-7b84ffc8c9-2dzhp" podUID="f4853ad5-9ed6-4d38-9f94-4b0d085c93f2" containerName="payload-processing" containerID="cri-o://93d6423bc2f458a78481fc54ea604e505f95f791db71766faf9a309ef2decaf8" gracePeriod=30
Sep 18 17:41:03.851120 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:03.851094    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66efa6-587f5466b6-vdf5c"]
Sep 18 17:41:03.851282 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:03.851262    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66efa6-587f5466b6-vdf5c" podUID="8194a9e1-5e29-4e70-8899-8631ffb2c2d6" containerName="payload-pre-processing" containerID="cri-o://0bf1cba711c885b74bb609a870b06ad913b9d69ef7a5a2a1ac677b9ca563acea" gracePeriod=30
Sep 18 17:41:04.095831 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:04.095773    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66efa6-7b84ffc8c9-2dzhp"
Sep 18 17:41:04.108625 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:04.108596    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-66efa6-d6kqw"]
Sep 18 17:41:04.114723 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:04.114696    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-66efa6-d6kqw"]
Sep 18 17:41:04.126532 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:04.126507    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66efa6-587f5466b6-vdf5c"
Sep 18 17:41:04.152990 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:04.152966    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4853ad5-9ed6-4d38-9f94-4b0d085c93f2-plugins-config-volume\") pod \"f4853ad5-9ed6-4d38-9f94-4b0d085c93f2\" (UID: \"f4853ad5-9ed6-4d38-9f94-4b0d085c93f2\") "
Sep 18 17:41:04.153133 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:04.153066    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xl4ck\" (UniqueName: \"kubernetes.io/projected/f4853ad5-9ed6-4d38-9f94-4b0d085c93f2-kube-api-access-xl4ck\") pod \"f4853ad5-9ed6-4d38-9f94-4b0d085c93f2\" (UID: \"f4853ad5-9ed6-4d38-9f94-4b0d085c93f2\") "
Sep 18 17:41:04.153369 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:04.153341    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4853ad5-9ed6-4d38-9f94-4b0d085c93f2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f4853ad5-9ed6-4d38-9f94-4b0d085c93f2" (UID: "f4853ad5-9ed6-4d38-9f94-4b0d085c93f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:41:04.155037 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:04.155012    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4853ad5-9ed6-4d38-9f94-4b0d085c93f2-kube-api-access-xl4ck" (OuterVolumeSpecName: "kube-api-access-xl4ck") pod "f4853ad5-9ed6-4d38-9f94-4b0d085c93f2" (UID: "f4853ad5-9ed6-4d38-9f94-4b0d085c93f2"). InnerVolumeSpecName "kube-api-access-xl4ck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:41:04.254473 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:04.254439    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8194a9e1-5e29-4e70-8899-8631ffb2c2d6-plugins-config-volume\") pod \"8194a9e1-5e29-4e70-8899-8631ffb2c2d6\" (UID: \"8194a9e1-5e29-4e70-8899-8631ffb2c2d6\") "
Sep 18 17:41:04.254632 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:04.254592    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvh7v\" (UniqueName: \"kubernetes.io/projected/8194a9e1-5e29-4e70-8899-8631ffb2c2d6-kube-api-access-nvh7v\") pod \"8194a9e1-5e29-4e70-8899-8631ffb2c2d6\" (UID: \"8194a9e1-5e29-4e70-8899-8631ffb2c2d6\") "
Sep 18 17:41:04.254912 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:04.254887    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8194a9e1-5e29-4e70-8899-8631ffb2c2d6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8194a9e1-5e29-4e70-8899-8631ffb2c2d6" (UID: "8194a9e1-5e29-4e70-8899-8631ffb2c2d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:41:04.254966 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:04.254897    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4853ad5-9ed6-4d38-9f94-4b0d085c93f2-plugins-config-volume\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:41:04.254966 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:04.254935    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl4ck\" (UniqueName: \"kubernetes.io/projected/f4853ad5-9ed6-4d38-9f94-4b0d085c93f2-kube-api-access-xl4ck\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:41:04.256549 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:04.256526    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8194a9e1-5e29-4e70-8899-8631ffb2c2d6-kube-api-access-nvh7v" (OuterVolumeSpecName: "kube-api-access-nvh7v") pod "8194a9e1-5e29-4e70-8899-8631ffb2c2d6" (UID: "8194a9e1-5e29-4e70-8899-8631ffb2c2d6"). InnerVolumeSpecName "kube-api-access-nvh7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:41:04.355847 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:04.355757    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8194a9e1-5e29-4e70-8899-8631ffb2c2d6-plugins-config-volume\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:41:04.355847 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:04.355790    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvh7v\" (UniqueName: \"kubernetes.io/projected/8194a9e1-5e29-4e70-8899-8631ffb2c2d6-kube-api-access-nvh7v\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:41:04.800876 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:04.800824    2569 generic.go:356] "Generic (PLEG): container finished" podID="f4853ad5-9ed6-4d38-9f94-4b0d085c93f2" containerID="93d6423bc2f458a78481fc54ea604e505f95f791db71766faf9a309ef2decaf8" exitCode=0
Sep 18 17:41:04.801029 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:04.800890    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66efa6-7b84ffc8c9-2dzhp"
Sep 18 17:41:04.801029 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:04.800911    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66efa6-7b84ffc8c9-2dzhp" event={"ID":"f4853ad5-9ed6-4d38-9f94-4b0d085c93f2","Type":"ContainerDied","Data":"93d6423bc2f458a78481fc54ea604e505f95f791db71766faf9a309ef2decaf8"}
Sep 18 17:41:04.801029 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:04.800970    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66efa6-7b84ffc8c9-2dzhp" event={"ID":"f4853ad5-9ed6-4d38-9f94-4b0d085c93f2","Type":"ContainerDied","Data":"db92329bd3663019b1ce93415cc8528a1e354f93ff12cfec0e02023dfb11e7fb"}
Sep 18 17:41:04.801029 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:04.801002    2569 scope.go:117] "RemoveContainer" containerID="93d6423bc2f458a78481fc54ea604e505f95f791db71766faf9a309ef2decaf8"
Sep 18 17:41:04.802325 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:04.802300    2569 generic.go:356] "Generic (PLEG): container finished" podID="8194a9e1-5e29-4e70-8899-8631ffb2c2d6" containerID="0bf1cba711c885b74bb609a870b06ad913b9d69ef7a5a2a1ac677b9ca563acea" exitCode=0
Sep 18 17:41:04.802447 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:04.802333    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66efa6-587f5466b6-vdf5c" event={"ID":"8194a9e1-5e29-4e70-8899-8631ffb2c2d6","Type":"ContainerDied","Data":"0bf1cba711c885b74bb609a870b06ad913b9d69ef7a5a2a1ac677b9ca563acea"}
Sep 18 17:41:04.802447 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:04.802362    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66efa6-587f5466b6-vdf5c"
Sep 18 17:41:04.802447 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:04.802394    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66efa6-587f5466b6-vdf5c" event={"ID":"8194a9e1-5e29-4e70-8899-8631ffb2c2d6","Type":"ContainerDied","Data":"ad8fe82df484ec3fafe3d270773b83281db00394e7cd92167233bd73c0faeebe"}
Sep 18 17:41:04.809736 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:04.809717    2569 scope.go:117] "RemoveContainer" containerID="93d6423bc2f458a78481fc54ea604e505f95f791db71766faf9a309ef2decaf8"
Sep 18 17:41:04.810051 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:41:04.810031    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93d6423bc2f458a78481fc54ea604e505f95f791db71766faf9a309ef2decaf8\": container with ID starting with 93d6423bc2f458a78481fc54ea604e505f95f791db71766faf9a309ef2decaf8 not found: ID does not exist" containerID="93d6423bc2f458a78481fc54ea604e505f95f791db71766faf9a309ef2decaf8"
Sep 18 17:41:04.810109 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:04.810060    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93d6423bc2f458a78481fc54ea604e505f95f791db71766faf9a309ef2decaf8"} err="failed to get container status \"93d6423bc2f458a78481fc54ea604e505f95f791db71766faf9a309ef2decaf8\": rpc error: code = NotFound desc = could not find container \"93d6423bc2f458a78481fc54ea604e505f95f791db71766faf9a309ef2decaf8\": container with ID starting with 93d6423bc2f458a78481fc54ea604e505f95f791db71766faf9a309ef2decaf8 not found: ID does not exist"
Sep 18 17:41:04.810109 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:04.810077    2569 scope.go:117] "RemoveContainer" containerID="0bf1cba711c885b74bb609a870b06ad913b9d69ef7a5a2a1ac677b9ca563acea"
Sep 18 17:41:04.818523 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:04.818506    2569 scope.go:117] "RemoveContainer" containerID="0bf1cba711c885b74bb609a870b06ad913b9d69ef7a5a2a1ac677b9ca563acea"
Sep 18 17:41:04.818798 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:41:04.818778    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bf1cba711c885b74bb609a870b06ad913b9d69ef7a5a2a1ac677b9ca563acea\": container with ID starting with 0bf1cba711c885b74bb609a870b06ad913b9d69ef7a5a2a1ac677b9ca563acea not found: ID does not exist" containerID="0bf1cba711c885b74bb609a870b06ad913b9d69ef7a5a2a1ac677b9ca563acea"
Sep 18 17:41:04.818855 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:04.818811    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bf1cba711c885b74bb609a870b06ad913b9d69ef7a5a2a1ac677b9ca563acea"} err="failed to get container status \"0bf1cba711c885b74bb609a870b06ad913b9d69ef7a5a2a1ac677b9ca563acea\": rpc error: code = NotFound desc = could not find container \"0bf1cba711c885b74bb609a870b06ad913b9d69ef7a5a2a1ac677b9ca563acea\": container with ID starting with 0bf1cba711c885b74bb609a870b06ad913b9d69ef7a5a2a1ac677b9ca563acea not found: ID does not exist"
Sep 18 17:41:04.870616 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:04.870586    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-66efa6-7b84ffc8c9-2dzhp"]
Sep 18 17:41:04.878557 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:04.878525    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-66efa6-7b84ffc8c9-2dzhp"]
Sep 18 17:41:04.901622 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:04.901588    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66efa6-587f5466b6-vdf5c"]
Sep 18 17:41:04.914111 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:04.914082    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66efa6-587f5466b6-vdf5c"]
Sep 18 17:41:05.764342 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:05.764311    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8194a9e1-5e29-4e70-8899-8631ffb2c2d6" path="/var/lib/kubelet/pods/8194a9e1-5e29-4e70-8899-8631ffb2c2d6/volumes"
Sep 18 17:41:05.764682 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:05.764670    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddd28558-6da2-429b-8b82-0fbb0abad22b" path="/var/lib/kubelet/pods/ddd28558-6da2-429b-8b82-0fbb0abad22b/volumes"
Sep 18 17:41:05.765013 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:05.765000    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4853ad5-9ed6-4d38-9f94-4b0d085c93f2" path="/var/lib/kubelet/pods/f4853ad5-9ed6-4d38-9f94-4b0d085c93f2/volumes"
Sep 18 17:41:25.082663 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:25.082624    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5b6d69-ldv4w"]
Sep 18 17:41:25.083159 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:25.083013    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4853ad5-9ed6-4d38-9f94-4b0d085c93f2" containerName="payload-processing"
Sep 18 17:41:25.083159 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:25.083026    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4853ad5-9ed6-4d38-9f94-4b0d085c93f2" containerName="payload-processing"
Sep 18 17:41:25.083159 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:25.083039    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8194a9e1-5e29-4e70-8899-8631ffb2c2d6" containerName="payload-pre-processing"
Sep 18 17:41:25.083159 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:25.083046    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8194a9e1-5e29-4e70-8899-8631ffb2c2d6" containerName="payload-pre-processing"
Sep 18 17:41:25.083159 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:25.083062    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddd28558-6da2-429b-8b82-0fbb0abad22b" containerName="revoke-keys"
Sep 18 17:41:25.083159 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:25.083068    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddd28558-6da2-429b-8b82-0fbb0abad22b" containerName="revoke-keys"
Sep 18 17:41:25.083159 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:25.083133    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4853ad5-9ed6-4d38-9f94-4b0d085c93f2" containerName="payload-processing"
Sep 18 17:41:25.083159 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:25.083144    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8194a9e1-5e29-4e70-8899-8631ffb2c2d6" containerName="payload-pre-processing"
Sep 18 17:41:25.083159 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:25.083149    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddd28558-6da2-429b-8b82-0fbb0abad22b" containerName="revoke-keys"
Sep 18 17:41:25.086280 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:25.086265    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5b6d69-ldv4w"
Sep 18 17:41:25.088986 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:25.088958    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 17:41:25.089101 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:25.088997    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qt978\""
Sep 18 17:41:25.094094 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:25.094068    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5b6d69-ldv4w"]
Sep 18 17:41:25.252094 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:25.252055    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13ffb22a-ad01-42d5-a393-911bbadb8f26-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-5b6d69-ldv4w\" (UID: \"13ffb22a-ad01-42d5-a393-911bbadb8f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5b6d69-ldv4w"
Sep 18 17:41:25.353190 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:25.353098    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13ffb22a-ad01-42d5-a393-911bbadb8f26-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-5b6d69-ldv4w\" (UID: \"13ffb22a-ad01-42d5-a393-911bbadb8f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5b6d69-ldv4w"
Sep 18 17:41:25.353720 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:25.353699    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13ffb22a-ad01-42d5-a393-911bbadb8f26-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-5b6d69-ldv4w\" (UID: \"13ffb22a-ad01-42d5-a393-911bbadb8f26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5b6d69-ldv4w"
Sep 18 17:41:25.398102 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:25.398071    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5b6d69-ldv4w"
Sep 18 17:41:25.669529 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:25.669504    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5b6d69-ldv4w"]
Sep 18 17:41:25.674568 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:41:25.674539    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13ffb22a_ad01_42d5_a393_911bbadb8f26.slice/crio-649a04ce09640fca161edd25d3dc083c7b67fa33022504c316d98687c3999a7e WatchSource:0}: Error finding container 649a04ce09640fca161edd25d3dc083c7b67fa33022504c316d98687c3999a7e: Status 404 returned error can't find the container with id 649a04ce09640fca161edd25d3dc083c7b67fa33022504c316d98687c3999a7e
Sep 18 17:41:25.911032 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:25.910998    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5b6d69-ldv4w" event={"ID":"13ffb22a-ad01-42d5-a393-911bbadb8f26","Type":"ContainerStarted","Data":"649a04ce09640fca161edd25d3dc083c7b67fa33022504c316d98687c3999a7e"}
Sep 18 17:41:26.917155 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:26.917121    2569 generic.go:356] "Generic (PLEG): container finished" podID="13ffb22a-ad01-42d5-a393-911bbadb8f26" containerID="132fc4722ab5beca5aac7c4ffd531d61b1d3edbf00a07ff977c4e31d110dbff6" exitCode=0
Sep 18 17:41:26.917565 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:26.917160    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5b6d69-ldv4w" event={"ID":"13ffb22a-ad01-42d5-a393-911bbadb8f26","Type":"ContainerDied","Data":"132fc4722ab5beca5aac7c4ffd531d61b1d3edbf00a07ff977c4e31d110dbff6"}
Sep 18 17:41:28.046314 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:28.046289    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5b6d69-ldv4w"
Sep 18 17:41:28.079450 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:28.079422    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13ffb22a-ad01-42d5-a393-911bbadb8f26-maas-api-service-ca\") pod \"13ffb22a-ad01-42d5-a393-911bbadb8f26\" (UID: \"13ffb22a-ad01-42d5-a393-911bbadb8f26\") "
Sep 18 17:41:28.079807 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:28.079787    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13ffb22a-ad01-42d5-a393-911bbadb8f26-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "13ffb22a-ad01-42d5-a393-911bbadb8f26" (UID: "13ffb22a-ad01-42d5-a393-911bbadb8f26"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:41:28.180374 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:28.180280    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13ffb22a-ad01-42d5-a393-911bbadb8f26-maas-api-service-ca\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:41:28.928580 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:28.928549    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5b6d69-ldv4w"
Sep 18 17:41:28.928766 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:28.928588    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5b6d69-ldv4w" event={"ID":"13ffb22a-ad01-42d5-a393-911bbadb8f26","Type":"ContainerDied","Data":"649a04ce09640fca161edd25d3dc083c7b67fa33022504c316d98687c3999a7e"}
Sep 18 17:41:28.928766 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:28.928629    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="649a04ce09640fca161edd25d3dc083c7b67fa33022504c316d98687c3999a7e"
Sep 18 17:41:29.959563 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:29.959527    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9"]
Sep 18 17:41:29.973019 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:29.972992    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd"]
Sep 18 17:41:40.882189 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:40.882110    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-5b6d69-29829210-ln4kp"]
Sep 18 17:41:40.888735 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:40.888707    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-5b6d69-29829210-ln4kp"]
Sep 18 17:41:41.094510 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:41.094473    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5b6d69-ldv4w"]
Sep 18 17:41:41.097120 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:41.097089    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5b6d69-ldv4w"]
Sep 18 17:41:41.764583 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:41.764548    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13ffb22a-ad01-42d5-a393-911bbadb8f26" path="/var/lib/kubelet/pods/13ffb22a-ad01-42d5-a393-911bbadb8f26/volumes"
Sep 18 17:41:41.764844 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:41.764831    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8b79e5e-54f1-4eb5-88ca-3c4f4c8885ae" path="/var/lib/kubelet/pods/e8b79e5e-54f1-4eb5-88ca-3c4f4c8885ae/volumes"
Sep 18 17:41:44.125743 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:44.125704    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7726-9734fc"]
Sep 18 17:41:44.126142 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:44.126034    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13ffb22a-ad01-42d5-a393-911bbadb8f26" containerName="revoke-keys"
Sep 18 17:41:44.126142 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:44.126047    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="13ffb22a-ad01-42d5-a393-911bbadb8f26" containerName="revoke-keys"
Sep 18 17:41:44.126142 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:44.126113    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="13ffb22a-ad01-42d5-a393-911bbadb8f26" containerName="revoke-keys"
Sep 18 17:41:44.129198 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:44.129180    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7726-9734fc"
Sep 18 17:41:44.143217 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:44.143190    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mjflw\""
Sep 18 17:41:44.146527 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:44.146498    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7726-9734fc"]
Sep 18 17:41:44.225032 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:44.224996    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbvkw\" (UniqueName: \"kubernetes.io/projected/916859ed-3d77-451e-be8e-fb78e88983b7-kube-api-access-rbvkw\") pod \"test-curl-7726-9734fc\" (UID: \"916859ed-3d77-451e-be8e-fb78e88983b7\") " pod="opendatahub/test-curl-7726-9734fc"
Sep 18 17:41:44.325992 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:44.325956    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbvkw\" (UniqueName: \"kubernetes.io/projected/916859ed-3d77-451e-be8e-fb78e88983b7-kube-api-access-rbvkw\") pod \"test-curl-7726-9734fc\" (UID: \"916859ed-3d77-451e-be8e-fb78e88983b7\") " pod="opendatahub/test-curl-7726-9734fc"
Sep 18 17:41:44.340514 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:44.340491    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbvkw\" (UniqueName: \"kubernetes.io/projected/916859ed-3d77-451e-be8e-fb78e88983b7-kube-api-access-rbvkw\") pod \"test-curl-7726-9734fc\" (UID: \"916859ed-3d77-451e-be8e-fb78e88983b7\") " pod="opendatahub/test-curl-7726-9734fc"
Sep 18 17:41:44.438362 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:44.438279    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7726-9734fc"
Sep 18 17:41:44.708253 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:44.708156    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7726-9734fc"]
Sep 18 17:41:44.712461 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:41:44.712429    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod916859ed_3d77_451e_be8e_fb78e88983b7.slice/crio-5933e46f27592073876be970daa01683e6ad19a16cffbd32174c814b618086bd WatchSource:0}: Error finding container 5933e46f27592073876be970daa01683e6ad19a16cffbd32174c814b618086bd: Status 404 returned error can't find the container with id 5933e46f27592073876be970daa01683e6ad19a16cffbd32174c814b618086bd
Sep 18 17:41:44.978622 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:44.978540    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9" podUID="718a9c7a-e629-4cb5-aa54-520a737523d8" containerName="main" containerID="cri-o://dddbc5af93ac946b0a79c9abf85e5de9fac5a4030c8d512f602ba833f3e4f744" gracePeriod=45
Sep 18 17:41:44.993233 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:44.993199    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd" podUID="93943d04-0dd3-4f9d-828e-f41af19c5d86" containerName="main" containerID="cri-o://c2e648bfa94039f48fa7274dfcdab9e21a5c8075b534002ed9211d4389177982" gracePeriod=45
Sep 18 17:41:45.012041 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.012012    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7726-9734fc" event={"ID":"916859ed-3d77-451e-be8e-fb78e88983b7","Type":"ContainerStarted","Data":"f0f2d407b2b087b0e4b699f567a3e42135123df92deed61b5ca360cf9d651c45"}
Sep 18 17:41:45.012132 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.012047    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7726-9734fc" event={"ID":"916859ed-3d77-451e-be8e-fb78e88983b7","Type":"ContainerStarted","Data":"5933e46f27592073876be970daa01683e6ad19a16cffbd32174c814b618086bd"}
Sep 18 17:41:45.030790 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.030749    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7726-9734fc" podStartSLOduration=1.030733678 podStartE2EDuration="1.030733678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:41:45.028130279 +0000 UTC m=+1617.722057808" watchObservedRunningTime="2026-09-18 17:41:45.030733678 +0000 UTC m=+1617.724661384"
Sep 18 17:41:45.264134 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.264086    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9"
Sep 18 17:41:45.278131 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.278105    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd"
Sep 18 17:41:45.336726 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.336693    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93943d04-0dd3-4f9d-828e-f41af19c5d86-home\") pod \"93943d04-0dd3-4f9d-828e-f41af19c5d86\" (UID: \"93943d04-0dd3-4f9d-828e-f41af19c5d86\") "
Sep 18 17:41:45.336908 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.336747    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/718a9c7a-e629-4cb5-aa54-520a737523d8-tmp-dir\") pod \"718a9c7a-e629-4cb5-aa54-520a737523d8\" (UID: \"718a9c7a-e629-4cb5-aa54-520a737523d8\") "
Sep 18 17:41:45.336908 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.336766    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/718a9c7a-e629-4cb5-aa54-520a737523d8-tls-certs\") pod \"718a9c7a-e629-4cb5-aa54-520a737523d8\" (UID: \"718a9c7a-e629-4cb5-aa54-520a737523d8\") "
Sep 18 17:41:45.336908 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.336811    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/718a9c7a-e629-4cb5-aa54-520a737523d8-home\") pod \"718a9c7a-e629-4cb5-aa54-520a737523d8\" (UID: \"718a9c7a-e629-4cb5-aa54-520a737523d8\") "
Sep 18 17:41:45.336908 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.336835    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/718a9c7a-e629-4cb5-aa54-520a737523d8-dshm\") pod \"718a9c7a-e629-4cb5-aa54-520a737523d8\" (UID: \"718a9c7a-e629-4cb5-aa54-520a737523d8\") "
Sep 18 17:41:45.336908 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.336892    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4fht\" (UniqueName: \"kubernetes.io/projected/93943d04-0dd3-4f9d-828e-f41af19c5d86-kube-api-access-j4fht\") pod \"93943d04-0dd3-4f9d-828e-f41af19c5d86\" (UID: \"93943d04-0dd3-4f9d-828e-f41af19c5d86\") "
Sep 18 17:41:45.337202 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.336921    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/718a9c7a-e629-4cb5-aa54-520a737523d8-model-cache\") pod \"718a9c7a-e629-4cb5-aa54-520a737523d8\" (UID: \"718a9c7a-e629-4cb5-aa54-520a737523d8\") "
Sep 18 17:41:45.337202 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.336953    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93943d04-0dd3-4f9d-828e-f41af19c5d86-tmp-dir\") pod \"93943d04-0dd3-4f9d-828e-f41af19c5d86\" (UID: \"93943d04-0dd3-4f9d-828e-f41af19c5d86\") "
Sep 18 17:41:45.337202 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.336989    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hggj\" (UniqueName: \"kubernetes.io/projected/718a9c7a-e629-4cb5-aa54-520a737523d8-kube-api-access-8hggj\") pod \"718a9c7a-e629-4cb5-aa54-520a737523d8\" (UID: \"718a9c7a-e629-4cb5-aa54-520a737523d8\") "
Sep 18 17:41:45.337202 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.336991    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93943d04-0dd3-4f9d-828e-f41af19c5d86-home" (OuterVolumeSpecName: "home") pod "93943d04-0dd3-4f9d-828e-f41af19c5d86" (UID: "93943d04-0dd3-4f9d-828e-f41af19c5d86"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:41:45.337202 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.337008    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/718a9c7a-e629-4cb5-aa54-520a737523d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "718a9c7a-e629-4cb5-aa54-520a737523d8" (UID: "718a9c7a-e629-4cb5-aa54-520a737523d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:41:45.337202 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.337026    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93943d04-0dd3-4f9d-828e-f41af19c5d86-model-cache\") pod \"93943d04-0dd3-4f9d-828e-f41af19c5d86\" (UID: \"93943d04-0dd3-4f9d-828e-f41af19c5d86\") "
Sep 18 17:41:45.337202 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.337072    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93943d04-0dd3-4f9d-828e-f41af19c5d86-dshm\") pod \"93943d04-0dd3-4f9d-828e-f41af19c5d86\" (UID: \"93943d04-0dd3-4f9d-828e-f41af19c5d86\") "
Sep 18 17:41:45.337202 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.337101    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/718a9c7a-e629-4cb5-aa54-520a737523d8-home" (OuterVolumeSpecName: "home") pod "718a9c7a-e629-4cb5-aa54-520a737523d8" (UID: "718a9c7a-e629-4cb5-aa54-520a737523d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:41:45.337202 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.337126    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93943d04-0dd3-4f9d-828e-f41af19c5d86-tls-certs\") pod \"93943d04-0dd3-4f9d-828e-f41af19c5d86\" (UID: \"93943d04-0dd3-4f9d-828e-f41af19c5d86\") "
Sep 18 17:41:45.337202 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.337190    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/718a9c7a-e629-4cb5-aa54-520a737523d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "718a9c7a-e629-4cb5-aa54-520a737523d8" (UID: "718a9c7a-e629-4cb5-aa54-520a737523d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:41:45.337743 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.337371    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93943d04-0dd3-4f9d-828e-f41af19c5d86-model-cache" (OuterVolumeSpecName: "model-cache") pod "93943d04-0dd3-4f9d-828e-f41af19c5d86" (UID: "93943d04-0dd3-4f9d-828e-f41af19c5d86"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:41:45.337743 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.337442    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93943d04-0dd3-4f9d-828e-f41af19c5d86-home\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:41:45.337743 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.337462    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/718a9c7a-e629-4cb5-aa54-520a737523d8-tmp-dir\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:41:45.337743 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.337479    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/718a9c7a-e629-4cb5-aa54-520a737523d8-home\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:41:45.337743 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.337490    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/718a9c7a-e629-4cb5-aa54-520a737523d8-model-cache\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:41:45.337743 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.337554    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93943d04-0dd3-4f9d-828e-f41af19c5d86-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "93943d04-0dd3-4f9d-828e-f41af19c5d86" (UID: "93943d04-0dd3-4f9d-828e-f41af19c5d86"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:41:45.339546 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.339511    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/718a9c7a-e629-4cb5-aa54-520a737523d8-dshm" (OuterVolumeSpecName: "dshm") pod "718a9c7a-e629-4cb5-aa54-520a737523d8" (UID: "718a9c7a-e629-4cb5-aa54-520a737523d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:41:45.339782 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.339753    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93943d04-0dd3-4f9d-828e-f41af19c5d86-kube-api-access-j4fht" (OuterVolumeSpecName: "kube-api-access-j4fht") pod "93943d04-0dd3-4f9d-828e-f41af19c5d86" (UID: "93943d04-0dd3-4f9d-828e-f41af19c5d86"). InnerVolumeSpecName "kube-api-access-j4fht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:41:45.339923 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.339900    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/718a9c7a-e629-4cb5-aa54-520a737523d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "718a9c7a-e629-4cb5-aa54-520a737523d8" (UID: "718a9c7a-e629-4cb5-aa54-520a737523d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:41:45.339991 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.339969    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93943d04-0dd3-4f9d-828e-f41af19c5d86-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "93943d04-0dd3-4f9d-828e-f41af19c5d86" (UID: "93943d04-0dd3-4f9d-828e-f41af19c5d86"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:41:45.340163 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.340136    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/718a9c7a-e629-4cb5-aa54-520a737523d8-kube-api-access-8hggj" (OuterVolumeSpecName: "kube-api-access-8hggj") pod "718a9c7a-e629-4cb5-aa54-520a737523d8" (UID: "718a9c7a-e629-4cb5-aa54-520a737523d8"). InnerVolumeSpecName "kube-api-access-8hggj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:41:45.340288 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.340170    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93943d04-0dd3-4f9d-828e-f41af19c5d86-dshm" (OuterVolumeSpecName: "dshm") pod "93943d04-0dd3-4f9d-828e-f41af19c5d86" (UID: "93943d04-0dd3-4f9d-828e-f41af19c5d86"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:41:45.438608 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.438580    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/718a9c7a-e629-4cb5-aa54-520a737523d8-dshm\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:41:45.438608 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.438605    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4fht\" (UniqueName: \"kubernetes.io/projected/93943d04-0dd3-4f9d-828e-f41af19c5d86-kube-api-access-j4fht\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:41:45.438775 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.438617    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93943d04-0dd3-4f9d-828e-f41af19c5d86-tmp-dir\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:41:45.438775 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.438629    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hggj\" (UniqueName: \"kubernetes.io/projected/718a9c7a-e629-4cb5-aa54-520a737523d8-kube-api-access-8hggj\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:41:45.438775 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.438636    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93943d04-0dd3-4f9d-828e-f41af19c5d86-model-cache\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:41:45.438775 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.438644    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93943d04-0dd3-4f9d-828e-f41af19c5d86-dshm\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:41:45.438775 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.438651    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93943d04-0dd3-4f9d-828e-f41af19c5d86-tls-certs\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:41:45.438775 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.438659    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/718a9c7a-e629-4cb5-aa54-520a737523d8-tls-certs\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:41:45.624599 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.624500    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7726-9734fc"]
Sep 18 17:41:45.627014 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:45.626989    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7726-9734fc"]
Sep 18 17:41:46.018194 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:46.018158    2569 generic.go:356] "Generic (PLEG): container finished" podID="93943d04-0dd3-4f9d-828e-f41af19c5d86" containerID="c2e648bfa94039f48fa7274dfcdab9e21a5c8075b534002ed9211d4389177982" exitCode=0
Sep 18 17:41:46.018362 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:46.018211    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd"
Sep 18 17:41:46.018362 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:46.018241    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd" event={"ID":"93943d04-0dd3-4f9d-828e-f41af19c5d86","Type":"ContainerDied","Data":"c2e648bfa94039f48fa7274dfcdab9e21a5c8075b534002ed9211d4389177982"}
Sep 18 17:41:46.018362 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:46.018282    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd" event={"ID":"93943d04-0dd3-4f9d-828e-f41af19c5d86","Type":"ContainerDied","Data":"9d49094189826f1095ea3f5cf54ff8e3878ac5de61655b0e66cba93597f0dee6"}
Sep 18 17:41:46.018362 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:46.018303    2569 scope.go:117] "RemoveContainer" containerID="c2e648bfa94039f48fa7274dfcdab9e21a5c8075b534002ed9211d4389177982"
Sep 18 17:41:46.019785 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:46.019762    2569 generic.go:356] "Generic (PLEG): container finished" podID="718a9c7a-e629-4cb5-aa54-520a737523d8" containerID="dddbc5af93ac946b0a79c9abf85e5de9fac5a4030c8d512f602ba833f3e4f744" exitCode=0
Sep 18 17:41:46.019987 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:46.019822    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9"
Sep 18 17:41:46.019987 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:46.019849    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9" event={"ID":"718a9c7a-e629-4cb5-aa54-520a737523d8","Type":"ContainerDied","Data":"dddbc5af93ac946b0a79c9abf85e5de9fac5a4030c8d512f602ba833f3e4f744"}
Sep 18 17:41:46.019987 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:46.019904    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9" event={"ID":"718a9c7a-e629-4cb5-aa54-520a737523d8","Type":"ContainerDied","Data":"b706cb3a7ed4980302fa467bd09b07bc1702892398ee5058d978764c63d0990f"}
Sep 18 17:41:46.019987 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:41:46.019962    2569 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7726-9734fc" container="test-curl-7726-9734fc"
Sep 18 17:41:46.026913 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:46.026895    2569 scope.go:117] "RemoveContainer" containerID="c2e648bfa94039f48fa7274dfcdab9e21a5c8075b534002ed9211d4389177982"
Sep 18 17:41:46.027166 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:41:46.027147    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2e648bfa94039f48fa7274dfcdab9e21a5c8075b534002ed9211d4389177982\": container with ID starting with c2e648bfa94039f48fa7274dfcdab9e21a5c8075b534002ed9211d4389177982 not found: ID does not exist" containerID="c2e648bfa94039f48fa7274dfcdab9e21a5c8075b534002ed9211d4389177982"
Sep 18 17:41:46.027243 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:46.027172    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2e648bfa94039f48fa7274dfcdab9e21a5c8075b534002ed9211d4389177982"} err="failed to get container status \"c2e648bfa94039f48fa7274dfcdab9e21a5c8075b534002ed9211d4389177982\": rpc error: code = NotFound desc = could not find container \"c2e648bfa94039f48fa7274dfcdab9e21a5c8075b534002ed9211d4389177982\": container with ID starting with c2e648bfa94039f48fa7274dfcdab9e21a5c8075b534002ed9211d4389177982 not found: ID does not exist"
Sep 18 17:41:46.027243 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:46.027188    2569 scope.go:117] "RemoveContainer" containerID="dddbc5af93ac946b0a79c9abf85e5de9fac5a4030c8d512f602ba833f3e4f744"
Sep 18 17:41:46.034775 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:46.034761    2569 scope.go:117] "RemoveContainer" containerID="dddbc5af93ac946b0a79c9abf85e5de9fac5a4030c8d512f602ba833f3e4f744"
Sep 18 17:41:46.035052 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:41:46.035032    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dddbc5af93ac946b0a79c9abf85e5de9fac5a4030c8d512f602ba833f3e4f744\": container with ID starting with dddbc5af93ac946b0a79c9abf85e5de9fac5a4030c8d512f602ba833f3e4f744 not found: ID does not exist" containerID="dddbc5af93ac946b0a79c9abf85e5de9fac5a4030c8d512f602ba833f3e4f744"
Sep 18 17:41:46.035128 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:46.035057    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dddbc5af93ac946b0a79c9abf85e5de9fac5a4030c8d512f602ba833f3e4f744"} err="failed to get container status \"dddbc5af93ac946b0a79c9abf85e5de9fac5a4030c8d512f602ba833f3e4f744\": rpc error: code = NotFound desc = could not find container \"dddbc5af93ac946b0a79c9abf85e5de9fac5a4030c8d512f602ba833f3e4f744\": container with ID starting with dddbc5af93ac946b0a79c9abf85e5de9fac5a4030c8d512f602ba833f3e4f744 not found: ID does not exist"
Sep 18 17:41:46.043149 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:46.043123    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9"]
Sep 18 17:41:46.046297 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:46.046278    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-5b6d69/premium-w4-5b6d69-kserve-5485464d4-5j8k9"]
Sep 18 17:41:46.058105 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:46.058084    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd"]
Sep 18 17:41:46.065206 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:46.065187    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-5b6d69/simulator-w4-5b6d69-kserve-6dfb56b8cc-dsmgd"]
Sep 18 17:41:46.990411 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:46.990375    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5"]
Sep 18 17:41:47.026830 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:47.026786    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7726-9734fc" podUID="916859ed-3d77-451e-be8e-fb78e88983b7" containerName="test-curl-7726-9734fc" containerID="cri-o://f0f2d407b2b087b0e4b699f567a3e42135123df92deed61b5ca360cf9d651c45" gracePeriod=2
Sep 18 17:41:47.765146 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:47.765116    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="718a9c7a-e629-4cb5-aa54-520a737523d8" path="/var/lib/kubelet/pods/718a9c7a-e629-4cb5-aa54-520a737523d8/volumes"
Sep 18 17:41:47.765513 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:47.765500    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93943d04-0dd3-4f9d-828e-f41af19c5d86" path="/var/lib/kubelet/pods/93943d04-0dd3-4f9d-828e-f41af19c5d86/volumes"
Sep 18 17:41:47.996839 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:47.996812    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh"]
Sep 18 17:41:48.110844 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:48.110740    2569 scope.go:117] "RemoveContainer" containerID="530e44366f27cefed9791912efd0cda29a829cc4abb66e8825504f02dc7f9394"
Sep 18 17:41:49.203891 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:49.203849    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7726-9734fc"
Sep 18 17:41:49.275569 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:49.275536    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbvkw\" (UniqueName: \"kubernetes.io/projected/916859ed-3d77-451e-be8e-fb78e88983b7-kube-api-access-rbvkw\") pod \"916859ed-3d77-451e-be8e-fb78e88983b7\" (UID: \"916859ed-3d77-451e-be8e-fb78e88983b7\") "
Sep 18 17:41:49.277562 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:49.277530    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916859ed-3d77-451e-be8e-fb78e88983b7-kube-api-access-rbvkw" (OuterVolumeSpecName: "kube-api-access-rbvkw") pod "916859ed-3d77-451e-be8e-fb78e88983b7" (UID: "916859ed-3d77-451e-be8e-fb78e88983b7"). InnerVolumeSpecName "kube-api-access-rbvkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:41:49.377120 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:49.377082    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbvkw\" (UniqueName: \"kubernetes.io/projected/916859ed-3d77-451e-be8e-fb78e88983b7-kube-api-access-rbvkw\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:41:49.764359 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:49.764324    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="916859ed-3d77-451e-be8e-fb78e88983b7" path="/var/lib/kubelet/pods/916859ed-3d77-451e-be8e-fb78e88983b7/volumes"
Sep 18 17:41:50.045014 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:50.044917    2569 generic.go:356] "Generic (PLEG): container finished" podID="916859ed-3d77-451e-be8e-fb78e88983b7" containerID="f0f2d407b2b087b0e4b699f567a3e42135123df92deed61b5ca360cf9d651c45" exitCode=137
Sep 18 17:41:50.045014 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:50.044958    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7726-9734fc"
Sep 18 17:41:50.045014 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:50.044985    2569 scope.go:117] "RemoveContainer" containerID="f0f2d407b2b087b0e4b699f567a3e42135123df92deed61b5ca360cf9d651c45"
Sep 18 17:41:50.053322 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:50.053305    2569 scope.go:117] "RemoveContainer" containerID="f0f2d407b2b087b0e4b699f567a3e42135123df92deed61b5ca360cf9d651c45"
Sep 18 17:41:50.053577 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:41:50.053556    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0f2d407b2b087b0e4b699f567a3e42135123df92deed61b5ca360cf9d651c45\": container with ID starting with f0f2d407b2b087b0e4b699f567a3e42135123df92deed61b5ca360cf9d651c45 not found: ID does not exist" containerID="f0f2d407b2b087b0e4b699f567a3e42135123df92deed61b5ca360cf9d651c45"
Sep 18 17:41:50.053616 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:50.053584    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0f2d407b2b087b0e4b699f567a3e42135123df92deed61b5ca360cf9d651c45"} err="failed to get container status \"f0f2d407b2b087b0e4b699f567a3e42135123df92deed61b5ca360cf9d651c45\": rpc error: code = NotFound desc = could not find container \"f0f2d407b2b087b0e4b699f567a3e42135123df92deed61b5ca360cf9d651c45\": container with ID starting with f0f2d407b2b087b0e4b699f567a3e42135123df92deed61b5ca360cf9d651c45 not found: ID does not exist"
Sep 18 17:41:56.710111 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:56.710072    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr"]
Sep 18 17:41:56.710553 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:56.710381    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="718a9c7a-e629-4cb5-aa54-520a737523d8" containerName="main"
Sep 18 17:41:56.710553 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:56.710393    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="718a9c7a-e629-4cb5-aa54-520a737523d8" containerName="main"
Sep 18 17:41:56.710553 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:56.710411    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93943d04-0dd3-4f9d-828e-f41af19c5d86" containerName="main"
Sep 18 17:41:56.710553 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:56.710416    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="93943d04-0dd3-4f9d-828e-f41af19c5d86" containerName="main"
Sep 18 17:41:56.710553 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:56.710427    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="916859ed-3d77-451e-be8e-fb78e88983b7" containerName="test-curl-7726-9734fc"
Sep 18 17:41:56.710553 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:56.710433    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="916859ed-3d77-451e-be8e-fb78e88983b7" containerName="test-curl-7726-9734fc"
Sep 18 17:41:56.710553 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:56.710497    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="93943d04-0dd3-4f9d-828e-f41af19c5d86" containerName="main"
Sep 18 17:41:56.710553 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:56.710507    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="916859ed-3d77-451e-be8e-fb78e88983b7" containerName="test-curl-7726-9734fc"
Sep 18 17:41:56.710553 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:56.710513    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="718a9c7a-e629-4cb5-aa54-520a737523d8" containerName="main"
Sep 18 17:41:56.714711 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:56.714691    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr"
Sep 18 17:41:56.717536 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:56.717515    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-eeab6e\"/\"sub-test-model-eeab6e-kserve-self-signed-certs\""
Sep 18 17:41:56.723729 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:56.723708    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr"]
Sep 18 17:41:56.849590 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:56.849550    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-tmp-dir\") pod \"sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr\" (UID: \"bf13fa17-9ec1-4e60-ab9f-2f392d7ada44\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr"
Sep 18 17:41:56.849797 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:56.849610    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-model-cache\") pod \"sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr\" (UID: \"bf13fa17-9ec1-4e60-ab9f-2f392d7ada44\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr"
Sep 18 17:41:56.849797 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:56.849646    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-tls-certs\") pod \"sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr\" (UID: \"bf13fa17-9ec1-4e60-ab9f-2f392d7ada44\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr"
Sep 18 17:41:56.849797 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:56.849670    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-home\") pod \"sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr\" (UID: \"bf13fa17-9ec1-4e60-ab9f-2f392d7ada44\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr"
Sep 18 17:41:56.849797 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:56.849771    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-dshm\") pod \"sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr\" (UID: \"bf13fa17-9ec1-4e60-ab9f-2f392d7ada44\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr"
Sep 18 17:41:56.850029 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:56.849814    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbdns\" (UniqueName: \"kubernetes.io/projected/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-kube-api-access-tbdns\") pod \"sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr\" (UID: \"bf13fa17-9ec1-4e60-ab9f-2f392d7ada44\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr"
Sep 18 17:41:56.950510 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:56.950468    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-model-cache\") pod \"sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr\" (UID: \"bf13fa17-9ec1-4e60-ab9f-2f392d7ada44\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr"
Sep 18 17:41:56.950510 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:56.950515    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-tls-certs\") pod \"sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr\" (UID: \"bf13fa17-9ec1-4e60-ab9f-2f392d7ada44\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr"
Sep 18 17:41:56.950720 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:56.950555    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-home\") pod \"sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr\" (UID: \"bf13fa17-9ec1-4e60-ab9f-2f392d7ada44\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr"
Sep 18 17:41:56.950720 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:56.950628    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-dshm\") pod \"sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr\" (UID: \"bf13fa17-9ec1-4e60-ab9f-2f392d7ada44\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr"
Sep 18 17:41:56.950720 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:56.950656    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbdns\" (UniqueName: \"kubernetes.io/projected/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-kube-api-access-tbdns\") pod \"sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr\" (UID: \"bf13fa17-9ec1-4e60-ab9f-2f392d7ada44\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr"
Sep 18 17:41:56.950720 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:56.950710    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-tmp-dir\") pod \"sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr\" (UID: \"bf13fa17-9ec1-4e60-ab9f-2f392d7ada44\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr"
Sep 18 17:41:56.950996 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:56.950975    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-model-cache\") pod \"sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr\" (UID: \"bf13fa17-9ec1-4e60-ab9f-2f392d7ada44\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr"
Sep 18 17:41:56.951050 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:56.951009    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-home\") pod \"sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr\" (UID: \"bf13fa17-9ec1-4e60-ab9f-2f392d7ada44\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr"
Sep 18 17:41:56.951114 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:56.951057    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-tmp-dir\") pod \"sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr\" (UID: \"bf13fa17-9ec1-4e60-ab9f-2f392d7ada44\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr"
Sep 18 17:41:56.952899 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:56.952857    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-dshm\") pod \"sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr\" (UID: \"bf13fa17-9ec1-4e60-ab9f-2f392d7ada44\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr"
Sep 18 17:41:56.953158 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:56.953141    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-tls-certs\") pod \"sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr\" (UID: \"bf13fa17-9ec1-4e60-ab9f-2f392d7ada44\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr"
Sep 18 17:41:56.959148 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:56.959126    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbdns\" (UniqueName: \"kubernetes.io/projected/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-kube-api-access-tbdns\") pod \"sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr\" (UID: \"bf13fa17-9ec1-4e60-ab9f-2f392d7ada44\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr"
Sep 18 17:41:57.023786 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:57.023761    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr"
Sep 18 17:41:57.295933 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:57.295908    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr"]
Sep 18 17:41:57.297731 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:41:57.297704    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf13fa17_9ec1_4e60_ab9f_2f392d7ada44.slice/crio-a47d4e6a07d6876e86143be5aed6cb735f0e0d90f9253c703d4bd58e3ee376c7 WatchSource:0}: Error finding container a47d4e6a07d6876e86143be5aed6cb735f0e0d90f9253c703d4bd58e3ee376c7: Status 404 returned error can't find the container with id a47d4e6a07d6876e86143be5aed6cb735f0e0d90f9253c703d4bd58e3ee376c7
Sep 18 17:41:58.092243 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:58.092203    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr" event={"ID":"bf13fa17-9ec1-4e60-ab9f-2f392d7ada44","Type":"ContainerStarted","Data":"e8a0985fa072c693ee1daab4dd96cdd300709dcff647607e707f9a398d2e48e7"}
Sep 18 17:41:58.092243 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:58.092250    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr" event={"ID":"bf13fa17-9ec1-4e60-ab9f-2f392d7ada44","Type":"ContainerStarted","Data":"a47d4e6a07d6876e86143be5aed6cb735f0e0d90f9253c703d4bd58e3ee376c7"}
Sep 18 17:41:58.110773 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:41:58.110720    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr" podStartSLOduration=2.11070056 podStartE2EDuration="2.11070056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:41:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:41:58.109079762 +0000 UTC m=+1630.803007314" watchObservedRunningTime="2026-09-18 17:41:58.11070056 +0000 UTC m=+1630.804628088"
Sep 18 17:42:02.009190 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:02.009152    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5" podUID="ffa59d70-a4b8-4e96-9f66-9319ac5c48a6" containerName="main" containerID="cri-o://4b1e1b8a28104195d99611b519c8cccc8167c08497fe9dcc1f5c5577b74e55a7" gracePeriod=45
Sep 18 17:42:02.260313 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:02.260250    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5"
Sep 18 17:42:02.294639 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:02.294613    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-model-cache\") pod \"ffa59d70-a4b8-4e96-9f66-9319ac5c48a6\" (UID: \"ffa59d70-a4b8-4e96-9f66-9319ac5c48a6\") "
Sep 18 17:42:02.294811 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:02.294650    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-tmp-dir\") pod \"ffa59d70-a4b8-4e96-9f66-9319ac5c48a6\" (UID: \"ffa59d70-a4b8-4e96-9f66-9319ac5c48a6\") "
Sep 18 17:42:02.294811 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:02.294684    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fvrx\" (UniqueName: \"kubernetes.io/projected/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-kube-api-access-8fvrx\") pod \"ffa59d70-a4b8-4e96-9f66-9319ac5c48a6\" (UID: \"ffa59d70-a4b8-4e96-9f66-9319ac5c48a6\") "
Sep 18 17:42:02.294811 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:02.294713    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-tls-certs\") pod \"ffa59d70-a4b8-4e96-9f66-9319ac5c48a6\" (UID: \"ffa59d70-a4b8-4e96-9f66-9319ac5c48a6\") "
Sep 18 17:42:02.294811 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:02.294792    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-dshm\") pod \"ffa59d70-a4b8-4e96-9f66-9319ac5c48a6\" (UID: \"ffa59d70-a4b8-4e96-9f66-9319ac5c48a6\") "
Sep 18 17:42:02.295052 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:02.294849    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-home\") pod \"ffa59d70-a4b8-4e96-9f66-9319ac5c48a6\" (UID: \"ffa59d70-a4b8-4e96-9f66-9319ac5c48a6\") "
Sep 18 17:42:02.295052 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:02.294947    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-model-cache" (OuterVolumeSpecName: "model-cache") pod "ffa59d70-a4b8-4e96-9f66-9319ac5c48a6" (UID: "ffa59d70-a4b8-4e96-9f66-9319ac5c48a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:42:02.295052 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:02.294995    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ffa59d70-a4b8-4e96-9f66-9319ac5c48a6" (UID: "ffa59d70-a4b8-4e96-9f66-9319ac5c48a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:42:02.295223 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:02.295111    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-model-cache\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:42:02.295223 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:02.295128    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-tmp-dir\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:42:02.295223 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:02.295151    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-home" (OuterVolumeSpecName: "home") pod "ffa59d70-a4b8-4e96-9f66-9319ac5c48a6" (UID: "ffa59d70-a4b8-4e96-9f66-9319ac5c48a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:42:02.297082 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:02.297058    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-dshm" (OuterVolumeSpecName: "dshm") pod "ffa59d70-a4b8-4e96-9f66-9319ac5c48a6" (UID: "ffa59d70-a4b8-4e96-9f66-9319ac5c48a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:42:02.297359 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:02.297335    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-kube-api-access-8fvrx" (OuterVolumeSpecName: "kube-api-access-8fvrx") pod "ffa59d70-a4b8-4e96-9f66-9319ac5c48a6" (UID: "ffa59d70-a4b8-4e96-9f66-9319ac5c48a6"). InnerVolumeSpecName "kube-api-access-8fvrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:42:02.297415 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:02.297334    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ffa59d70-a4b8-4e96-9f66-9319ac5c48a6" (UID: "ffa59d70-a4b8-4e96-9f66-9319ac5c48a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:42:02.395934 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:02.395901    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fvrx\" (UniqueName: \"kubernetes.io/projected/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-kube-api-access-8fvrx\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:42:02.395934 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:02.395929    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-tls-certs\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:42:02.395934 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:02.395940    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-dshm\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:42:02.395934 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:02.395947    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6-home\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:42:03.015841 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:03.015798    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh" podUID="76083dd6-57b2-4c34-96d5-17b73d8c8654" containerName="main" containerID="cri-o://ea94513aa202645cbe777e877e829f6abb7d219da7e475ae68f30b958213a726" gracePeriod=45
Sep 18 17:42:03.126649 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:03.126593    2569 generic.go:356] "Generic (PLEG): container finished" podID="ffa59d70-a4b8-4e96-9f66-9319ac5c48a6" containerID="4b1e1b8a28104195d99611b519c8cccc8167c08497fe9dcc1f5c5577b74e55a7" exitCode=0
Sep 18 17:42:03.126817 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:03.126681    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5"
Sep 18 17:42:03.126817 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:03.126693    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5" event={"ID":"ffa59d70-a4b8-4e96-9f66-9319ac5c48a6","Type":"ContainerDied","Data":"4b1e1b8a28104195d99611b519c8cccc8167c08497fe9dcc1f5c5577b74e55a7"}
Sep 18 17:42:03.126817 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:03.126782    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5" event={"ID":"ffa59d70-a4b8-4e96-9f66-9319ac5c48a6","Type":"ContainerDied","Data":"5b3f82ef6420559429491a42990406441fed8be22dc5a73cc78bf87263a4be78"}
Sep 18 17:42:03.126967 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:03.126823    2569 scope.go:117] "RemoveContainer" containerID="4b1e1b8a28104195d99611b519c8cccc8167c08497fe9dcc1f5c5577b74e55a7"
Sep 18 17:42:03.138107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:03.138075    2569 scope.go:117] "RemoveContainer" containerID="4b1e1b8a28104195d99611b519c8cccc8167c08497fe9dcc1f5c5577b74e55a7"
Sep 18 17:42:03.138414 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:42:03.138391    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b1e1b8a28104195d99611b519c8cccc8167c08497fe9dcc1f5c5577b74e55a7\": container with ID starting with 4b1e1b8a28104195d99611b519c8cccc8167c08497fe9dcc1f5c5577b74e55a7 not found: ID does not exist" containerID="4b1e1b8a28104195d99611b519c8cccc8167c08497fe9dcc1f5c5577b74e55a7"
Sep 18 17:42:03.138511 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:03.138422    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b1e1b8a28104195d99611b519c8cccc8167c08497fe9dcc1f5c5577b74e55a7"} err="failed to get container status \"4b1e1b8a28104195d99611b519c8cccc8167c08497fe9dcc1f5c5577b74e55a7\": rpc error: code = NotFound desc = could not find container \"4b1e1b8a28104195d99611b519c8cccc8167c08497fe9dcc1f5c5577b74e55a7\": container with ID starting with 4b1e1b8a28104195d99611b519c8cccc8167c08497fe9dcc1f5c5577b74e55a7 not found: ID does not exist"
Sep 18 17:42:03.151310 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:03.151274    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5"]
Sep 18 17:42:03.158514 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:03.158479    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-eeab6e/auth-test-model-eeab6e-kserve-594dd8fff5-fq2p5"]
Sep 18 17:42:03.283325 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:03.283302    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh"
Sep 18 17:42:03.303005 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:03.302976    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76083dd6-57b2-4c34-96d5-17b73d8c8654-tls-certs\") pod \"76083dd6-57b2-4c34-96d5-17b73d8c8654\" (UID: \"76083dd6-57b2-4c34-96d5-17b73d8c8654\") "
Sep 18 17:42:03.303165 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:03.303021    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76083dd6-57b2-4c34-96d5-17b73d8c8654-home\") pod \"76083dd6-57b2-4c34-96d5-17b73d8c8654\" (UID: \"76083dd6-57b2-4c34-96d5-17b73d8c8654\") "
Sep 18 17:42:03.303165 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:03.303070    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76083dd6-57b2-4c34-96d5-17b73d8c8654-dshm\") pod \"76083dd6-57b2-4c34-96d5-17b73d8c8654\" (UID: \"76083dd6-57b2-4c34-96d5-17b73d8c8654\") "
Sep 18 17:42:03.303165 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:03.303095    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76083dd6-57b2-4c34-96d5-17b73d8c8654-tmp-dir\") pod \"76083dd6-57b2-4c34-96d5-17b73d8c8654\" (UID: \"76083dd6-57b2-4c34-96d5-17b73d8c8654\") "
Sep 18 17:42:03.303165 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:03.303142    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7q6h\" (UniqueName: \"kubernetes.io/projected/76083dd6-57b2-4c34-96d5-17b73d8c8654-kube-api-access-f7q6h\") pod \"76083dd6-57b2-4c34-96d5-17b73d8c8654\" (UID: \"76083dd6-57b2-4c34-96d5-17b73d8c8654\") "
Sep 18 17:42:03.303405 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:03.303201    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76083dd6-57b2-4c34-96d5-17b73d8c8654-model-cache\") pod \"76083dd6-57b2-4c34-96d5-17b73d8c8654\" (UID: \"76083dd6-57b2-4c34-96d5-17b73d8c8654\") "
Sep 18 17:42:03.303405 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:03.303287    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76083dd6-57b2-4c34-96d5-17b73d8c8654-home" (OuterVolumeSpecName: "home") pod "76083dd6-57b2-4c34-96d5-17b73d8c8654" (UID: "76083dd6-57b2-4c34-96d5-17b73d8c8654"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:42:03.303405 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:03.303384    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76083dd6-57b2-4c34-96d5-17b73d8c8654-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "76083dd6-57b2-4c34-96d5-17b73d8c8654" (UID: "76083dd6-57b2-4c34-96d5-17b73d8c8654"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:42:03.303559 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:03.303490    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76083dd6-57b2-4c34-96d5-17b73d8c8654-home\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:42:03.303559 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:03.303534    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76083dd6-57b2-4c34-96d5-17b73d8c8654-tmp-dir\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:42:03.303559 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:03.303488    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76083dd6-57b2-4c34-96d5-17b73d8c8654-model-cache" (OuterVolumeSpecName: "model-cache") pod "76083dd6-57b2-4c34-96d5-17b73d8c8654" (UID: "76083dd6-57b2-4c34-96d5-17b73d8c8654"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:42:03.305471 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:03.305440    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76083dd6-57b2-4c34-96d5-17b73d8c8654-kube-api-access-f7q6h" (OuterVolumeSpecName: "kube-api-access-f7q6h") pod "76083dd6-57b2-4c34-96d5-17b73d8c8654" (UID: "76083dd6-57b2-4c34-96d5-17b73d8c8654"). InnerVolumeSpecName "kube-api-access-f7q6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:42:03.305618 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:03.305603    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76083dd6-57b2-4c34-96d5-17b73d8c8654-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "76083dd6-57b2-4c34-96d5-17b73d8c8654" (UID: "76083dd6-57b2-4c34-96d5-17b73d8c8654"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:42:03.305699 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:03.305614    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76083dd6-57b2-4c34-96d5-17b73d8c8654-dshm" (OuterVolumeSpecName: "dshm") pod "76083dd6-57b2-4c34-96d5-17b73d8c8654" (UID: "76083dd6-57b2-4c34-96d5-17b73d8c8654"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:42:03.404239 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:03.404211    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76083dd6-57b2-4c34-96d5-17b73d8c8654-dshm\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:42:03.404239 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:03.404237    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7q6h\" (UniqueName: \"kubernetes.io/projected/76083dd6-57b2-4c34-96d5-17b73d8c8654-kube-api-access-f7q6h\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:42:03.404413 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:03.404248    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76083dd6-57b2-4c34-96d5-17b73d8c8654-model-cache\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:42:03.404413 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:03.404257    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76083dd6-57b2-4c34-96d5-17b73d8c8654-tls-certs\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:42:03.765143 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:03.765108    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffa59d70-a4b8-4e96-9f66-9319ac5c48a6" path="/var/lib/kubelet/pods/ffa59d70-a4b8-4e96-9f66-9319ac5c48a6/volumes"
Sep 18 17:42:04.133782 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:04.133696    2569 generic.go:356] "Generic (PLEG): container finished" podID="76083dd6-57b2-4c34-96d5-17b73d8c8654" containerID="ea94513aa202645cbe777e877e829f6abb7d219da7e475ae68f30b958213a726" exitCode=0
Sep 18 17:42:04.133782 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:04.133736    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh"
Sep 18 17:42:04.134271 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:04.133782    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh" event={"ID":"76083dd6-57b2-4c34-96d5-17b73d8c8654","Type":"ContainerDied","Data":"ea94513aa202645cbe777e877e829f6abb7d219da7e475ae68f30b958213a726"}
Sep 18 17:42:04.134271 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:04.133823    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh" event={"ID":"76083dd6-57b2-4c34-96d5-17b73d8c8654","Type":"ContainerDied","Data":"9da1a17dccf4866bf3443970abc41de8a5b08242d25c8e1333cb6aea7852e76e"}
Sep 18 17:42:04.134271 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:04.133844    2569 scope.go:117] "RemoveContainer" containerID="ea94513aa202645cbe777e877e829f6abb7d219da7e475ae68f30b958213a726"
Sep 18 17:42:04.142039 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:04.142016    2569 scope.go:117] "RemoveContainer" containerID="ea94513aa202645cbe777e877e829f6abb7d219da7e475ae68f30b958213a726"
Sep 18 17:42:04.142305 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:42:04.142287    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea94513aa202645cbe777e877e829f6abb7d219da7e475ae68f30b958213a726\": container with ID starting with ea94513aa202645cbe777e877e829f6abb7d219da7e475ae68f30b958213a726 not found: ID does not exist" containerID="ea94513aa202645cbe777e877e829f6abb7d219da7e475ae68f30b958213a726"
Sep 18 17:42:04.142377 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:04.142311    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea94513aa202645cbe777e877e829f6abb7d219da7e475ae68f30b958213a726"} err="failed to get container status \"ea94513aa202645cbe777e877e829f6abb7d219da7e475ae68f30b958213a726\": rpc error: code = NotFound desc = could not find container \"ea94513aa202645cbe777e877e829f6abb7d219da7e475ae68f30b958213a726\": container with ID starting with ea94513aa202645cbe777e877e829f6abb7d219da7e475ae68f30b958213a726 not found: ID does not exist"
Sep 18 17:42:04.150630 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:04.150601    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh"]
Sep 18 17:42:04.154186 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:04.154164    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-808470/auth-test-model-808470-kserve-54757fc7b7-n9dnh"]
Sep 18 17:42:05.764085 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:05.764051    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76083dd6-57b2-4c34-96d5-17b73d8c8654" path="/var/lib/kubelet/pods/76083dd6-57b2-4c34-96d5-17b73d8c8654/volumes"
Sep 18 17:42:07.024917 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:07.024857    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr"
Sep 18 17:42:07.025301 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:07.024932    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr"
Sep 18 17:42:07.037511 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:07.037487    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr"
Sep 18 17:42:07.161022 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:07.160997    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr"
Sep 18 17:42:22.911037 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:22.911003    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t"]
Sep 18 17:42:22.911497 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:22.911358    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffa59d70-a4b8-4e96-9f66-9319ac5c48a6" containerName="main"
Sep 18 17:42:22.911497 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:22.911374    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffa59d70-a4b8-4e96-9f66-9319ac5c48a6" containerName="main"
Sep 18 17:42:22.911497 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:22.911399    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76083dd6-57b2-4c34-96d5-17b73d8c8654" containerName="main"
Sep 18 17:42:22.911497 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:22.911404    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="76083dd6-57b2-4c34-96d5-17b73d8c8654" containerName="main"
Sep 18 17:42:22.911497 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:22.911458    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffa59d70-a4b8-4e96-9f66-9319ac5c48a6" containerName="main"
Sep 18 17:42:22.911497 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:22.911466    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="76083dd6-57b2-4c34-96d5-17b73d8c8654" containerName="main"
Sep 18 17:42:22.914327 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:22.914312    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t"
Sep 18 17:42:22.917557 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:22.917522    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-808470\"/\"openshift-service-ca.crt\""
Sep 18 17:42:22.917704 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:22.917565    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-808470\"/\"default-dockercfg-j42zw\""
Sep 18 17:42:22.917704 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:22.917576    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-808470\"/\"sub-test-model-808470-kserve-self-signed-certs\""
Sep 18 17:42:22.917704 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:22.917565    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-808470\"/\"kube-root-ca.crt\""
Sep 18 17:42:22.923429 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:22.923406    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t"]
Sep 18 17:42:22.974688 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:22.974659    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/659f338f-f44d-4f6e-8cb9-10e65f9cca07-tls-certs\") pod \"sub-test-model-808470-kserve-56b4df9749-ddg7t\" (UID: \"659f338f-f44d-4f6e-8cb9-10e65f9cca07\") " pod="ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t"
Sep 18 17:42:22.974856 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:22.974712    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/659f338f-f44d-4f6e-8cb9-10e65f9cca07-dshm\") pod \"sub-test-model-808470-kserve-56b4df9749-ddg7t\" (UID: \"659f338f-f44d-4f6e-8cb9-10e65f9cca07\") " pod="ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t"
Sep 18 17:42:22.974856 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:22.974735    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrlgl\" (UniqueName: \"kubernetes.io/projected/659f338f-f44d-4f6e-8cb9-10e65f9cca07-kube-api-access-xrlgl\") pod \"sub-test-model-808470-kserve-56b4df9749-ddg7t\" (UID: \"659f338f-f44d-4f6e-8cb9-10e65f9cca07\") " pod="ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t"
Sep 18 17:42:22.980589 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:22.975311    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/659f338f-f44d-4f6e-8cb9-10e65f9cca07-tmp-dir\") pod \"sub-test-model-808470-kserve-56b4df9749-ddg7t\" (UID: \"659f338f-f44d-4f6e-8cb9-10e65f9cca07\") " pod="ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t"
Sep 18 17:42:22.980589 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:22.975404    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/659f338f-f44d-4f6e-8cb9-10e65f9cca07-model-cache\") pod \"sub-test-model-808470-kserve-56b4df9749-ddg7t\" (UID: \"659f338f-f44d-4f6e-8cb9-10e65f9cca07\") " pod="ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t"
Sep 18 17:42:22.980589 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:22.975463    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/659f338f-f44d-4f6e-8cb9-10e65f9cca07-home\") pod \"sub-test-model-808470-kserve-56b4df9749-ddg7t\" (UID: \"659f338f-f44d-4f6e-8cb9-10e65f9cca07\") " pod="ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t"
Sep 18 17:42:23.076529 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:23.076489    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/659f338f-f44d-4f6e-8cb9-10e65f9cca07-tls-certs\") pod \"sub-test-model-808470-kserve-56b4df9749-ddg7t\" (UID: \"659f338f-f44d-4f6e-8cb9-10e65f9cca07\") " pod="ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t"
Sep 18 17:42:23.076745 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:23.076559    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/659f338f-f44d-4f6e-8cb9-10e65f9cca07-dshm\") pod \"sub-test-model-808470-kserve-56b4df9749-ddg7t\" (UID: \"659f338f-f44d-4f6e-8cb9-10e65f9cca07\") " pod="ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t"
Sep 18 17:42:23.076745 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:23.076579    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrlgl\" (UniqueName: \"kubernetes.io/projected/659f338f-f44d-4f6e-8cb9-10e65f9cca07-kube-api-access-xrlgl\") pod \"sub-test-model-808470-kserve-56b4df9749-ddg7t\" (UID: \"659f338f-f44d-4f6e-8cb9-10e65f9cca07\") " pod="ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t"
Sep 18 17:42:23.076745 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:23.076597    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/659f338f-f44d-4f6e-8cb9-10e65f9cca07-tmp-dir\") pod \"sub-test-model-808470-kserve-56b4df9749-ddg7t\" (UID: \"659f338f-f44d-4f6e-8cb9-10e65f9cca07\") " pod="ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t"
Sep 18 17:42:23.076745 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:23.076615    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/659f338f-f44d-4f6e-8cb9-10e65f9cca07-model-cache\") pod \"sub-test-model-808470-kserve-56b4df9749-ddg7t\" (UID: \"659f338f-f44d-4f6e-8cb9-10e65f9cca07\") " pod="ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t"
Sep 18 17:42:23.076745 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:23.076633    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/659f338f-f44d-4f6e-8cb9-10e65f9cca07-home\") pod \"sub-test-model-808470-kserve-56b4df9749-ddg7t\" (UID: \"659f338f-f44d-4f6e-8cb9-10e65f9cca07\") " pod="ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t"
Sep 18 17:42:23.077086 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:23.077064    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/659f338f-f44d-4f6e-8cb9-10e65f9cca07-home\") pod \"sub-test-model-808470-kserve-56b4df9749-ddg7t\" (UID: \"659f338f-f44d-4f6e-8cb9-10e65f9cca07\") " pod="ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t"
Sep 18 17:42:23.077149 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:23.077123    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/659f338f-f44d-4f6e-8cb9-10e65f9cca07-model-cache\") pod \"sub-test-model-808470-kserve-56b4df9749-ddg7t\" (UID: \"659f338f-f44d-4f6e-8cb9-10e65f9cca07\") " pod="ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t"
Sep 18 17:42:23.077204 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:23.077157    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/659f338f-f44d-4f6e-8cb9-10e65f9cca07-tmp-dir\") pod \"sub-test-model-808470-kserve-56b4df9749-ddg7t\" (UID: \"659f338f-f44d-4f6e-8cb9-10e65f9cca07\") " pod="ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t"
Sep 18 17:42:23.078916 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:23.078847    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/659f338f-f44d-4f6e-8cb9-10e65f9cca07-dshm\") pod \"sub-test-model-808470-kserve-56b4df9749-ddg7t\" (UID: \"659f338f-f44d-4f6e-8cb9-10e65f9cca07\") " pod="ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t"
Sep 18 17:42:23.079145 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:23.079126    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/659f338f-f44d-4f6e-8cb9-10e65f9cca07-tls-certs\") pod \"sub-test-model-808470-kserve-56b4df9749-ddg7t\" (UID: \"659f338f-f44d-4f6e-8cb9-10e65f9cca07\") " pod="ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t"
Sep 18 17:42:23.085647 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:23.085625    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrlgl\" (UniqueName: \"kubernetes.io/projected/659f338f-f44d-4f6e-8cb9-10e65f9cca07-kube-api-access-xrlgl\") pod \"sub-test-model-808470-kserve-56b4df9749-ddg7t\" (UID: \"659f338f-f44d-4f6e-8cb9-10e65f9cca07\") " pod="ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t"
Sep 18 17:42:23.224215 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:23.224138    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t"
Sep 18 17:42:23.495289 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:23.495263    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t"]
Sep 18 17:42:23.497460 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:42:23.497435    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod659f338f_f44d_4f6e_8cb9_10e65f9cca07.slice/crio-303c5531f480daf9a8fd6cdca86fce2485ff439d071ef64f4cfd5a63fd98fd99 WatchSource:0}: Error finding container 303c5531f480daf9a8fd6cdca86fce2485ff439d071ef64f4cfd5a63fd98fd99: Status 404 returned error can't find the container with id 303c5531f480daf9a8fd6cdca86fce2485ff439d071ef64f4cfd5a63fd98fd99
Sep 18 17:42:24.232561 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:24.232527    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t" event={"ID":"659f338f-f44d-4f6e-8cb9-10e65f9cca07","Type":"ContainerStarted","Data":"db4b41c8ceb390d9e5030205c09c7dacfff1467a352e8224731ae6d95e82a51c"}
Sep 18 17:42:24.232982 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:24.232568    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t" event={"ID":"659f338f-f44d-4f6e-8cb9-10e65f9cca07","Type":"ContainerStarted","Data":"303c5531f480daf9a8fd6cdca86fce2485ff439d071ef64f4cfd5a63fd98fd99"}
Sep 18 17:42:24.252333 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:24.252282    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t" podStartSLOduration=2.252268851 podStartE2EDuration="2.252268851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:42:24.250826996 +0000 UTC m=+1656.944754545" watchObservedRunningTime="2026-09-18 17:42:24.252268851 +0000 UTC m=+1656.946196379"
Sep 18 17:42:33.224662 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:33.224614    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t"
Sep 18 17:42:33.224662 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:33.224679    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t"
Sep 18 17:42:33.237157 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:33.237130    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t"
Sep 18 17:42:33.288520 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:33.288496    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t"
Sep 18 17:42:48.146118 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:42:48.146075    2569 scope.go:117] "RemoveContainer" containerID="885b832cafc5efe9ee0e06f00cfc3b4d6b1fc69238e96c535d6dcae20d2ff136"
Sep 18 17:43:09.122530 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:09.122494    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7726-496a9e"]
Sep 18 17:43:09.125916 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:09.125897    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7726-496a9e"
Sep 18 17:43:09.128998 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:09.128979    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mjflw\""
Sep 18 17:43:09.131968 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:09.131945    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7726-496a9e"]
Sep 18 17:43:09.186015 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:09.185980    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fz2g\" (UniqueName: \"kubernetes.io/projected/136023d6-3c87-43e5-9bc2-37e912eb7030-kube-api-access-7fz2g\") pod \"test-curl-7726-496a9e\" (UID: \"136023d6-3c87-43e5-9bc2-37e912eb7030\") " pod="opendatahub/test-curl-7726-496a9e"
Sep 18 17:43:09.287125 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:09.287082    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fz2g\" (UniqueName: \"kubernetes.io/projected/136023d6-3c87-43e5-9bc2-37e912eb7030-kube-api-access-7fz2g\") pod \"test-curl-7726-496a9e\" (UID: \"136023d6-3c87-43e5-9bc2-37e912eb7030\") " pod="opendatahub/test-curl-7726-496a9e"
Sep 18 17:43:09.296496 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:09.296473    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fz2g\" (UniqueName: \"kubernetes.io/projected/136023d6-3c87-43e5-9bc2-37e912eb7030-kube-api-access-7fz2g\") pod \"test-curl-7726-496a9e\" (UID: \"136023d6-3c87-43e5-9bc2-37e912eb7030\") " pod="opendatahub/test-curl-7726-496a9e"
Sep 18 17:43:09.436045 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:09.435945    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7726-496a9e"
Sep 18 17:43:09.710808 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:09.708389    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7726-496a9e"]
Sep 18 17:43:10.469137 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:10.469039    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7726-496a9e" event={"ID":"136023d6-3c87-43e5-9bc2-37e912eb7030","Type":"ContainerStarted","Data":"7802baaeba0c9b7d0734f8a64db7603e3da9bb3433b0364b75c2052e502e1829"}
Sep 18 17:43:10.469137 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:10.469087    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7726-496a9e" event={"ID":"136023d6-3c87-43e5-9bc2-37e912eb7030","Type":"ContainerStarted","Data":"14803198efc43440ec7daeeb446f952d1a207bd4bf4ff64bb9dea3e139b56b23"}
Sep 18 17:43:10.487692 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:10.487645    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7726-496a9e" podStartSLOduration=1.487631642 podStartE2EDuration="1.487631642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:43:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:43:10.486282875 +0000 UTC m=+1703.180210403" watchObservedRunningTime="2026-09-18 17:43:10.487631642 +0000 UTC m=+1703.181559169"
Sep 18 17:43:11.206072 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:11.206037    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7726-496a9e"]
Sep 18 17:43:11.210385 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:11.210358    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7726-496a9e"]
Sep 18 17:43:11.336064 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:11.336025    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7726-6e45b1"]
Sep 18 17:43:11.336526 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:11.336507    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="136023d6-3c87-43e5-9bc2-37e912eb7030" containerName="test-curl-7726-496a9e"
Sep 18 17:43:11.336586 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:11.336530    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="136023d6-3c87-43e5-9bc2-37e912eb7030" containerName="test-curl-7726-496a9e"
Sep 18 17:43:11.336644 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:11.336630    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="136023d6-3c87-43e5-9bc2-37e912eb7030" containerName="test-curl-7726-496a9e"
Sep 18 17:43:11.339597 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:11.339583    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7726-6e45b1"
Sep 18 17:43:11.355108 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:11.355085    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7726-6e45b1"]
Sep 18 17:43:11.404258 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:11.404231    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jtnb\" (UniqueName: \"kubernetes.io/projected/db05dc3f-5025-4a23-8179-4e9adde97b8b-kube-api-access-8jtnb\") pod \"test-curl-7726-6e45b1\" (UID: \"db05dc3f-5025-4a23-8179-4e9adde97b8b\") " pod="opendatahub/test-curl-7726-6e45b1"
Sep 18 17:43:11.474383 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:43:11.474356    2569 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7726-496a9e" container="test-curl-7726-496a9e"
Sep 18 17:43:11.505150 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:11.505112    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jtnb\" (UniqueName: \"kubernetes.io/projected/db05dc3f-5025-4a23-8179-4e9adde97b8b-kube-api-access-8jtnb\") pod \"test-curl-7726-6e45b1\" (UID: \"db05dc3f-5025-4a23-8179-4e9adde97b8b\") " pod="opendatahub/test-curl-7726-6e45b1"
Sep 18 17:43:11.517607 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:11.517580    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jtnb\" (UniqueName: \"kubernetes.io/projected/db05dc3f-5025-4a23-8179-4e9adde97b8b-kube-api-access-8jtnb\") pod \"test-curl-7726-6e45b1\" (UID: \"db05dc3f-5025-4a23-8179-4e9adde97b8b\") " pod="opendatahub/test-curl-7726-6e45b1"
Sep 18 17:43:11.649004 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:11.648922    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7726-6e45b1"
Sep 18 17:43:11.926895 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:11.926852    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7726-6e45b1"]
Sep 18 17:43:11.927692 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:43:11.927665    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb05dc3f_5025_4a23_8179_4e9adde97b8b.slice/crio-73fc3d248af39f2f8350dbc9f876f76fc0969a58459a7e26609083dd2fbbec2a WatchSource:0}: Error finding container 73fc3d248af39f2f8350dbc9f876f76fc0969a58459a7e26609083dd2fbbec2a: Status 404 returned error can't find the container with id 73fc3d248af39f2f8350dbc9f876f76fc0969a58459a7e26609083dd2fbbec2a
Sep 18 17:43:12.480209 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:12.480174    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7726-6e45b1" event={"ID":"db05dc3f-5025-4a23-8179-4e9adde97b8b","Type":"ContainerStarted","Data":"131e1a3e182cd79daa43d343a6f835cb31de02d02056dfa0b8b1dbbbea4d01f4"}
Sep 18 17:43:12.480209 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:12.480221    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7726-6e45b1" event={"ID":"db05dc3f-5025-4a23-8179-4e9adde97b8b","Type":"ContainerStarted","Data":"73fc3d248af39f2f8350dbc9f876f76fc0969a58459a7e26609083dd2fbbec2a"}
Sep 18 17:43:12.480711 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:12.480246    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7726-496a9e" podUID="136023d6-3c87-43e5-9bc2-37e912eb7030" containerName="test-curl-7726-496a9e" containerID="cri-o://7802baaeba0c9b7d0734f8a64db7603e3da9bb3433b0364b75c2052e502e1829" gracePeriod=2
Sep 18 17:43:12.502811 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:12.502753    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7726-6e45b1" podStartSLOduration=1.502737443 podStartE2EDuration="1.502737443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:43:12.502388937 +0000 UTC m=+1705.196316468" watchObservedRunningTime="2026-09-18 17:43:12.502737443 +0000 UTC m=+1705.196664970"
Sep 18 17:43:13.319626 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:13.319538    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7726-6e45b1"]
Sep 18 17:43:13.325233 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:13.325202    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7726-6e45b1"]
Sep 18 17:43:13.485306 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:43:13.485275    2569 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7726-6e45b1" container="test-curl-7726-6e45b1"
Sep 18 17:43:14.490321 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:14.490283    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7726-6e45b1" podUID="db05dc3f-5025-4a23-8179-4e9adde97b8b" containerName="test-curl-7726-6e45b1" containerID="cri-o://131e1a3e182cd79daa43d343a6f835cb31de02d02056dfa0b8b1dbbbea4d01f4" gracePeriod=2
Sep 18 17:43:14.621478 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:14.621451    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7726-496a9e"
Sep 18 17:43:14.733983 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:14.733895    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fz2g\" (UniqueName: \"kubernetes.io/projected/136023d6-3c87-43e5-9bc2-37e912eb7030-kube-api-access-7fz2g\") pod \"136023d6-3c87-43e5-9bc2-37e912eb7030\" (UID: \"136023d6-3c87-43e5-9bc2-37e912eb7030\") "
Sep 18 17:43:14.736034 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:14.736004    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/136023d6-3c87-43e5-9bc2-37e912eb7030-kube-api-access-7fz2g" (OuterVolumeSpecName: "kube-api-access-7fz2g") pod "136023d6-3c87-43e5-9bc2-37e912eb7030" (UID: "136023d6-3c87-43e5-9bc2-37e912eb7030"). InnerVolumeSpecName "kube-api-access-7fz2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:43:14.835249 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:14.835201    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fz2g\" (UniqueName: \"kubernetes.io/projected/136023d6-3c87-43e5-9bc2-37e912eb7030-kube-api-access-7fz2g\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:43:15.495995 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:15.495962    2569 generic.go:356] "Generic (PLEG): container finished" podID="136023d6-3c87-43e5-9bc2-37e912eb7030" containerID="7802baaeba0c9b7d0734f8a64db7603e3da9bb3433b0364b75c2052e502e1829" exitCode=137
Sep 18 17:43:15.496408 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:15.496004    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7726-496a9e"
Sep 18 17:43:15.496408 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:15.496063    2569 scope.go:117] "RemoveContainer" containerID="7802baaeba0c9b7d0734f8a64db7603e3da9bb3433b0364b75c2052e502e1829"
Sep 18 17:43:15.504930 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:15.504892    2569 scope.go:117] "RemoveContainer" containerID="7802baaeba0c9b7d0734f8a64db7603e3da9bb3433b0364b75c2052e502e1829"
Sep 18 17:43:15.505171 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:43:15.505149    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7802baaeba0c9b7d0734f8a64db7603e3da9bb3433b0364b75c2052e502e1829\": container with ID starting with 7802baaeba0c9b7d0734f8a64db7603e3da9bb3433b0364b75c2052e502e1829 not found: ID does not exist" containerID="7802baaeba0c9b7d0734f8a64db7603e3da9bb3433b0364b75c2052e502e1829"
Sep 18 17:43:15.505239 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:15.505178    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7802baaeba0c9b7d0734f8a64db7603e3da9bb3433b0364b75c2052e502e1829"} err="failed to get container status \"7802baaeba0c9b7d0734f8a64db7603e3da9bb3433b0364b75c2052e502e1829\": rpc error: code = NotFound desc = could not find container \"7802baaeba0c9b7d0734f8a64db7603e3da9bb3433b0364b75c2052e502e1829\": container with ID starting with 7802baaeba0c9b7d0734f8a64db7603e3da9bb3433b0364b75c2052e502e1829 not found: ID does not exist"
Sep 18 17:43:15.511979 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:43:15.511957    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7726-496a9e\" is forbidden: User \"system:node:ip-10-0-139-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-122.ec2.internal' and this object" podUID="136023d6-3c87-43e5-9bc2-37e912eb7030" pod="opendatahub/test-curl-7726-496a9e"
Sep 18 17:43:15.726225 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:15.726194    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr"]
Sep 18 17:43:15.728881 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:43:15.728837    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7726-496a9e\" is forbidden: User \"system:node:ip-10-0-139-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-122.ec2.internal' and this object" podUID="136023d6-3c87-43e5-9bc2-37e912eb7030" pod="opendatahub/test-curl-7726-496a9e"
Sep 18 17:43:15.764587 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:15.764528    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="136023d6-3c87-43e5-9bc2-37e912eb7030" path="/var/lib/kubelet/pods/136023d6-3c87-43e5-9bc2-37e912eb7030/volumes"
Sep 18 17:43:16.632106 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:16.632081    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7726-6e45b1"
Sep 18 17:43:16.719163 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:16.719134    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t"]
Sep 18 17:43:16.751640 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:16.751615    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jtnb\" (UniqueName: \"kubernetes.io/projected/db05dc3f-5025-4a23-8179-4e9adde97b8b-kube-api-access-8jtnb\") pod \"db05dc3f-5025-4a23-8179-4e9adde97b8b\" (UID: \"db05dc3f-5025-4a23-8179-4e9adde97b8b\") "
Sep 18 17:43:16.753522 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:16.753498    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db05dc3f-5025-4a23-8179-4e9adde97b8b-kube-api-access-8jtnb" (OuterVolumeSpecName: "kube-api-access-8jtnb") pod "db05dc3f-5025-4a23-8179-4e9adde97b8b" (UID: "db05dc3f-5025-4a23-8179-4e9adde97b8b"). InnerVolumeSpecName "kube-api-access-8jtnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:43:16.852892 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:16.852789    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jtnb\" (UniqueName: \"kubernetes.io/projected/db05dc3f-5025-4a23-8179-4e9adde97b8b-kube-api-access-8jtnb\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:43:17.508566 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:17.508529    2569 generic.go:356] "Generic (PLEG): container finished" podID="db05dc3f-5025-4a23-8179-4e9adde97b8b" containerID="131e1a3e182cd79daa43d343a6f835cb31de02d02056dfa0b8b1dbbbea4d01f4" exitCode=137
Sep 18 17:43:17.508743 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:17.508573    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7726-6e45b1"
Sep 18 17:43:17.508743 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:17.508625    2569 scope.go:117] "RemoveContainer" containerID="131e1a3e182cd79daa43d343a6f835cb31de02d02056dfa0b8b1dbbbea4d01f4"
Sep 18 17:43:17.517594 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:17.517571    2569 scope.go:117] "RemoveContainer" containerID="131e1a3e182cd79daa43d343a6f835cb31de02d02056dfa0b8b1dbbbea4d01f4"
Sep 18 17:43:17.517882 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:43:17.517845    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"131e1a3e182cd79daa43d343a6f835cb31de02d02056dfa0b8b1dbbbea4d01f4\": container with ID starting with 131e1a3e182cd79daa43d343a6f835cb31de02d02056dfa0b8b1dbbbea4d01f4 not found: ID does not exist" containerID="131e1a3e182cd79daa43d343a6f835cb31de02d02056dfa0b8b1dbbbea4d01f4"
Sep 18 17:43:17.517942 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:17.517890    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"131e1a3e182cd79daa43d343a6f835cb31de02d02056dfa0b8b1dbbbea4d01f4"} err="failed to get container status \"131e1a3e182cd79daa43d343a6f835cb31de02d02056dfa0b8b1dbbbea4d01f4\": rpc error: code = NotFound desc = could not find container \"131e1a3e182cd79daa43d343a6f835cb31de02d02056dfa0b8b1dbbbea4d01f4\": container with ID starting with 131e1a3e182cd79daa43d343a6f835cb31de02d02056dfa0b8b1dbbbea4d01f4 not found: ID does not exist"
Sep 18 17:43:17.519069 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:43:17.519048    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7726-6e45b1\" is forbidden: User \"system:node:ip-10-0-139-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-122.ec2.internal' and this object" podUID="db05dc3f-5025-4a23-8179-4e9adde97b8b" pod="opendatahub/test-curl-7726-6e45b1"
Sep 18 17:43:17.763280 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:43:17.763200    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7726-6e45b1\" is forbidden: User \"system:node:ip-10-0-139-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-122.ec2.internal' and this object" podUID="db05dc3f-5025-4a23-8179-4e9adde97b8b" pod="opendatahub/test-curl-7726-6e45b1"
Sep 18 17:43:17.764461 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:17.764438    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db05dc3f-5025-4a23-8179-4e9adde97b8b" path="/var/lib/kubelet/pods/db05dc3f-5025-4a23-8179-4e9adde97b8b/volumes"
Sep 18 17:43:28.318117 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:28.318083    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82"]
Sep 18 17:43:28.318483 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:28.318399    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db05dc3f-5025-4a23-8179-4e9adde97b8b" containerName="test-curl-7726-6e45b1"
Sep 18 17:43:28.318483 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:28.318411    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="db05dc3f-5025-4a23-8179-4e9adde97b8b" containerName="test-curl-7726-6e45b1"
Sep 18 17:43:28.318483 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:28.318474    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="db05dc3f-5025-4a23-8179-4e9adde97b8b" containerName="test-curl-7726-6e45b1"
Sep 18 17:43:28.321196 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:28.321180    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82"
Sep 18 17:43:28.324363 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:28.324342    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-eeab6e\"/\"rate-test-model-eeab6e-kserve-self-signed-certs\""
Sep 18 17:43:28.338548 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:28.338520    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82"]
Sep 18 17:43:28.458447 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:28.458408    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a07ab377-db7a-4873-81c6-eada976206d7-home\") pod \"rate-test-model-eeab6e-kserve-667bd4b775-d2c82\" (UID: \"a07ab377-db7a-4873-81c6-eada976206d7\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82"
Sep 18 17:43:28.458617 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:28.458453    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a07ab377-db7a-4873-81c6-eada976206d7-model-cache\") pod \"rate-test-model-eeab6e-kserve-667bd4b775-d2c82\" (UID: \"a07ab377-db7a-4873-81c6-eada976206d7\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82"
Sep 18 17:43:28.458617 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:28.458483    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a07ab377-db7a-4873-81c6-eada976206d7-tmp-dir\") pod \"rate-test-model-eeab6e-kserve-667bd4b775-d2c82\" (UID: \"a07ab377-db7a-4873-81c6-eada976206d7\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82"
Sep 18 17:43:28.458617 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:28.458528    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv9p8\" (UniqueName: \"kubernetes.io/projected/a07ab377-db7a-4873-81c6-eada976206d7-kube-api-access-vv9p8\") pod \"rate-test-model-eeab6e-kserve-667bd4b775-d2c82\" (UID: \"a07ab377-db7a-4873-81c6-eada976206d7\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82"
Sep 18 17:43:28.458617 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:28.458545    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a07ab377-db7a-4873-81c6-eada976206d7-tls-certs\") pod \"rate-test-model-eeab6e-kserve-667bd4b775-d2c82\" (UID: \"a07ab377-db7a-4873-81c6-eada976206d7\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82"
Sep 18 17:43:28.458617 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:28.458593    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a07ab377-db7a-4873-81c6-eada976206d7-dshm\") pod \"rate-test-model-eeab6e-kserve-667bd4b775-d2c82\" (UID: \"a07ab377-db7a-4873-81c6-eada976206d7\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82"
Sep 18 17:43:28.559048 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:28.559018    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a07ab377-db7a-4873-81c6-eada976206d7-dshm\") pod \"rate-test-model-eeab6e-kserve-667bd4b775-d2c82\" (UID: \"a07ab377-db7a-4873-81c6-eada976206d7\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82"
Sep 18 17:43:28.559202 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:28.559074    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a07ab377-db7a-4873-81c6-eada976206d7-home\") pod \"rate-test-model-eeab6e-kserve-667bd4b775-d2c82\" (UID: \"a07ab377-db7a-4873-81c6-eada976206d7\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82"
Sep 18 17:43:28.559202 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:28.559106    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a07ab377-db7a-4873-81c6-eada976206d7-model-cache\") pod \"rate-test-model-eeab6e-kserve-667bd4b775-d2c82\" (UID: \"a07ab377-db7a-4873-81c6-eada976206d7\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82"
Sep 18 17:43:28.559202 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:28.559141    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a07ab377-db7a-4873-81c6-eada976206d7-tmp-dir\") pod \"rate-test-model-eeab6e-kserve-667bd4b775-d2c82\" (UID: \"a07ab377-db7a-4873-81c6-eada976206d7\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82"
Sep 18 17:43:28.559202 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:28.559184    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv9p8\" (UniqueName: \"kubernetes.io/projected/a07ab377-db7a-4873-81c6-eada976206d7-kube-api-access-vv9p8\") pod \"rate-test-model-eeab6e-kserve-667bd4b775-d2c82\" (UID: \"a07ab377-db7a-4873-81c6-eada976206d7\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82"
Sep 18 17:43:28.559414 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:28.559210    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a07ab377-db7a-4873-81c6-eada976206d7-tls-certs\") pod \"rate-test-model-eeab6e-kserve-667bd4b775-d2c82\" (UID: \"a07ab377-db7a-4873-81c6-eada976206d7\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82"
Sep 18 17:43:28.559582 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:28.559553    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a07ab377-db7a-4873-81c6-eada976206d7-tmp-dir\") pod \"rate-test-model-eeab6e-kserve-667bd4b775-d2c82\" (UID: \"a07ab377-db7a-4873-81c6-eada976206d7\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82"
Sep 18 17:43:28.559699 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:28.559565    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a07ab377-db7a-4873-81c6-eada976206d7-home\") pod \"rate-test-model-eeab6e-kserve-667bd4b775-d2c82\" (UID: \"a07ab377-db7a-4873-81c6-eada976206d7\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82"
Sep 18 17:43:28.559699 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:28.559618    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a07ab377-db7a-4873-81c6-eada976206d7-model-cache\") pod \"rate-test-model-eeab6e-kserve-667bd4b775-d2c82\" (UID: \"a07ab377-db7a-4873-81c6-eada976206d7\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82"
Sep 18 17:43:28.561967 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:28.561943    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a07ab377-db7a-4873-81c6-eada976206d7-dshm\") pod \"rate-test-model-eeab6e-kserve-667bd4b775-d2c82\" (UID: \"a07ab377-db7a-4873-81c6-eada976206d7\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82"
Sep 18 17:43:28.562155 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:28.562140    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a07ab377-db7a-4873-81c6-eada976206d7-tls-certs\") pod \"rate-test-model-eeab6e-kserve-667bd4b775-d2c82\" (UID: \"a07ab377-db7a-4873-81c6-eada976206d7\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82"
Sep 18 17:43:28.570999 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:28.570942    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv9p8\" (UniqueName: \"kubernetes.io/projected/a07ab377-db7a-4873-81c6-eada976206d7-kube-api-access-vv9p8\") pod \"rate-test-model-eeab6e-kserve-667bd4b775-d2c82\" (UID: \"a07ab377-db7a-4873-81c6-eada976206d7\") " pod="ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82"
Sep 18 17:43:28.630066 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:28.630030    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82"
Sep 18 17:43:28.903385 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:28.903301    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82"]
Sep 18 17:43:28.906044 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:43:28.906014    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda07ab377_db7a_4873_81c6_eada976206d7.slice/crio-48a2a183599aa5c417f84a59a30209e8fc93f8f562a6b284d43ec48b6589f88e WatchSource:0}: Error finding container 48a2a183599aa5c417f84a59a30209e8fc93f8f562a6b284d43ec48b6589f88e: Status 404 returned error can't find the container with id 48a2a183599aa5c417f84a59a30209e8fc93f8f562a6b284d43ec48b6589f88e
Sep 18 17:43:29.570672 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:29.570641    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82" event={"ID":"a07ab377-db7a-4873-81c6-eada976206d7","Type":"ContainerStarted","Data":"e72da8e73b1380d1778d6686daf4973ac6eee13e459bd52abe2dc4c100ddb5c6"}
Sep 18 17:43:29.571071 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:29.570679    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82" event={"ID":"a07ab377-db7a-4873-81c6-eada976206d7","Type":"ContainerStarted","Data":"48a2a183599aa5c417f84a59a30209e8fc93f8f562a6b284d43ec48b6589f88e"}
Sep 18 17:43:29.590006 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:29.589947    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82" podStartSLOduration=1.5899306869999998 podStartE2EDuration="1.589930687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:43:29.58987854 +0000 UTC m=+1722.283806079" watchObservedRunningTime="2026-09-18 17:43:29.589930687 +0000 UTC m=+1722.283858214"
Sep 18 17:43:30.744764 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:30.744721    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr" podUID="bf13fa17-9ec1-4e60-ab9f-2f392d7ada44" containerName="main" containerID="cri-o://e8a0985fa072c693ee1daab4dd96cdd300709dcff647607e707f9a398d2e48e7" gracePeriod=45
Sep 18 17:43:30.979050 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:30.979026    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr"
Sep 18 17:43:31.084826 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:31.084741    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-home\") pod \"bf13fa17-9ec1-4e60-ab9f-2f392d7ada44\" (UID: \"bf13fa17-9ec1-4e60-ab9f-2f392d7ada44\") "
Sep 18 17:43:31.084826 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:31.084814    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-model-cache\") pod \"bf13fa17-9ec1-4e60-ab9f-2f392d7ada44\" (UID: \"bf13fa17-9ec1-4e60-ab9f-2f392d7ada44\") "
Sep 18 17:43:31.085090 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:31.084844    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-tls-certs\") pod \"bf13fa17-9ec1-4e60-ab9f-2f392d7ada44\" (UID: \"bf13fa17-9ec1-4e60-ab9f-2f392d7ada44\") "
Sep 18 17:43:31.085090 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:31.084886    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-tmp-dir\") pod \"bf13fa17-9ec1-4e60-ab9f-2f392d7ada44\" (UID: \"bf13fa17-9ec1-4e60-ab9f-2f392d7ada44\") "
Sep 18 17:43:31.085090 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:31.084915    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbdns\" (UniqueName: \"kubernetes.io/projected/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-kube-api-access-tbdns\") pod \"bf13fa17-9ec1-4e60-ab9f-2f392d7ada44\" (UID: \"bf13fa17-9ec1-4e60-ab9f-2f392d7ada44\") "
Sep 18 17:43:31.085090 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:31.084937    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-dshm\") pod \"bf13fa17-9ec1-4e60-ab9f-2f392d7ada44\" (UID: \"bf13fa17-9ec1-4e60-ab9f-2f392d7ada44\") "
Sep 18 17:43:31.085294 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:31.085062    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-home" (OuterVolumeSpecName: "home") pod "bf13fa17-9ec1-4e60-ab9f-2f392d7ada44" (UID: "bf13fa17-9ec1-4e60-ab9f-2f392d7ada44"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:43:31.085294 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:31.085139    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-model-cache" (OuterVolumeSpecName: "model-cache") pod "bf13fa17-9ec1-4e60-ab9f-2f392d7ada44" (UID: "bf13fa17-9ec1-4e60-ab9f-2f392d7ada44"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:43:31.085294 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:31.085148    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bf13fa17-9ec1-4e60-ab9f-2f392d7ada44" (UID: "bf13fa17-9ec1-4e60-ab9f-2f392d7ada44"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:43:31.085294 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:31.085234    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-home\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:43:31.085294 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:31.085246    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-model-cache\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:43:31.085294 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:31.085255    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-tmp-dir\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:43:31.087129 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:31.087094    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-dshm" (OuterVolumeSpecName: "dshm") pod "bf13fa17-9ec1-4e60-ab9f-2f392d7ada44" (UID: "bf13fa17-9ec1-4e60-ab9f-2f392d7ada44"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:43:31.087239 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:31.087188    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bf13fa17-9ec1-4e60-ab9f-2f392d7ada44" (UID: "bf13fa17-9ec1-4e60-ab9f-2f392d7ada44"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:43:31.087239 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:31.087211    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-kube-api-access-tbdns" (OuterVolumeSpecName: "kube-api-access-tbdns") pod "bf13fa17-9ec1-4e60-ab9f-2f392d7ada44" (UID: "bf13fa17-9ec1-4e60-ab9f-2f392d7ada44"). InnerVolumeSpecName "kube-api-access-tbdns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:43:31.185687 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:31.185655    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-tls-certs\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:43:31.185687 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:31.185684    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbdns\" (UniqueName: \"kubernetes.io/projected/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-kube-api-access-tbdns\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:43:31.185687 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:31.185693    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44-dshm\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:43:31.582803 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:31.582763    2569 generic.go:356] "Generic (PLEG): container finished" podID="bf13fa17-9ec1-4e60-ab9f-2f392d7ada44" containerID="e8a0985fa072c693ee1daab4dd96cdd300709dcff647607e707f9a398d2e48e7" exitCode=0
Sep 18 17:43:31.583035 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:31.582810    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr"
Sep 18 17:43:31.583035 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:31.582825    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr" event={"ID":"bf13fa17-9ec1-4e60-ab9f-2f392d7ada44","Type":"ContainerDied","Data":"e8a0985fa072c693ee1daab4dd96cdd300709dcff647607e707f9a398d2e48e7"}
Sep 18 17:43:31.583035 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:31.582883    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr" event={"ID":"bf13fa17-9ec1-4e60-ab9f-2f392d7ada44","Type":"ContainerDied","Data":"a47d4e6a07d6876e86143be5aed6cb735f0e0d90f9253c703d4bd58e3ee376c7"}
Sep 18 17:43:31.583035 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:31.582898    2569 scope.go:117] "RemoveContainer" containerID="e8a0985fa072c693ee1daab4dd96cdd300709dcff647607e707f9a398d2e48e7"
Sep 18 17:43:31.593424 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:31.593403    2569 scope.go:117] "RemoveContainer" containerID="e8a0985fa072c693ee1daab4dd96cdd300709dcff647607e707f9a398d2e48e7"
Sep 18 17:43:31.593707 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:43:31.593685    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8a0985fa072c693ee1daab4dd96cdd300709dcff647607e707f9a398d2e48e7\": container with ID starting with e8a0985fa072c693ee1daab4dd96cdd300709dcff647607e707f9a398d2e48e7 not found: ID does not exist" containerID="e8a0985fa072c693ee1daab4dd96cdd300709dcff647607e707f9a398d2e48e7"
Sep 18 17:43:31.593796 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:31.593714    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8a0985fa072c693ee1daab4dd96cdd300709dcff647607e707f9a398d2e48e7"} err="failed to get container status \"e8a0985fa072c693ee1daab4dd96cdd300709dcff647607e707f9a398d2e48e7\": rpc error: code = NotFound desc = could not find container \"e8a0985fa072c693ee1daab4dd96cdd300709dcff647607e707f9a398d2e48e7\": container with ID starting with e8a0985fa072c693ee1daab4dd96cdd300709dcff647607e707f9a398d2e48e7 not found: ID does not exist"
Sep 18 17:43:31.608026 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:31.607991    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr"]
Sep 18 17:43:31.611460 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:31.611436    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-eeab6e/sub-test-model-eeab6e-kserve-7cdf5d5fcf-jhlhr"]
Sep 18 17:43:31.737728 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:31.737681    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t" podUID="659f338f-f44d-4f6e-8cb9-10e65f9cca07" containerName="main" containerID="cri-o://db4b41c8ceb390d9e5030205c09c7dacfff1467a352e8224731ae6d95e82a51c" gracePeriod=45
Sep 18 17:43:31.764970 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:31.764932    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf13fa17-9ec1-4e60-ab9f-2f392d7ada44" path="/var/lib/kubelet/pods/bf13fa17-9ec1-4e60-ab9f-2f392d7ada44/volumes"
Sep 18 17:43:32.003071 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:32.003045    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t"
Sep 18 17:43:32.195846 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:32.195749    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/659f338f-f44d-4f6e-8cb9-10e65f9cca07-home\") pod \"659f338f-f44d-4f6e-8cb9-10e65f9cca07\" (UID: \"659f338f-f44d-4f6e-8cb9-10e65f9cca07\") "
Sep 18 17:43:32.195846 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:32.195813    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/659f338f-f44d-4f6e-8cb9-10e65f9cca07-model-cache\") pod \"659f338f-f44d-4f6e-8cb9-10e65f9cca07\" (UID: \"659f338f-f44d-4f6e-8cb9-10e65f9cca07\") "
Sep 18 17:43:32.196108 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:32.195877    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/659f338f-f44d-4f6e-8cb9-10e65f9cca07-tmp-dir\") pod \"659f338f-f44d-4f6e-8cb9-10e65f9cca07\" (UID: \"659f338f-f44d-4f6e-8cb9-10e65f9cca07\") "
Sep 18 17:43:32.196108 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:32.195923    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/659f338f-f44d-4f6e-8cb9-10e65f9cca07-dshm\") pod \"659f338f-f44d-4f6e-8cb9-10e65f9cca07\" (UID: \"659f338f-f44d-4f6e-8cb9-10e65f9cca07\") "
Sep 18 17:43:32.196108 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:32.195953    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrlgl\" (UniqueName: \"kubernetes.io/projected/659f338f-f44d-4f6e-8cb9-10e65f9cca07-kube-api-access-xrlgl\") pod \"659f338f-f44d-4f6e-8cb9-10e65f9cca07\" (UID: \"659f338f-f44d-4f6e-8cb9-10e65f9cca07\") "
Sep 18 17:43:32.196108 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:32.195986    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/659f338f-f44d-4f6e-8cb9-10e65f9cca07-tls-certs\") pod \"659f338f-f44d-4f6e-8cb9-10e65f9cca07\" (UID: \"659f338f-f44d-4f6e-8cb9-10e65f9cca07\") "
Sep 18 17:43:32.196108 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:32.196051    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/659f338f-f44d-4f6e-8cb9-10e65f9cca07-home" (OuterVolumeSpecName: "home") pod "659f338f-f44d-4f6e-8cb9-10e65f9cca07" (UID: "659f338f-f44d-4f6e-8cb9-10e65f9cca07"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:43:32.196108 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:32.196074    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/659f338f-f44d-4f6e-8cb9-10e65f9cca07-model-cache" (OuterVolumeSpecName: "model-cache") pod "659f338f-f44d-4f6e-8cb9-10e65f9cca07" (UID: "659f338f-f44d-4f6e-8cb9-10e65f9cca07"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:43:32.196396 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:32.196128    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/659f338f-f44d-4f6e-8cb9-10e65f9cca07-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "659f338f-f44d-4f6e-8cb9-10e65f9cca07" (UID: "659f338f-f44d-4f6e-8cb9-10e65f9cca07"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:43:32.196396 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:32.196308    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/659f338f-f44d-4f6e-8cb9-10e65f9cca07-home\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:43:32.196396 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:32.196325    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/659f338f-f44d-4f6e-8cb9-10e65f9cca07-model-cache\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:43:32.196396 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:32.196334    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/659f338f-f44d-4f6e-8cb9-10e65f9cca07-tmp-dir\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:43:32.198227 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:32.198199    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/659f338f-f44d-4f6e-8cb9-10e65f9cca07-kube-api-access-xrlgl" (OuterVolumeSpecName: "kube-api-access-xrlgl") pod "659f338f-f44d-4f6e-8cb9-10e65f9cca07" (UID: "659f338f-f44d-4f6e-8cb9-10e65f9cca07"). InnerVolumeSpecName "kube-api-access-xrlgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:43:32.198335 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:32.198238    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/659f338f-f44d-4f6e-8cb9-10e65f9cca07-dshm" (OuterVolumeSpecName: "dshm") pod "659f338f-f44d-4f6e-8cb9-10e65f9cca07" (UID: "659f338f-f44d-4f6e-8cb9-10e65f9cca07"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:43:32.198411 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:32.198364    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/659f338f-f44d-4f6e-8cb9-10e65f9cca07-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "659f338f-f44d-4f6e-8cb9-10e65f9cca07" (UID: "659f338f-f44d-4f6e-8cb9-10e65f9cca07"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:43:32.297708 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:32.297661    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrlgl\" (UniqueName: \"kubernetes.io/projected/659f338f-f44d-4f6e-8cb9-10e65f9cca07-kube-api-access-xrlgl\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:43:32.297708 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:32.297707    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/659f338f-f44d-4f6e-8cb9-10e65f9cca07-tls-certs\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:43:32.297708 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:32.297722    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/659f338f-f44d-4f6e-8cb9-10e65f9cca07-dshm\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:43:32.594134 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:32.594100    2569 generic.go:356] "Generic (PLEG): container finished" podID="659f338f-f44d-4f6e-8cb9-10e65f9cca07" containerID="db4b41c8ceb390d9e5030205c09c7dacfff1467a352e8224731ae6d95e82a51c" exitCode=0
Sep 18 17:43:32.594303 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:32.594150    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t"
Sep 18 17:43:32.594303 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:32.594189    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t" event={"ID":"659f338f-f44d-4f6e-8cb9-10e65f9cca07","Type":"ContainerDied","Data":"db4b41c8ceb390d9e5030205c09c7dacfff1467a352e8224731ae6d95e82a51c"}
Sep 18 17:43:32.594303 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:32.594241    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t" event={"ID":"659f338f-f44d-4f6e-8cb9-10e65f9cca07","Type":"ContainerDied","Data":"303c5531f480daf9a8fd6cdca86fce2485ff439d071ef64f4cfd5a63fd98fd99"}
Sep 18 17:43:32.594303 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:32.594264    2569 scope.go:117] "RemoveContainer" containerID="db4b41c8ceb390d9e5030205c09c7dacfff1467a352e8224731ae6d95e82a51c"
Sep 18 17:43:32.603177 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:32.603155    2569 scope.go:117] "RemoveContainer" containerID="db4b41c8ceb390d9e5030205c09c7dacfff1467a352e8224731ae6d95e82a51c"
Sep 18 17:43:32.603462 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:43:32.603444    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db4b41c8ceb390d9e5030205c09c7dacfff1467a352e8224731ae6d95e82a51c\": container with ID starting with db4b41c8ceb390d9e5030205c09c7dacfff1467a352e8224731ae6d95e82a51c not found: ID does not exist" containerID="db4b41c8ceb390d9e5030205c09c7dacfff1467a352e8224731ae6d95e82a51c"
Sep 18 17:43:32.603509 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:32.603470    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db4b41c8ceb390d9e5030205c09c7dacfff1467a352e8224731ae6d95e82a51c"} err="failed to get container status \"db4b41c8ceb390d9e5030205c09c7dacfff1467a352e8224731ae6d95e82a51c\": rpc error: code = NotFound desc = could not find container \"db4b41c8ceb390d9e5030205c09c7dacfff1467a352e8224731ae6d95e82a51c\": container with ID starting with db4b41c8ceb390d9e5030205c09c7dacfff1467a352e8224731ae6d95e82a51c not found: ID does not exist"
Sep 18 17:43:32.624033 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:32.624004    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t"]
Sep 18 17:43:32.632956 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:32.632930    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-808470/sub-test-model-808470-kserve-56b4df9749-ddg7t"]
Sep 18 17:43:33.765602 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:33.765566    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="659f338f-f44d-4f6e-8cb9-10e65f9cca07" path="/var/lib/kubelet/pods/659f338f-f44d-4f6e-8cb9-10e65f9cca07/volumes"
Sep 18 17:43:38.630857 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:38.630823    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82"
Sep 18 17:43:38.631274 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:38.630883    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82"
Sep 18 17:43:38.643359 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:38.643330    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82"
Sep 18 17:43:39.642641 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:39.642607    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82"
Sep 18 17:43:47.698159 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:47.698128    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf"]
Sep 18 17:43:47.698632 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:47.698448    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf13fa17-9ec1-4e60-ab9f-2f392d7ada44" containerName="main"
Sep 18 17:43:47.698632 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:47.698461    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf13fa17-9ec1-4e60-ab9f-2f392d7ada44" containerName="main"
Sep 18 17:43:47.698632 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:47.698470    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="659f338f-f44d-4f6e-8cb9-10e65f9cca07" containerName="main"
Sep 18 17:43:47.698632 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:47.698476    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="659f338f-f44d-4f6e-8cb9-10e65f9cca07" containerName="main"
Sep 18 17:43:47.698632 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:47.698529    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="659f338f-f44d-4f6e-8cb9-10e65f9cca07" containerName="main"
Sep 18 17:43:47.698632 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:47.698539    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf13fa17-9ec1-4e60-ab9f-2f392d7ada44" containerName="main"
Sep 18 17:43:47.702731 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:47.702716    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf"
Sep 18 17:43:47.707917 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:47.707896    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-home\") pod \"rate-test-model-808470-kserve-f797948c-xfvmf\" (UID: \"a99e64a1-4253-4c32-99d9-ff2b8261b1fa\") " pod="ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf"
Sep 18 17:43:47.708031 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:47.707923    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpfb7\" (UniqueName: \"kubernetes.io/projected/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-kube-api-access-cpfb7\") pod \"rate-test-model-808470-kserve-f797948c-xfvmf\" (UID: \"a99e64a1-4253-4c32-99d9-ff2b8261b1fa\") " pod="ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf"
Sep 18 17:43:47.708031 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:47.707956    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-model-cache\") pod \"rate-test-model-808470-kserve-f797948c-xfvmf\" (UID: \"a99e64a1-4253-4c32-99d9-ff2b8261b1fa\") " pod="ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf"
Sep 18 17:43:47.708109 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:47.708042    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-tmp-dir\") pod \"rate-test-model-808470-kserve-f797948c-xfvmf\" (UID: \"a99e64a1-4253-4c32-99d9-ff2b8261b1fa\") " pod="ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf"
Sep 18 17:43:47.708156 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:47.708139    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-dshm\") pod \"rate-test-model-808470-kserve-f797948c-xfvmf\" (UID: \"a99e64a1-4253-4c32-99d9-ff2b8261b1fa\") " pod="ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf"
Sep 18 17:43:47.708225 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:47.708209    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-tls-certs\") pod \"rate-test-model-808470-kserve-f797948c-xfvmf\" (UID: \"a99e64a1-4253-4c32-99d9-ff2b8261b1fa\") " pod="ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf"
Sep 18 17:43:47.719096 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:47.719074    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-808470\"/\"kube-root-ca.crt\""
Sep 18 17:43:47.720348 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:47.720329    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-808470\"/\"rate-test-model-808470-kserve-self-signed-certs\""
Sep 18 17:43:47.721279 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:47.721263    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-808470\"/\"openshift-service-ca.crt\""
Sep 18 17:43:47.723041 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:47.723026    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-808470\"/\"default-dockercfg-j42zw\""
Sep 18 17:43:47.740959 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:47.740935    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf"]
Sep 18 17:43:47.808811 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:47.808781    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-dshm\") pod \"rate-test-model-808470-kserve-f797948c-xfvmf\" (UID: \"a99e64a1-4253-4c32-99d9-ff2b8261b1fa\") " pod="ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf"
Sep 18 17:43:47.808991 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:47.808831    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-tls-certs\") pod \"rate-test-model-808470-kserve-f797948c-xfvmf\" (UID: \"a99e64a1-4253-4c32-99d9-ff2b8261b1fa\") " pod="ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf"
Sep 18 17:43:47.808991 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:47.808857    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-home\") pod \"rate-test-model-808470-kserve-f797948c-xfvmf\" (UID: \"a99e64a1-4253-4c32-99d9-ff2b8261b1fa\") " pod="ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf"
Sep 18 17:43:47.808991 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:47.808895    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpfb7\" (UniqueName: \"kubernetes.io/projected/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-kube-api-access-cpfb7\") pod \"rate-test-model-808470-kserve-f797948c-xfvmf\" (UID: \"a99e64a1-4253-4c32-99d9-ff2b8261b1fa\") " pod="ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf"
Sep 18 17:43:47.808991 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:47.808922    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-model-cache\") pod \"rate-test-model-808470-kserve-f797948c-xfvmf\" (UID: \"a99e64a1-4253-4c32-99d9-ff2b8261b1fa\") " pod="ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf"
Sep 18 17:43:47.808991 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:47.808942    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-tmp-dir\") pod \"rate-test-model-808470-kserve-f797948c-xfvmf\" (UID: \"a99e64a1-4253-4c32-99d9-ff2b8261b1fa\") " pod="ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf"
Sep 18 17:43:47.809366 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:47.809343    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-home\") pod \"rate-test-model-808470-kserve-f797948c-xfvmf\" (UID: \"a99e64a1-4253-4c32-99d9-ff2b8261b1fa\") " pod="ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf"
Sep 18 17:43:47.809443 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:47.809370    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-model-cache\") pod \"rate-test-model-808470-kserve-f797948c-xfvmf\" (UID: \"a99e64a1-4253-4c32-99d9-ff2b8261b1fa\") " pod="ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf"
Sep 18 17:43:47.809443 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:47.809401    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-tmp-dir\") pod \"rate-test-model-808470-kserve-f797948c-xfvmf\" (UID: \"a99e64a1-4253-4c32-99d9-ff2b8261b1fa\") " pod="ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf"
Sep 18 17:43:47.810983 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:47.810962    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-dshm\") pod \"rate-test-model-808470-kserve-f797948c-xfvmf\" (UID: \"a99e64a1-4253-4c32-99d9-ff2b8261b1fa\") " pod="ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf"
Sep 18 17:43:47.811290 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:47.811272    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-tls-certs\") pod \"rate-test-model-808470-kserve-f797948c-xfvmf\" (UID: \"a99e64a1-4253-4c32-99d9-ff2b8261b1fa\") " pod="ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf"
Sep 18 17:43:47.827418 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:47.827395    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpfb7\" (UniqueName: \"kubernetes.io/projected/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-kube-api-access-cpfb7\") pod \"rate-test-model-808470-kserve-f797948c-xfvmf\" (UID: \"a99e64a1-4253-4c32-99d9-ff2b8261b1fa\") " pod="ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf"
Sep 18 17:43:48.011745 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:48.011701    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf"
Sep 18 17:43:48.299009 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:48.298981    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf"]
Sep 18 17:43:48.300207 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:43:48.300181    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda99e64a1_4253_4c32_99d9_ff2b8261b1fa.slice/crio-a828ff77a7847fb1567bf09a0c9e4614fd513c1857811b62541adc182502e030 WatchSource:0}: Error finding container a828ff77a7847fb1567bf09a0c9e4614fd513c1857811b62541adc182502e030: Status 404 returned error can't find the container with id a828ff77a7847fb1567bf09a0c9e4614fd513c1857811b62541adc182502e030
Sep 18 17:43:48.676605 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:48.676520    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf" event={"ID":"a99e64a1-4253-4c32-99d9-ff2b8261b1fa","Type":"ContainerStarted","Data":"1d60fd568c0d1a63a82784d9499b92d6495c29021f855ce1439a67dac998f43c"}
Sep 18 17:43:48.676605 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:48.676565    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf" event={"ID":"a99e64a1-4253-4c32-99d9-ff2b8261b1fa","Type":"ContainerStarted","Data":"a828ff77a7847fb1567bf09a0c9e4614fd513c1857811b62541adc182502e030"}
Sep 18 17:43:48.714005 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:48.713950    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf" podStartSLOduration=1.713935604 podStartE2EDuration="1.713935604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:43:48.711670397 +0000 UTC m=+1741.405597925" watchObservedRunningTime="2026-09-18 17:43:48.713935604 +0000 UTC m=+1741.407863219"
Sep 18 17:43:58.012168 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:58.012110    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf"
Sep 18 17:43:58.012168 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:58.012182    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf"
Sep 18 17:43:58.024705 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:58.024681    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf"
Sep 18 17:43:58.738130 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:43:58.738097    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf"
Sep 18 17:44:51.320152 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:51.320116    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82"]
Sep 18 17:44:52.417476 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:52.417436    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf"]
Sep 18 17:44:58.398936 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:58.398903    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv"]
Sep 18 17:44:58.402319 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:58.402297    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv"
Sep 18 17:44:58.405884 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:58.405832    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-828a86-serving-cert\""
Sep 18 17:44:58.406037 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:58.405856    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 18 17:44:58.406037 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:58.405933    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sjtq4\""
Sep 18 17:44:58.406037 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:58.405855    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 17:44:58.406037 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:58.405940    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 17:44:58.413630 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:58.413609    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv"]
Sep 18 17:44:58.546163 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:58.546113    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69a7f0df-3f15-4367-91ff-79f0267c7ac4-metrics-certs\") pod \"maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv\" (UID: \"69a7f0df-3f15-4367-91ff-79f0267c7ac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv"
Sep 18 17:44:58.546372 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:58.546197    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69a7f0df-3f15-4367-91ff-79f0267c7ac4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv\" (UID: \"69a7f0df-3f15-4367-91ff-79f0267c7ac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv"
Sep 18 17:44:58.546372 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:58.546237    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9644\" (UniqueName: \"kubernetes.io/projected/69a7f0df-3f15-4367-91ff-79f0267c7ac4-kube-api-access-t9644\") pod \"maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv\" (UID: \"69a7f0df-3f15-4367-91ff-79f0267c7ac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv"
Sep 18 17:44:58.546372 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:58.546321    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69a7f0df-3f15-4367-91ff-79f0267c7ac4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv\" (UID: \"69a7f0df-3f15-4367-91ff-79f0267c7ac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv"
Sep 18 17:44:58.647605 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:58.647568    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69a7f0df-3f15-4367-91ff-79f0267c7ac4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv\" (UID: \"69a7f0df-3f15-4367-91ff-79f0267c7ac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv"
Sep 18 17:44:58.647813 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:58.647654    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69a7f0df-3f15-4367-91ff-79f0267c7ac4-metrics-certs\") pod \"maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv\" (UID: \"69a7f0df-3f15-4367-91ff-79f0267c7ac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv"
Sep 18 17:44:58.647813 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:58.647712    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69a7f0df-3f15-4367-91ff-79f0267c7ac4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv\" (UID: \"69a7f0df-3f15-4367-91ff-79f0267c7ac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv"
Sep 18 17:44:58.647813 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:58.647749    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9644\" (UniqueName: \"kubernetes.io/projected/69a7f0df-3f15-4367-91ff-79f0267c7ac4-kube-api-access-t9644\") pod \"maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv\" (UID: \"69a7f0df-3f15-4367-91ff-79f0267c7ac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv"
Sep 18 17:44:58.648268 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:58.648246    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69a7f0df-3f15-4367-91ff-79f0267c7ac4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv\" (UID: \"69a7f0df-3f15-4367-91ff-79f0267c7ac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv"
Sep 18 17:44:58.650230 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:58.650186    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69a7f0df-3f15-4367-91ff-79f0267c7ac4-metrics-certs\") pod \"maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv\" (UID: \"69a7f0df-3f15-4367-91ff-79f0267c7ac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv"
Sep 18 17:44:58.650299 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:58.650233    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69a7f0df-3f15-4367-91ff-79f0267c7ac4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv\" (UID: \"69a7f0df-3f15-4367-91ff-79f0267c7ac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv"
Sep 18 17:44:58.658300 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:58.658277    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9644\" (UniqueName: \"kubernetes.io/projected/69a7f0df-3f15-4367-91ff-79f0267c7ac4-kube-api-access-t9644\") pod \"maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv\" (UID: \"69a7f0df-3f15-4367-91ff-79f0267c7ac4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv"
Sep 18 17:44:58.713332 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:58.713298    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv"
Sep 18 17:44:58.994075 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:58.994045    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv"]
Sep 18 17:44:58.996690 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:44:58.996661    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a7f0df_3f15_4367_91ff_79f0267c7ac4.slice/crio-da120982d7f5ea75dcabe3963526dbcf3cc4017dfd1b60e8fb5dd3de6175a827 WatchSource:0}: Error finding container da120982d7f5ea75dcabe3963526dbcf3cc4017dfd1b60e8fb5dd3de6175a827: Status 404 returned error can't find the container with id da120982d7f5ea75dcabe3963526dbcf3cc4017dfd1b60e8fb5dd3de6175a827
Sep 18 17:44:58.998085 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:58.998062    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 17:44:59.023399 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:59.023361    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv" event={"ID":"69a7f0df-3f15-4367-91ff-79f0267c7ac4","Type":"ContainerStarted","Data":"da120982d7f5ea75dcabe3963526dbcf3cc4017dfd1b60e8fb5dd3de6175a827"}
Sep 18 17:44:59.058621 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:59.058580    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-828a86-59d7877675-x6fdt"]
Sep 18 17:44:59.063290 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:59.063269    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-828a86-59d7877675-x6fdt"
Sep 18 17:44:59.067152 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:59.067131    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-828a86\""
Sep 18 17:44:59.071849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:59.071828    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-828a86-dockercfg-f2fxc\""
Sep 18 17:44:59.077912 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:59.077884    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-828a86-59d7877675-x6fdt"]
Sep 18 17:44:59.152032 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:59.151990    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f0a29e-9531-4b07-a8dd-41cafe6c74d6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-828a86-59d7877675-x6fdt\" (UID: \"58f0a29e-9531-4b07-a8dd-41cafe6c74d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-828a86-59d7877675-x6fdt"
Sep 18 17:44:59.152216 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:59.152037    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xgh5\" (UniqueName: \"kubernetes.io/projected/58f0a29e-9531-4b07-a8dd-41cafe6c74d6-kube-api-access-6xgh5\") pod \"payload-processing-e2e-ipp-a-828a86-59d7877675-x6fdt\" (UID: \"58f0a29e-9531-4b07-a8dd-41cafe6c74d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-828a86-59d7877675-x6fdt"
Sep 18 17:44:59.253343 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:59.253243    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f0a29e-9531-4b07-a8dd-41cafe6c74d6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-828a86-59d7877675-x6fdt\" (UID: \"58f0a29e-9531-4b07-a8dd-41cafe6c74d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-828a86-59d7877675-x6fdt"
Sep 18 17:44:59.253343 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:59.253295    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xgh5\" (UniqueName: \"kubernetes.io/projected/58f0a29e-9531-4b07-a8dd-41cafe6c74d6-kube-api-access-6xgh5\") pod \"payload-processing-e2e-ipp-a-828a86-59d7877675-x6fdt\" (UID: \"58f0a29e-9531-4b07-a8dd-41cafe6c74d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-828a86-59d7877675-x6fdt"
Sep 18 17:44:59.253979 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:59.253959    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f0a29e-9531-4b07-a8dd-41cafe6c74d6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-828a86-59d7877675-x6fdt\" (UID: \"58f0a29e-9531-4b07-a8dd-41cafe6c74d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-828a86-59d7877675-x6fdt"
Sep 18 17:44:59.264662 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:59.264642    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xgh5\" (UniqueName: \"kubernetes.io/projected/58f0a29e-9531-4b07-a8dd-41cafe6c74d6-kube-api-access-6xgh5\") pod \"payload-processing-e2e-ipp-a-828a86-59d7877675-x6fdt\" (UID: \"58f0a29e-9531-4b07-a8dd-41cafe6c74d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-828a86-59d7877675-x6fdt"
Sep 18 17:44:59.373979 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:59.373941    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-828a86-59d7877675-x6fdt"
Sep 18 17:44:59.646608 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:44:59.646576    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-828a86-59d7877675-x6fdt"]
Sep 18 17:44:59.650127 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:44:59.650102    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58f0a29e_9531_4b07_a8dd_41cafe6c74d6.slice/crio-aae5c176b92707ab9c4450553cfee89cb7234ec12cefa4fc80605d2c50212954 WatchSource:0}: Error finding container aae5c176b92707ab9c4450553cfee89cb7234ec12cefa4fc80605d2c50212954: Status 404 returned error can't find the container with id aae5c176b92707ab9c4450553cfee89cb7234ec12cefa4fc80605d2c50212954
Sep 18 17:45:00.034136 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:00.034097    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv" event={"ID":"69a7f0df-3f15-4367-91ff-79f0267c7ac4","Type":"ContainerStarted","Data":"09f502a6cee891ed2656a0111306c113c17871d0cdc19035bbbd1faa9e5cb37f"}
Sep 18 17:45:00.034379 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:00.034275    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv"
Sep 18 17:45:00.035894 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:00.035854    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-828a86-59d7877675-x6fdt" event={"ID":"58f0a29e-9531-4b07-a8dd-41cafe6c74d6","Type":"ContainerStarted","Data":"4bb2231738ae1f0c572889ba0b8c7eec4bb0e93e079c98c723900d3bb8765020"}
Sep 18 17:45:00.036036 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:00.035908    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-828a86-59d7877675-x6fdt" event={"ID":"58f0a29e-9531-4b07-a8dd-41cafe6c74d6","Type":"ContainerStarted","Data":"aae5c176b92707ab9c4450553cfee89cb7234ec12cefa4fc80605d2c50212954"}
Sep 18 17:45:00.036036 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:00.035943    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-828a86-59d7877675-x6fdt"
Sep 18 17:45:00.058962 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:00.058913    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv" podStartSLOduration=1.592464882 podStartE2EDuration="2.058897271s" totalImagesPullingTime="466.432389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:44:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:45:00.05674429 +0000 UTC m=+1812.750671818" watchObservedRunningTime="2026-09-18 17:45:00.058897271 +0000 UTC m=+1812.752824823"
Sep 18 17:45:00.079741 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:00.079697    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-828a86-59d7877675-x6fdt" podStartSLOduration=1.079682761 podStartE2EDuration="1.079682761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:44:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:45:00.078928796 +0000 UTC m=+1812.772856351" watchObservedRunningTime="2026-09-18 17:45:00.079682761 +0000 UTC m=+1812.773610289"
Sep 18 17:45:00.149250 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:00.149214    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829225-gffbg"]
Sep 18 17:45:00.152529 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:00.152513    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829225-gffbg"
Sep 18 17:45:00.158632 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:00.158613    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qt978\""
Sep 18 17:45:00.163483 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:00.160811    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-828a86-29829225-cchfc"]
Sep 18 17:45:00.165718 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:00.165694    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-828a86-29829225-cchfc"
Sep 18 17:45:00.170913 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:00.170891    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829225-gffbg"
Sep 18 17:45:00.179263 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:00.177683    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829225-gffbg"]
Sep 18 17:45:00.184360 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:00.183705    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-828a86-29829225-cchfc"
Sep 18 17:45:00.185822 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:00.185062    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-828a86-29829225-cchfc"]
Sep 18 17:45:00.464983 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:00.464913    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829225-gffbg"]
Sep 18 17:45:00.468506 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:45:00.468483    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod763b5e7b_e33e_4163_9e71_2d25bfba9634.slice/crio-b649b1d2b6d68e0390e564d6af9d48a7275df4e0263240092cbf635b630d5cbc WatchSource:0}: Error finding container b649b1d2b6d68e0390e564d6af9d48a7275df4e0263240092cbf635b630d5cbc: Status 404 returned error can't find the container with id b649b1d2b6d68e0390e564d6af9d48a7275df4e0263240092cbf635b630d5cbc
Sep 18 17:45:00.487525 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:00.487500    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-828a86-29829225-cchfc"]
Sep 18 17:45:00.488527 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:45:00.488503    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04338d36_1035_418a_8dc4_7ce131103557.slice/crio-483853e03ab7b983cac9c2915805633b1a2c4b6e82bb1668c070a1c355b6f4e4 WatchSource:0}: Error finding container 483853e03ab7b983cac9c2915805633b1a2c4b6e82bb1668c070a1c355b6f4e4: Status 404 returned error can't find the container with id 483853e03ab7b983cac9c2915805633b1a2c4b6e82bb1668c070a1c355b6f4e4
Sep 18 17:45:01.042572 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:01.042470    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-828a86-29829225-cchfc" event={"ID":"04338d36-1035-418a-8dc4-7ce131103557","Type":"ContainerStarted","Data":"ebbaba7c202d394032e44cc78816550610f2bf3f62125662e31ca97015abf050"}
Sep 18 17:45:01.042572 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:01.042534    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-828a86-29829225-cchfc" event={"ID":"04338d36-1035-418a-8dc4-7ce131103557","Type":"ContainerStarted","Data":"483853e03ab7b983cac9c2915805633b1a2c4b6e82bb1668c070a1c355b6f4e4"}
Sep 18 17:45:01.044357 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:01.044325    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829225-gffbg" event={"ID":"763b5e7b-e33e-4163-9e71-2d25bfba9634","Type":"ContainerStarted","Data":"cc0bcdeec979dea17e6865a9f869b69db7e11bdf915962847a0e1d3ec0320335"}
Sep 18 17:45:01.044516 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:01.044376    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829225-gffbg" event={"ID":"763b5e7b-e33e-4163-9e71-2d25bfba9634","Type":"ContainerStarted","Data":"b649b1d2b6d68e0390e564d6af9d48a7275df4e0263240092cbf635b630d5cbc"}
Sep 18 17:45:01.064713 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:01.064651    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-828a86-29829225-cchfc" podStartSLOduration=1.064636136 podStartE2EDuration="1.064636136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:45:01.063248776 +0000 UTC m=+1813.757176329" watchObservedRunningTime="2026-09-18 17:45:01.064636136 +0000 UTC m=+1813.758563663"
Sep 18 17:45:01.080581 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:01.080541    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829225-gffbg" podStartSLOduration=1.08052671 podStartE2EDuration="1.08052671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:45:01.079651455 +0000 UTC m=+1813.773579007" watchObservedRunningTime="2026-09-18 17:45:01.08052671 +0000 UTC m=+1813.774454238"
Sep 18 17:45:02.050380 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:02.050349    2569 generic.go:356] "Generic (PLEG): container finished" podID="763b5e7b-e33e-4163-9e71-2d25bfba9634" containerID="cc0bcdeec979dea17e6865a9f869b69db7e11bdf915962847a0e1d3ec0320335" exitCode=0
Sep 18 17:45:02.050785 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:02.050445    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829225-gffbg" event={"ID":"763b5e7b-e33e-4163-9e71-2d25bfba9634","Type":"ContainerDied","Data":"cc0bcdeec979dea17e6865a9f869b69db7e11bdf915962847a0e1d3ec0320335"}
Sep 18 17:45:02.051913 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:02.051888    2569 generic.go:356] "Generic (PLEG): container finished" podID="04338d36-1035-418a-8dc4-7ce131103557" containerID="ebbaba7c202d394032e44cc78816550610f2bf3f62125662e31ca97015abf050" exitCode=7
Sep 18 17:45:02.052004 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:02.051950    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-828a86-29829225-cchfc" event={"ID":"04338d36-1035-418a-8dc4-7ce131103557","Type":"ContainerDied","Data":"ebbaba7c202d394032e44cc78816550610f2bf3f62125662e31ca97015abf050"}
Sep 18 17:45:02.052075 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:02.052061    2569 scope.go:117] "RemoveContainer" containerID="ebbaba7c202d394032e44cc78816550610f2bf3f62125662e31ca97015abf050"
Sep 18 17:45:03.059454 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:03.059420    2569 generic.go:356] "Generic (PLEG): container finished" podID="04338d36-1035-418a-8dc4-7ce131103557" containerID="4a81b9157108eb24585ff3bbb855612d17cb6d44c1229be19bb4c48e29c8f88c" exitCode=7
Sep 18 17:45:03.059906 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:03.059472    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-828a86-29829225-cchfc" event={"ID":"04338d36-1035-418a-8dc4-7ce131103557","Type":"ContainerDied","Data":"4a81b9157108eb24585ff3bbb855612d17cb6d44c1229be19bb4c48e29c8f88c"}
Sep 18 17:45:03.059906 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:03.059546    2569 scope.go:117] "RemoveContainer" containerID="ebbaba7c202d394032e44cc78816550610f2bf3f62125662e31ca97015abf050"
Sep 18 17:45:03.059906 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:03.059697    2569 scope.go:117] "RemoveContainer" containerID="4a81b9157108eb24585ff3bbb855612d17cb6d44c1229be19bb4c48e29c8f88c"
Sep 18 17:45:03.060100 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:45:03.059972    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-828a86-29829225-cchfc_odh-ai-gateway-infra(04338d36-1035-418a-8dc4-7ce131103557)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-828a86-29829225-cchfc" podUID="04338d36-1035-418a-8dc4-7ce131103557"
Sep 18 17:45:03.197755 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:03.197726    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829225-gffbg"
Sep 18 17:45:04.065925 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:04.065809    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829225-gffbg"
Sep 18 17:45:04.065925 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:04.065805    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829225-gffbg" event={"ID":"763b5e7b-e33e-4163-9e71-2d25bfba9634","Type":"ContainerDied","Data":"b649b1d2b6d68e0390e564d6af9d48a7275df4e0263240092cbf635b630d5cbc"}
Sep 18 17:45:04.066426 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:04.065966    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b649b1d2b6d68e0390e564d6af9d48a7275df4e0263240092cbf635b630d5cbc"
Sep 18 17:45:04.067659 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:04.067637    2569 scope.go:117] "RemoveContainer" containerID="4a81b9157108eb24585ff3bbb855612d17cb6d44c1229be19bb4c48e29c8f88c"
Sep 18 17:45:04.067944 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:45:04.067917    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-828a86-29829225-cchfc_odh-ai-gateway-infra(04338d36-1035-418a-8dc4-7ce131103557)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-828a86-29829225-cchfc" podUID="04338d36-1035-418a-8dc4-7ce131103557"
Sep 18 17:45:06.050008 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:06.049976    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv"
Sep 18 17:45:06.341066 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:06.340967    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82" podUID="a07ab377-db7a-4873-81c6-eada976206d7" containerName="main" containerID="cri-o://e72da8e73b1380d1778d6686daf4973ac6eee13e459bd52abe2dc4c100ddb5c6" gracePeriod=45
Sep 18 17:45:06.593154 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:06.593100    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82"
Sep 18 17:45:06.736696 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:06.736649    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a07ab377-db7a-4873-81c6-eada976206d7-tls-certs\") pod \"a07ab377-db7a-4873-81c6-eada976206d7\" (UID: \"a07ab377-db7a-4873-81c6-eada976206d7\") "
Sep 18 17:45:06.736696 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:06.736703    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vv9p8\" (UniqueName: \"kubernetes.io/projected/a07ab377-db7a-4873-81c6-eada976206d7-kube-api-access-vv9p8\") pod \"a07ab377-db7a-4873-81c6-eada976206d7\" (UID: \"a07ab377-db7a-4873-81c6-eada976206d7\") "
Sep 18 17:45:06.736983 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:06.736732    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a07ab377-db7a-4873-81c6-eada976206d7-tmp-dir\") pod \"a07ab377-db7a-4873-81c6-eada976206d7\" (UID: \"a07ab377-db7a-4873-81c6-eada976206d7\") "
Sep 18 17:45:06.736983 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:06.736856    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a07ab377-db7a-4873-81c6-eada976206d7-dshm\") pod \"a07ab377-db7a-4873-81c6-eada976206d7\" (UID: \"a07ab377-db7a-4873-81c6-eada976206d7\") "
Sep 18 17:45:06.736983 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:06.736916    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a07ab377-db7a-4873-81c6-eada976206d7-model-cache\") pod \"a07ab377-db7a-4873-81c6-eada976206d7\" (UID: \"a07ab377-db7a-4873-81c6-eada976206d7\") "
Sep 18 17:45:06.736983 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:06.736940    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a07ab377-db7a-4873-81c6-eada976206d7-home\") pod \"a07ab377-db7a-4873-81c6-eada976206d7\" (UID: \"a07ab377-db7a-4873-81c6-eada976206d7\") "
Sep 18 17:45:06.737194 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:06.737116    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07ab377-db7a-4873-81c6-eada976206d7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a07ab377-db7a-4873-81c6-eada976206d7" (UID: "a07ab377-db7a-4873-81c6-eada976206d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:45:06.737252 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:06.737204    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07ab377-db7a-4873-81c6-eada976206d7-model-cache" (OuterVolumeSpecName: "model-cache") pod "a07ab377-db7a-4873-81c6-eada976206d7" (UID: "a07ab377-db7a-4873-81c6-eada976206d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:45:06.737252 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:06.737220    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07ab377-db7a-4873-81c6-eada976206d7-home" (OuterVolumeSpecName: "home") pod "a07ab377-db7a-4873-81c6-eada976206d7" (UID: "a07ab377-db7a-4873-81c6-eada976206d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:45:06.737344 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:06.737288    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a07ab377-db7a-4873-81c6-eada976206d7-tmp-dir\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:45:06.738959 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:06.738927    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07ab377-db7a-4873-81c6-eada976206d7-dshm" (OuterVolumeSpecName: "dshm") pod "a07ab377-db7a-4873-81c6-eada976206d7" (UID: "a07ab377-db7a-4873-81c6-eada976206d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:45:06.739053 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:06.738965    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a07ab377-db7a-4873-81c6-eada976206d7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a07ab377-db7a-4873-81c6-eada976206d7" (UID: "a07ab377-db7a-4873-81c6-eada976206d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:45:06.739053 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:06.739018    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07ab377-db7a-4873-81c6-eada976206d7-kube-api-access-vv9p8" (OuterVolumeSpecName: "kube-api-access-vv9p8") pod "a07ab377-db7a-4873-81c6-eada976206d7" (UID: "a07ab377-db7a-4873-81c6-eada976206d7"). InnerVolumeSpecName "kube-api-access-vv9p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:45:06.838319 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:06.838274    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a07ab377-db7a-4873-81c6-eada976206d7-dshm\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:45:06.838319 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:06.838313    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a07ab377-db7a-4873-81c6-eada976206d7-model-cache\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:45:06.838319 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:06.838324    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a07ab377-db7a-4873-81c6-eada976206d7-home\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:45:06.838319 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:06.838332    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a07ab377-db7a-4873-81c6-eada976206d7-tls-certs\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:45:06.838586 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:06.838341    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv9p8\" (UniqueName: \"kubernetes.io/projected/a07ab377-db7a-4873-81c6-eada976206d7-kube-api-access-vv9p8\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:45:07.086665 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:07.086628    2569 generic.go:356] "Generic (PLEG): container finished" podID="a07ab377-db7a-4873-81c6-eada976206d7" containerID="e72da8e73b1380d1778d6686daf4973ac6eee13e459bd52abe2dc4c100ddb5c6" exitCode=0
Sep 18 17:45:07.087201 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:07.086674    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82"
Sep 18 17:45:07.087201 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:07.086690    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82" event={"ID":"a07ab377-db7a-4873-81c6-eada976206d7","Type":"ContainerDied","Data":"e72da8e73b1380d1778d6686daf4973ac6eee13e459bd52abe2dc4c100ddb5c6"}
Sep 18 17:45:07.087201 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:07.086723    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82" event={"ID":"a07ab377-db7a-4873-81c6-eada976206d7","Type":"ContainerDied","Data":"48a2a183599aa5c417f84a59a30209e8fc93f8f562a6b284d43ec48b6589f88e"}
Sep 18 17:45:07.087201 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:07.086739    2569 scope.go:117] "RemoveContainer" containerID="e72da8e73b1380d1778d6686daf4973ac6eee13e459bd52abe2dc4c100ddb5c6"
Sep 18 17:45:07.096176 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:07.096138    2569 scope.go:117] "RemoveContainer" containerID="e72da8e73b1380d1778d6686daf4973ac6eee13e459bd52abe2dc4c100ddb5c6"
Sep 18 17:45:07.096523 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:45:07.096494    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e72da8e73b1380d1778d6686daf4973ac6eee13e459bd52abe2dc4c100ddb5c6\": container with ID starting with e72da8e73b1380d1778d6686daf4973ac6eee13e459bd52abe2dc4c100ddb5c6 not found: ID does not exist" containerID="e72da8e73b1380d1778d6686daf4973ac6eee13e459bd52abe2dc4c100ddb5c6"
Sep 18 17:45:07.096608 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:07.096531    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e72da8e73b1380d1778d6686daf4973ac6eee13e459bd52abe2dc4c100ddb5c6"} err="failed to get container status \"e72da8e73b1380d1778d6686daf4973ac6eee13e459bd52abe2dc4c100ddb5c6\": rpc error: code = NotFound desc = could not find container \"e72da8e73b1380d1778d6686daf4973ac6eee13e459bd52abe2dc4c100ddb5c6\": container with ID starting with e72da8e73b1380d1778d6686daf4973ac6eee13e459bd52abe2dc4c100ddb5c6 not found: ID does not exist"
Sep 18 17:45:07.108628 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:07.108602    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82"]
Sep 18 17:45:07.114537 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:07.114516    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-eeab6e/rate-test-model-eeab6e-kserve-667bd4b775-d2c82"]
Sep 18 17:45:07.436019 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:07.435933    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf" podUID="a99e64a1-4253-4c32-99d9-ff2b8261b1fa" containerName="main" containerID="cri-o://1d60fd568c0d1a63a82784d9499b92d6495c29021f855ce1439a67dac998f43c" gracePeriod=45
Sep 18 17:45:07.685195 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:07.685167    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf"
Sep 18 17:45:07.764799 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:07.764767    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a07ab377-db7a-4873-81c6-eada976206d7" path="/var/lib/kubelet/pods/a07ab377-db7a-4873-81c6-eada976206d7/volumes"
Sep 18 17:45:07.847905 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:07.847849    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-tls-certs\") pod \"a99e64a1-4253-4c32-99d9-ff2b8261b1fa\" (UID: \"a99e64a1-4253-4c32-99d9-ff2b8261b1fa\") "
Sep 18 17:45:07.848080 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:07.847912    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-model-cache\") pod \"a99e64a1-4253-4c32-99d9-ff2b8261b1fa\" (UID: \"a99e64a1-4253-4c32-99d9-ff2b8261b1fa\") "
Sep 18 17:45:07.848080 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:07.847971    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpfb7\" (UniqueName: \"kubernetes.io/projected/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-kube-api-access-cpfb7\") pod \"a99e64a1-4253-4c32-99d9-ff2b8261b1fa\" (UID: \"a99e64a1-4253-4c32-99d9-ff2b8261b1fa\") "
Sep 18 17:45:07.848080 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:07.848000    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-dshm\") pod \"a99e64a1-4253-4c32-99d9-ff2b8261b1fa\" (UID: \"a99e64a1-4253-4c32-99d9-ff2b8261b1fa\") "
Sep 18 17:45:07.848080 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:07.848040    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-tmp-dir\") pod \"a99e64a1-4253-4c32-99d9-ff2b8261b1fa\" (UID: \"a99e64a1-4253-4c32-99d9-ff2b8261b1fa\") "
Sep 18 17:45:07.848080 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:07.848075    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-home\") pod \"a99e64a1-4253-4c32-99d9-ff2b8261b1fa\" (UID: \"a99e64a1-4253-4c32-99d9-ff2b8261b1fa\") "
Sep 18 17:45:07.848343 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:07.848235    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-model-cache" (OuterVolumeSpecName: "model-cache") pod "a99e64a1-4253-4c32-99d9-ff2b8261b1fa" (UID: "a99e64a1-4253-4c32-99d9-ff2b8261b1fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:45:07.848406 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:07.848344    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a99e64a1-4253-4c32-99d9-ff2b8261b1fa" (UID: "a99e64a1-4253-4c32-99d9-ff2b8261b1fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:45:07.848406 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:07.848375    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-model-cache\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:45:07.848406 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:07.848381    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-home" (OuterVolumeSpecName: "home") pod "a99e64a1-4253-4c32-99d9-ff2b8261b1fa" (UID: "a99e64a1-4253-4c32-99d9-ff2b8261b1fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:45:07.850059 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:07.850025    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-dshm" (OuterVolumeSpecName: "dshm") pod "a99e64a1-4253-4c32-99d9-ff2b8261b1fa" (UID: "a99e64a1-4253-4c32-99d9-ff2b8261b1fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:45:07.850174 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:07.850072    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-kube-api-access-cpfb7" (OuterVolumeSpecName: "kube-api-access-cpfb7") pod "a99e64a1-4253-4c32-99d9-ff2b8261b1fa" (UID: "a99e64a1-4253-4c32-99d9-ff2b8261b1fa"). InnerVolumeSpecName "kube-api-access-cpfb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:45:07.850174 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:07.850087    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a99e64a1-4253-4c32-99d9-ff2b8261b1fa" (UID: "a99e64a1-4253-4c32-99d9-ff2b8261b1fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:45:07.948973 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:07.948933    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpfb7\" (UniqueName: \"kubernetes.io/projected/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-kube-api-access-cpfb7\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:45:07.948973 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:07.948962    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-dshm\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:45:07.948973 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:07.948972    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-tmp-dir\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:45:07.948973 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:07.948982    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-home\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:45:07.949303 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:07.948990    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a99e64a1-4253-4c32-99d9-ff2b8261b1fa-tls-certs\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:45:08.093183 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:08.093096    2569 generic.go:356] "Generic (PLEG): container finished" podID="a99e64a1-4253-4c32-99d9-ff2b8261b1fa" containerID="1d60fd568c0d1a63a82784d9499b92d6495c29021f855ce1439a67dac998f43c" exitCode=0
Sep 18 17:45:08.093183 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:08.093138    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf"
Sep 18 17:45:08.093698 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:08.093178    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf" event={"ID":"a99e64a1-4253-4c32-99d9-ff2b8261b1fa","Type":"ContainerDied","Data":"1d60fd568c0d1a63a82784d9499b92d6495c29021f855ce1439a67dac998f43c"}
Sep 18 17:45:08.093698 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:08.093224    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf" event={"ID":"a99e64a1-4253-4c32-99d9-ff2b8261b1fa","Type":"ContainerDied","Data":"a828ff77a7847fb1567bf09a0c9e4614fd513c1857811b62541adc182502e030"}
Sep 18 17:45:08.093698 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:08.093245    2569 scope.go:117] "RemoveContainer" containerID="1d60fd568c0d1a63a82784d9499b92d6495c29021f855ce1439a67dac998f43c"
Sep 18 17:45:08.102723 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:08.102705    2569 scope.go:117] "RemoveContainer" containerID="1d60fd568c0d1a63a82784d9499b92d6495c29021f855ce1439a67dac998f43c"
Sep 18 17:45:08.103008 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:45:08.102987    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d60fd568c0d1a63a82784d9499b92d6495c29021f855ce1439a67dac998f43c\": container with ID starting with 1d60fd568c0d1a63a82784d9499b92d6495c29021f855ce1439a67dac998f43c not found: ID does not exist" containerID="1d60fd568c0d1a63a82784d9499b92d6495c29021f855ce1439a67dac998f43c"
Sep 18 17:45:08.103087 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:08.103021    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d60fd568c0d1a63a82784d9499b92d6495c29021f855ce1439a67dac998f43c"} err="failed to get container status \"1d60fd568c0d1a63a82784d9499b92d6495c29021f855ce1439a67dac998f43c\": rpc error: code = NotFound desc = could not find container \"1d60fd568c0d1a63a82784d9499b92d6495c29021f855ce1439a67dac998f43c\": container with ID starting with 1d60fd568c0d1a63a82784d9499b92d6495c29021f855ce1439a67dac998f43c not found: ID does not exist"
Sep 18 17:45:08.114467 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:08.114442    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf"]
Sep 18 17:45:08.118941 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:08.118919    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-808470/rate-test-model-808470-kserve-f797948c-xfvmf"]
Sep 18 17:45:09.764125 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:09.764094    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a99e64a1-4253-4c32-99d9-ff2b8261b1fa" path="/var/lib/kubelet/pods/a99e64a1-4253-4c32-99d9-ff2b8261b1fa/volumes"
Sep 18 17:45:11.046922 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:11.046884    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-828a86-59d7877675-x6fdt"
Sep 18 17:45:15.758836 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:15.758791    2569 scope.go:117] "RemoveContainer" containerID="4a81b9157108eb24585ff3bbb855612d17cb6d44c1229be19bb4c48e29c8f88c"
Sep 18 17:45:16.135616 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:16.135536    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-828a86-29829225-cchfc" event={"ID":"04338d36-1035-418a-8dc4-7ce131103557","Type":"ContainerStarted","Data":"d65628d111b7f80a002c0455308fc45562ce4d8e8213a60283fffa4b422e543e"}
Sep 18 17:45:17.142226 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:17.142196    2569 generic.go:356] "Generic (PLEG): container finished" podID="04338d36-1035-418a-8dc4-7ce131103557" containerID="d65628d111b7f80a002c0455308fc45562ce4d8e8213a60283fffa4b422e543e" exitCode=0
Sep 18 17:45:17.142638 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:17.142249    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-828a86-29829225-cchfc" event={"ID":"04338d36-1035-418a-8dc4-7ce131103557","Type":"ContainerDied","Data":"d65628d111b7f80a002c0455308fc45562ce4d8e8213a60283fffa4b422e543e"}
Sep 18 17:45:17.142638 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:17.142296    2569 scope.go:117] "RemoveContainer" containerID="4a81b9157108eb24585ff3bbb855612d17cb6d44c1229be19bb4c48e29c8f88c"
Sep 18 17:45:17.171223 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:17.171194    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-828a86-29829225-cchfc"]
Sep 18 17:45:18.282151 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:18.282128    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-828a86-29829225-cchfc"
Sep 18 17:45:19.155478 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:19.155447    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-828a86-29829225-cchfc"
Sep 18 17:45:19.155632 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:19.155490    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-828a86-29829225-cchfc" event={"ID":"04338d36-1035-418a-8dc4-7ce131103557","Type":"ContainerDied","Data":"483853e03ab7b983cac9c2915805633b1a2c4b6e82bb1668c070a1c355b6f4e4"}
Sep 18 17:45:19.155632 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:19.155537    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="483853e03ab7b983cac9c2915805633b1a2c4b6e82bb1668c070a1c355b6f4e4"
Sep 18 17:45:19.187545 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:19.187511    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-828a86-29829225-cchfc"]
Sep 18 17:45:19.193206 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:19.193180    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-828a86-29829225-cchfc"]
Sep 18 17:45:19.763926 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:19.763894    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04338d36-1035-418a-8dc4-7ce131103557" path="/var/lib/kubelet/pods/04338d36-1035-418a-8dc4-7ce131103557/volumes"
Sep 18 17:45:34.200604 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.200565    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj"]
Sep 18 17:45:34.201177 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.201052    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a07ab377-db7a-4873-81c6-eada976206d7" containerName="main"
Sep 18 17:45:34.201177 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.201071    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a07ab377-db7a-4873-81c6-eada976206d7" containerName="main"
Sep 18 17:45:34.201177 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.201091    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a99e64a1-4253-4c32-99d9-ff2b8261b1fa" containerName="main"
Sep 18 17:45:34.201177 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.201099    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a99e64a1-4253-4c32-99d9-ff2b8261b1fa" containerName="main"
Sep 18 17:45:34.201177 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.201109    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04338d36-1035-418a-8dc4-7ce131103557" containerName="cleanup"
Sep 18 17:45:34.201177 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.201118    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="04338d36-1035-418a-8dc4-7ce131103557" containerName="cleanup"
Sep 18 17:45:34.201177 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.201132    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="763b5e7b-e33e-4163-9e71-2d25bfba9634" containerName="cleanup"
Sep 18 17:45:34.201177 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.201141    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="763b5e7b-e33e-4163-9e71-2d25bfba9634" containerName="cleanup"
Sep 18 17:45:34.201177 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.201156    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04338d36-1035-418a-8dc4-7ce131103557" containerName="cleanup"
Sep 18 17:45:34.201177 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.201166    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="04338d36-1035-418a-8dc4-7ce131103557" containerName="cleanup"
Sep 18 17:45:34.201177 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.201175    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04338d36-1035-418a-8dc4-7ce131103557" containerName="cleanup"
Sep 18 17:45:34.201177 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.201182    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="04338d36-1035-418a-8dc4-7ce131103557" containerName="cleanup"
Sep 18 17:45:34.201854 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.201278    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a07ab377-db7a-4873-81c6-eada976206d7" containerName="main"
Sep 18 17:45:34.201854 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.201290    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="763b5e7b-e33e-4163-9e71-2d25bfba9634" containerName="cleanup"
Sep 18 17:45:34.201854 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.201304    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="04338d36-1035-418a-8dc4-7ce131103557" containerName="cleanup"
Sep 18 17:45:34.201854 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.201314    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="04338d36-1035-418a-8dc4-7ce131103557" containerName="cleanup"
Sep 18 17:45:34.201854 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.201323    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a99e64a1-4253-4c32-99d9-ff2b8261b1fa" containerName="main"
Sep 18 17:45:34.201854 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.201332    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="04338d36-1035-418a-8dc4-7ce131103557" containerName="cleanup"
Sep 18 17:45:34.204901 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.204878    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj"
Sep 18 17:45:34.207546 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.207524    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a8bdd6-serving-cert\""
Sep 18 17:45:34.215734 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.215671    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj"]
Sep 18 17:45:34.272454 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.272420    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5c54736-0b79-4269-85e1-eb478223401a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj\" (UID: \"f5c54736-0b79-4269-85e1-eb478223401a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj"
Sep 18 17:45:34.272626 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.272484    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5c54736-0b79-4269-85e1-eb478223401a-metrics-certs\") pod \"maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj\" (UID: \"f5c54736-0b79-4269-85e1-eb478223401a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj"
Sep 18 17:45:34.272626 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.272526    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5c54736-0b79-4269-85e1-eb478223401a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj\" (UID: \"f5c54736-0b79-4269-85e1-eb478223401a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj"
Sep 18 17:45:34.272626 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.272574    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8r28\" (UniqueName: \"kubernetes.io/projected/f5c54736-0b79-4269-85e1-eb478223401a-kube-api-access-j8r28\") pod \"maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj\" (UID: \"f5c54736-0b79-4269-85e1-eb478223401a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj"
Sep 18 17:45:34.373490 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.373456    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5c54736-0b79-4269-85e1-eb478223401a-metrics-certs\") pod \"maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj\" (UID: \"f5c54736-0b79-4269-85e1-eb478223401a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj"
Sep 18 17:45:34.373687 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.373499    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5c54736-0b79-4269-85e1-eb478223401a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj\" (UID: \"f5c54736-0b79-4269-85e1-eb478223401a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj"
Sep 18 17:45:34.373687 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.373541    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8r28\" (UniqueName: \"kubernetes.io/projected/f5c54736-0b79-4269-85e1-eb478223401a-kube-api-access-j8r28\") pod \"maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj\" (UID: \"f5c54736-0b79-4269-85e1-eb478223401a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj"
Sep 18 17:45:34.373687 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.373566    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5c54736-0b79-4269-85e1-eb478223401a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj\" (UID: \"f5c54736-0b79-4269-85e1-eb478223401a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj"
Sep 18 17:45:34.373687 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:45:34.373650    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8bdd6-serving-cert: secret "maas-api-e2e-ipp-b-a8bdd6-serving-cert" not found
Sep 18 17:45:34.373929 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:45:34.373704    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5c54736-0b79-4269-85e1-eb478223401a-maas-api-tls podName:f5c54736-0b79-4269-85e1-eb478223401a nodeName:}" failed. No retries permitted until 2026-09-18 17:45:34.873687653 +0000 UTC m=+1847.567615160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f5c54736-0b79-4269-85e1-eb478223401a-maas-api-tls") pod "maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj" (UID: "f5c54736-0b79-4269-85e1-eb478223401a") : secret "maas-api-e2e-ipp-b-a8bdd6-serving-cert" not found
Sep 18 17:45:34.374152 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.374132    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5c54736-0b79-4269-85e1-eb478223401a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj\" (UID: \"f5c54736-0b79-4269-85e1-eb478223401a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj"
Sep 18 17:45:34.375896 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.375876    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5c54736-0b79-4269-85e1-eb478223401a-metrics-certs\") pod \"maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj\" (UID: \"f5c54736-0b79-4269-85e1-eb478223401a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj"
Sep 18 17:45:34.382239 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.382215    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8r28\" (UniqueName: \"kubernetes.io/projected/f5c54736-0b79-4269-85e1-eb478223401a-kube-api-access-j8r28\") pod \"maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj\" (UID: \"f5c54736-0b79-4269-85e1-eb478223401a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj"
Sep 18 17:45:34.819293 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.819260    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8bdd6-5b8b68d6b-tzl8m"]
Sep 18 17:45:34.823792 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.823775    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8bdd6-5b8b68d6b-tzl8m"
Sep 18 17:45:34.826329 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.826307    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a8bdd6\""
Sep 18 17:45:34.834899 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.834849    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8bdd6-5b8b68d6b-tzl8m"]
Sep 18 17:45:34.878421 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.878390    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5c54736-0b79-4269-85e1-eb478223401a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj\" (UID: \"f5c54736-0b79-4269-85e1-eb478223401a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj"
Sep 18 17:45:34.878595 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.878441    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09808280-044f-42e3-90a4-d0da31ce0420-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a8bdd6-5b8b68d6b-tzl8m\" (UID: \"09808280-044f-42e3-90a4-d0da31ce0420\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8bdd6-5b8b68d6b-tzl8m"
Sep 18 17:45:34.878595 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.878460    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz5x9\" (UniqueName: \"kubernetes.io/projected/09808280-044f-42e3-90a4-d0da31ce0420-kube-api-access-sz5x9\") pod \"payload-pre-processing-e2e-ipp-b-a8bdd6-5b8b68d6b-tzl8m\" (UID: \"09808280-044f-42e3-90a4-d0da31ce0420\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8bdd6-5b8b68d6b-tzl8m"
Sep 18 17:45:34.880884 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.880839    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5c54736-0b79-4269-85e1-eb478223401a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj\" (UID: \"f5c54736-0b79-4269-85e1-eb478223401a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj"
Sep 18 17:45:34.980029 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.979994    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09808280-044f-42e3-90a4-d0da31ce0420-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a8bdd6-5b8b68d6b-tzl8m\" (UID: \"09808280-044f-42e3-90a4-d0da31ce0420\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8bdd6-5b8b68d6b-tzl8m"
Sep 18 17:45:34.980029 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.980030    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz5x9\" (UniqueName: \"kubernetes.io/projected/09808280-044f-42e3-90a4-d0da31ce0420-kube-api-access-sz5x9\") pod \"payload-pre-processing-e2e-ipp-b-a8bdd6-5b8b68d6b-tzl8m\" (UID: \"09808280-044f-42e3-90a4-d0da31ce0420\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8bdd6-5b8b68d6b-tzl8m"
Sep 18 17:45:34.980647 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.980627    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09808280-044f-42e3-90a4-d0da31ce0420-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a8bdd6-5b8b68d6b-tzl8m\" (UID: \"09808280-044f-42e3-90a4-d0da31ce0420\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8bdd6-5b8b68d6b-tzl8m"
Sep 18 17:45:34.989309 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:34.989288    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz5x9\" (UniqueName: \"kubernetes.io/projected/09808280-044f-42e3-90a4-d0da31ce0420-kube-api-access-sz5x9\") pod \"payload-pre-processing-e2e-ipp-b-a8bdd6-5b8b68d6b-tzl8m\" (UID: \"09808280-044f-42e3-90a4-d0da31ce0420\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8bdd6-5b8b68d6b-tzl8m"
Sep 18 17:45:35.116475 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:35.116393    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj"
Sep 18 17:45:35.134659 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:35.134633    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8bdd6-5b8b68d6b-tzl8m"
Sep 18 17:45:35.396636 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:35.396606    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj"]
Sep 18 17:45:35.398740 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:45:35.398691    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5c54736_0b79_4269_85e1_eb478223401a.slice/crio-966f0383d5cc5f0c9d6b492701840617fbf13289dbd3ff3515ec8805888c8d1f WatchSource:0}: Error finding container 966f0383d5cc5f0c9d6b492701840617fbf13289dbd3ff3515ec8805888c8d1f: Status 404 returned error can't find the container with id 966f0383d5cc5f0c9d6b492701840617fbf13289dbd3ff3515ec8805888c8d1f
Sep 18 17:45:35.418966 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:35.418941    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8bdd6-5b8b68d6b-tzl8m"]
Sep 18 17:45:35.421300 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:45:35.421263    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09808280_044f_42e3_90a4_d0da31ce0420.slice/crio-6f6e00af9eff33f50d61680ebd4f6f83f812639827c7a89af7fde41f32b38529 WatchSource:0}: Error finding container 6f6e00af9eff33f50d61680ebd4f6f83f812639827c7a89af7fde41f32b38529: Status 404 returned error can't find the container with id 6f6e00af9eff33f50d61680ebd4f6f83f812639827c7a89af7fde41f32b38529
Sep 18 17:45:36.249288 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:36.249256    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8bdd6-5b8b68d6b-tzl8m" event={"ID":"09808280-044f-42e3-90a4-d0da31ce0420","Type":"ContainerStarted","Data":"2c6850e8eb95d6de33d03a84d6ddd9ac8365827b9960e535f1900611cd9efcec"}
Sep 18 17:45:36.249475 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:36.249307    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8bdd6-5b8b68d6b-tzl8m" event={"ID":"09808280-044f-42e3-90a4-d0da31ce0420","Type":"ContainerStarted","Data":"6f6e00af9eff33f50d61680ebd4f6f83f812639827c7a89af7fde41f32b38529"}
Sep 18 17:45:36.249475 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:36.249419    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8bdd6-5b8b68d6b-tzl8m"
Sep 18 17:45:36.250992 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:36.250969    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj" event={"ID":"f5c54736-0b79-4269-85e1-eb478223401a","Type":"ContainerStarted","Data":"5ea4c198682b20e8a3130c1d245f024cb13374f27dd3eb0a26a5c5d705f2c90f"}
Sep 18 17:45:36.251131 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:36.251004    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj" event={"ID":"f5c54736-0b79-4269-85e1-eb478223401a","Type":"ContainerStarted","Data":"966f0383d5cc5f0c9d6b492701840617fbf13289dbd3ff3515ec8805888c8d1f"}
Sep 18 17:45:36.251131 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:36.251046    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj"
Sep 18 17:45:36.268628 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:36.268586    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8bdd6-5b8b68d6b-tzl8m" podStartSLOduration=2.2685727829999998 podStartE2EDuration="2.268572783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:45:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:45:36.267421366 +0000 UTC m=+1848.961348912" watchObservedRunningTime="2026-09-18 17:45:36.268572783 +0000 UTC m=+1848.962500310"
Sep 18 17:45:36.288056 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:36.288014    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj" podStartSLOduration=1.791953038 podStartE2EDuration="2.287999614s" totalImagesPullingTime="496.046576ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:45:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:45:36.286572214 +0000 UTC m=+1848.980499763" watchObservedRunningTime="2026-09-18 17:45:36.287999614 +0000 UTC m=+1848.981927141"
Sep 18 17:45:42.263015 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:42.262983    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj"
Sep 18 17:45:47.259330 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:47.259300    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8bdd6-5b8b68d6b-tzl8m"
Sep 18 17:45:48.203211 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:48.203173    2569 scope.go:117] "RemoveContainer" containerID="13694fa5f0d8aa26359d65d308346f428ca84a76b8c2f8691f27e325a0beda1b"
Sep 18 17:45:48.211450 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:45:48.211429    2569 scope.go:117] "RemoveContainer" containerID="0d346758e1ef961f7ca714d389203a33755c80d7035f53cd89c79f8d2c2909ac"
Sep 18 17:46:24.265562 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:24.265487    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd"]
Sep 18 17:46:24.268873 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:24.268846    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd"
Sep 18 17:46:24.271457 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:24.271431    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-828a86\"/\"openshift-service-ca.crt\""
Sep 18 17:46:24.271576 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:24.271435    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-828a86\"/\"kube-root-ca.crt\""
Sep 18 17:46:24.272300 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:24.272281    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-828a86\"/\"default-dockercfg-pt8d9\""
Sep 18 17:46:24.272366 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:24.272287    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-828a86\"/\"ipp-route-828a86-kserve-self-signed-certs\""
Sep 18 17:46:24.280476 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:24.280449    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd"]
Sep 18 17:46:24.419974 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:24.419936    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61456fa8-cbab-4be5-ae6b-99091a1d595b-dshm\") pod \"ipp-route-828a86-kserve-76cd6cc67f-hrvvd\" (UID: \"61456fa8-cbab-4be5-ae6b-99091a1d595b\") " pod="ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd"
Sep 18 17:46:24.420148 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:24.419981    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61456fa8-cbab-4be5-ae6b-99091a1d595b-model-cache\") pod \"ipp-route-828a86-kserve-76cd6cc67f-hrvvd\" (UID: \"61456fa8-cbab-4be5-ae6b-99091a1d595b\") " pod="ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd"
Sep 18 17:46:24.420148 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:24.420023    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61456fa8-cbab-4be5-ae6b-99091a1d595b-tls-certs\") pod \"ipp-route-828a86-kserve-76cd6cc67f-hrvvd\" (UID: \"61456fa8-cbab-4be5-ae6b-99091a1d595b\") " pod="ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd"
Sep 18 17:46:24.420148 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:24.420047    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61456fa8-cbab-4be5-ae6b-99091a1d595b-home\") pod \"ipp-route-828a86-kserve-76cd6cc67f-hrvvd\" (UID: \"61456fa8-cbab-4be5-ae6b-99091a1d595b\") " pod="ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd"
Sep 18 17:46:24.420148 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:24.420071    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n9s5\" (UniqueName: \"kubernetes.io/projected/61456fa8-cbab-4be5-ae6b-99091a1d595b-kube-api-access-2n9s5\") pod \"ipp-route-828a86-kserve-76cd6cc67f-hrvvd\" (UID: \"61456fa8-cbab-4be5-ae6b-99091a1d595b\") " pod="ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd"
Sep 18 17:46:24.420289 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:24.420144    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61456fa8-cbab-4be5-ae6b-99091a1d595b-tmp-dir\") pod \"ipp-route-828a86-kserve-76cd6cc67f-hrvvd\" (UID: \"61456fa8-cbab-4be5-ae6b-99091a1d595b\") " pod="ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd"
Sep 18 17:46:24.521389 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:24.521294    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61456fa8-cbab-4be5-ae6b-99091a1d595b-dshm\") pod \"ipp-route-828a86-kserve-76cd6cc67f-hrvvd\" (UID: \"61456fa8-cbab-4be5-ae6b-99091a1d595b\") " pod="ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd"
Sep 18 17:46:24.521389 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:24.521344    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61456fa8-cbab-4be5-ae6b-99091a1d595b-model-cache\") pod \"ipp-route-828a86-kserve-76cd6cc67f-hrvvd\" (UID: \"61456fa8-cbab-4be5-ae6b-99091a1d595b\") " pod="ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd"
Sep 18 17:46:24.521626 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:24.521395    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61456fa8-cbab-4be5-ae6b-99091a1d595b-tls-certs\") pod \"ipp-route-828a86-kserve-76cd6cc67f-hrvvd\" (UID: \"61456fa8-cbab-4be5-ae6b-99091a1d595b\") " pod="ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd"
Sep 18 17:46:24.521626 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:24.521434    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61456fa8-cbab-4be5-ae6b-99091a1d595b-home\") pod \"ipp-route-828a86-kserve-76cd6cc67f-hrvvd\" (UID: \"61456fa8-cbab-4be5-ae6b-99091a1d595b\") " pod="ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd"
Sep 18 17:46:24.521626 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:24.521470    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n9s5\" (UniqueName: \"kubernetes.io/projected/61456fa8-cbab-4be5-ae6b-99091a1d595b-kube-api-access-2n9s5\") pod \"ipp-route-828a86-kserve-76cd6cc67f-hrvvd\" (UID: \"61456fa8-cbab-4be5-ae6b-99091a1d595b\") " pod="ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd"
Sep 18 17:46:24.521626 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:24.521514    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61456fa8-cbab-4be5-ae6b-99091a1d595b-tmp-dir\") pod \"ipp-route-828a86-kserve-76cd6cc67f-hrvvd\" (UID: \"61456fa8-cbab-4be5-ae6b-99091a1d595b\") " pod="ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd"
Sep 18 17:46:24.521851 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:24.521780    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61456fa8-cbab-4be5-ae6b-99091a1d595b-model-cache\") pod \"ipp-route-828a86-kserve-76cd6cc67f-hrvvd\" (UID: \"61456fa8-cbab-4be5-ae6b-99091a1d595b\") " pod="ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd"
Sep 18 17:46:24.521945 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:24.521894    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61456fa8-cbab-4be5-ae6b-99091a1d595b-tmp-dir\") pod \"ipp-route-828a86-kserve-76cd6cc67f-hrvvd\" (UID: \"61456fa8-cbab-4be5-ae6b-99091a1d595b\") " pod="ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd"
Sep 18 17:46:24.522425 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:24.522367    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61456fa8-cbab-4be5-ae6b-99091a1d595b-home\") pod \"ipp-route-828a86-kserve-76cd6cc67f-hrvvd\" (UID: \"61456fa8-cbab-4be5-ae6b-99091a1d595b\") " pod="ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd"
Sep 18 17:46:24.529446 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:24.524115    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61456fa8-cbab-4be5-ae6b-99091a1d595b-tls-certs\") pod \"ipp-route-828a86-kserve-76cd6cc67f-hrvvd\" (UID: \"61456fa8-cbab-4be5-ae6b-99091a1d595b\") " pod="ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd"
Sep 18 17:46:24.529909 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:24.529882    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61456fa8-cbab-4be5-ae6b-99091a1d595b-dshm\") pod \"ipp-route-828a86-kserve-76cd6cc67f-hrvvd\" (UID: \"61456fa8-cbab-4be5-ae6b-99091a1d595b\") " pod="ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd"
Sep 18 17:46:24.531447 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:24.531428    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n9s5\" (UniqueName: \"kubernetes.io/projected/61456fa8-cbab-4be5-ae6b-99091a1d595b-kube-api-access-2n9s5\") pod \"ipp-route-828a86-kserve-76cd6cc67f-hrvvd\" (UID: \"61456fa8-cbab-4be5-ae6b-99091a1d595b\") " pod="ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd"
Sep 18 17:46:24.577832 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:24.577807    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd"
Sep 18 17:46:24.851776 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:24.851753    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd"]
Sep 18 17:46:24.854004 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:46:24.853974    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61456fa8_cbab_4be5_ae6b_99091a1d595b.slice/crio-a96601d5d4f97b86e4fb51dd7751bc6396eb2f58891c63f897ff260c4fbfc4c1 WatchSource:0}: Error finding container a96601d5d4f97b86e4fb51dd7751bc6396eb2f58891c63f897ff260c4fbfc4c1: Status 404 returned error can't find the container with id a96601d5d4f97b86e4fb51dd7751bc6396eb2f58891c63f897ff260c4fbfc4c1
Sep 18 17:46:25.506295 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:25.506257    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd" event={"ID":"61456fa8-cbab-4be5-ae6b-99091a1d595b","Type":"ContainerStarted","Data":"ccd79cfb6bfd48986defc6e5844a7bd4013bfb03282a6fa743a67092ab86b843"}
Sep 18 17:46:25.506295 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:25.506300    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd" event={"ID":"61456fa8-cbab-4be5-ae6b-99091a1d595b","Type":"ContainerStarted","Data":"a96601d5d4f97b86e4fb51dd7751bc6396eb2f58891c63f897ff260c4fbfc4c1"}
Sep 18 17:46:25.526435 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:25.526378    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd" podStartSLOduration=1.5263615000000001 podStartE2EDuration="1.5263615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:46:25.524040765 +0000 UTC m=+1898.217968320" watchObservedRunningTime="2026-09-18 17:46:25.5263615 +0000 UTC m=+1898.220289077"
Sep 18 17:46:34.578353 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:34.578315    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd"
Sep 18 17:46:34.578845 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:34.578392    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd"
Sep 18 17:46:34.590693 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:34.590671    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd"
Sep 18 17:46:35.568163 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:35.568132    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd"
Sep 18 17:46:47.520252 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:47.520219    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-828a86-59d7877675-x6fdt_58f0a29e-9531-4b07-a8dd-41cafe6c74d6/payload-processing/0.log"
Sep 18 17:46:48.240388 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:48.240357    2569 scope.go:117] "RemoveContainer" containerID="69399f7edca4cb83e019a4960ff145634cac37532ef6bf1eb7ec1d1f5ad3407d"
Sep 18 17:46:48.248322 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:48.248302    2569 scope.go:117] "RemoveContainer" containerID="c2cdfbc54911028276eaf4e5057fa8b9f867131f71d9d56318fdabab9f606338"
Sep 18 17:46:56.147563 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:56.147522    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-88b9e1-7b945c94b-nlvzp"]
Sep 18 17:46:56.151503 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:56.151477    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-88b9e1-7b945c94b-nlvzp"
Sep 18 17:46:56.154079 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:56.154051    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-88b9e1\""
Sep 18 17:46:56.154217 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:56.154063    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-88b9e1-dockercfg-qsv2p\""
Sep 18 17:46:56.162631 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:56.162606    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-88b9e1-7b945c94b-nlvzp"]
Sep 18 17:46:56.199417 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:56.199392    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghmmk\" (UniqueName: \"kubernetes.io/projected/70a303db-3b6a-4cb5-81b2-4dea8288c66f-kube-api-access-ghmmk\") pod \"payload-pre-processing-e2e-ipp-cleanup-88b9e1-7b945c94b-nlvzp\" (UID: \"70a303db-3b6a-4cb5-81b2-4dea8288c66f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-88b9e1-7b945c94b-nlvzp"
Sep 18 17:46:56.199556 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:56.199425    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a303db-3b6a-4cb5-81b2-4dea8288c66f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-88b9e1-7b945c94b-nlvzp\" (UID: \"70a303db-3b6a-4cb5-81b2-4dea8288c66f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-88b9e1-7b945c94b-nlvzp"
Sep 18 17:46:56.300157 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:56.300123    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghmmk\" (UniqueName: \"kubernetes.io/projected/70a303db-3b6a-4cb5-81b2-4dea8288c66f-kube-api-access-ghmmk\") pod \"payload-pre-processing-e2e-ipp-cleanup-88b9e1-7b945c94b-nlvzp\" (UID: \"70a303db-3b6a-4cb5-81b2-4dea8288c66f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-88b9e1-7b945c94b-nlvzp"
Sep 18 17:46:56.300327 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:56.300165    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a303db-3b6a-4cb5-81b2-4dea8288c66f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-88b9e1-7b945c94b-nlvzp\" (UID: \"70a303db-3b6a-4cb5-81b2-4dea8288c66f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-88b9e1-7b945c94b-nlvzp"
Sep 18 17:46:56.300918 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:56.300899    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a303db-3b6a-4cb5-81b2-4dea8288c66f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-88b9e1-7b945c94b-nlvzp\" (UID: \"70a303db-3b6a-4cb5-81b2-4dea8288c66f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-88b9e1-7b945c94b-nlvzp"
Sep 18 17:46:56.308905 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:56.308854    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghmmk\" (UniqueName: \"kubernetes.io/projected/70a303db-3b6a-4cb5-81b2-4dea8288c66f-kube-api-access-ghmmk\") pod \"payload-pre-processing-e2e-ipp-cleanup-88b9e1-7b945c94b-nlvzp\" (UID: \"70a303db-3b6a-4cb5-81b2-4dea8288c66f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-88b9e1-7b945c94b-nlvzp"
Sep 18 17:46:56.461601 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:56.461516    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-88b9e1-7b945c94b-nlvzp"
Sep 18 17:46:56.734892 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:56.734721    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-88b9e1-7b945c94b-nlvzp"]
Sep 18 17:46:56.741420 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:46:56.741392    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70a303db_3b6a_4cb5_81b2_4dea8288c66f.slice/crio-01f6f945850643650decb7c0b85e32cdac2184c247d9b48b9e11394992dedbe8 WatchSource:0}: Error finding container 01f6f945850643650decb7c0b85e32cdac2184c247d9b48b9e11394992dedbe8: Status 404 returned error can't find the container with id 01f6f945850643650decb7c0b85e32cdac2184c247d9b48b9e11394992dedbe8
Sep 18 17:46:57.666845 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:57.666801    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-88b9e1-7b945c94b-nlvzp" event={"ID":"70a303db-3b6a-4cb5-81b2-4dea8288c66f","Type":"ContainerStarted","Data":"df41965de5ee4e68f8e1404751e959c9dad9303434d7543df133d9280a09136e"}
Sep 18 17:46:57.667368 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:57.666875    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-88b9e1-7b945c94b-nlvzp" event={"ID":"70a303db-3b6a-4cb5-81b2-4dea8288c66f","Type":"ContainerStarted","Data":"01f6f945850643650decb7c0b85e32cdac2184c247d9b48b9e11394992dedbe8"}
Sep 18 17:46:57.667368 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:57.667033    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-88b9e1-7b945c94b-nlvzp"
Sep 18 17:46:57.692701 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:46:57.692651    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-88b9e1-7b945c94b-nlvzp" podStartSLOduration=1.69263774 podStartE2EDuration="1.69263774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:46:57.691491247 +0000 UTC m=+1930.385418774" watchObservedRunningTime="2026-09-18 17:46:57.69263774 +0000 UTC m=+1930.386565304"
Sep 18 17:47:08.673713 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:08.673679    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-88b9e1-7b945c94b-nlvzp"
Sep 18 17:47:22.180819 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:22.180785    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-88b9e1-49h8l"]
Sep 18 17:47:22.184162 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:22.184145    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-88b9e1-49h8l"
Sep 18 17:47:22.186680 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:22.186659    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qt978\""
Sep 18 17:47:22.193366 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:22.193336    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-88b9e1-49h8l"]
Sep 18 17:47:22.333925 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:22.333888    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54721180-3e18-4f55-8d2b-a611b57b9c49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-88b9e1-49h8l\" (UID: \"54721180-3e18-4f55-8d2b-a611b57b9c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-88b9e1-49h8l"
Sep 18 17:47:22.435093 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:22.435010    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54721180-3e18-4f55-8d2b-a611b57b9c49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-88b9e1-49h8l\" (UID: \"54721180-3e18-4f55-8d2b-a611b57b9c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-88b9e1-49h8l"
Sep 18 17:47:22.435682 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:22.435660    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54721180-3e18-4f55-8d2b-a611b57b9c49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-88b9e1-49h8l\" (UID: \"54721180-3e18-4f55-8d2b-a611b57b9c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-88b9e1-49h8l"
Sep 18 17:47:22.495510 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:22.495477    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-88b9e1-49h8l"
Sep 18 17:47:22.766795 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:22.766710    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-88b9e1-49h8l"]
Sep 18 17:47:22.769482 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:47:22.769450    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54721180_3e18_4f55_8d2b_a611b57b9c49.slice/crio-11596b2361d9fd9183046f32f421d8e1797b90cd7d429b9462241e390d95555b WatchSource:0}: Error finding container 11596b2361d9fd9183046f32f421d8e1797b90cd7d429b9462241e390d95555b: Status 404 returned error can't find the container with id 11596b2361d9fd9183046f32f421d8e1797b90cd7d429b9462241e390d95555b
Sep 18 17:47:22.801553 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:22.801521    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-88b9e1-49h8l" event={"ID":"54721180-3e18-4f55-8d2b-a611b57b9c49","Type":"ContainerStarted","Data":"11596b2361d9fd9183046f32f421d8e1797b90cd7d429b9462241e390d95555b"}
Sep 18 17:47:23.808225 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:23.808184    2569 generic.go:356] "Generic (PLEG): container finished" podID="54721180-3e18-4f55-8d2b-a611b57b9c49" containerID="c664cd3e3fc435537d3e480e0b0ef91437405dc92b1f0005a30fa375311979a3" exitCode=0
Sep 18 17:47:23.808650 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:23.808242    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-88b9e1-49h8l" event={"ID":"54721180-3e18-4f55-8d2b-a611b57b9c49","Type":"ContainerDied","Data":"c664cd3e3fc435537d3e480e0b0ef91437405dc92b1f0005a30fa375311979a3"}
Sep 18 17:47:24.946324 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:24.946301    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-88b9e1-49h8l"
Sep 18 17:47:25.061116 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:25.061082    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54721180-3e18-4f55-8d2b-a611b57b9c49-maas-api-service-ca\") pod \"54721180-3e18-4f55-8d2b-a611b57b9c49\" (UID: \"54721180-3e18-4f55-8d2b-a611b57b9c49\") "
Sep 18 17:47:25.061445 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:25.061420    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54721180-3e18-4f55-8d2b-a611b57b9c49-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "54721180-3e18-4f55-8d2b-a611b57b9c49" (UID: "54721180-3e18-4f55-8d2b-a611b57b9c49"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:47:25.162753 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:25.162658    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54721180-3e18-4f55-8d2b-a611b57b9c49-maas-api-service-ca\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:47:25.819282 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:25.819254    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-88b9e1-49h8l"
Sep 18 17:47:25.819449 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:25.819253    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-88b9e1-49h8l" event={"ID":"54721180-3e18-4f55-8d2b-a611b57b9c49","Type":"ContainerDied","Data":"11596b2361d9fd9183046f32f421d8e1797b90cd7d429b9462241e390d95555b"}
Sep 18 17:47:25.819449 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:25.819357    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11596b2361d9fd9183046f32f421d8e1797b90cd7d429b9462241e390d95555b"
Sep 18 17:47:33.533265 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:33.533234    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-88b9e1-7b945c94b-nlvzp"]
Sep 18 17:47:33.533682 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:33.533427    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-88b9e1-7b945c94b-nlvzp" podUID="70a303db-3b6a-4cb5-81b2-4dea8288c66f" containerName="payload-pre-processing" containerID="cri-o://df41965de5ee4e68f8e1404751e959c9dad9303434d7543df133d9280a09136e" gracePeriod=30
Sep 18 17:47:33.789472 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:33.789413    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-88b9e1-7b945c94b-nlvzp"
Sep 18 17:47:33.835007 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:33.834978    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghmmk\" (UniqueName: \"kubernetes.io/projected/70a303db-3b6a-4cb5-81b2-4dea8288c66f-kube-api-access-ghmmk\") pod \"70a303db-3b6a-4cb5-81b2-4dea8288c66f\" (UID: \"70a303db-3b6a-4cb5-81b2-4dea8288c66f\") "
Sep 18 17:47:33.835164 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:33.835018    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a303db-3b6a-4cb5-81b2-4dea8288c66f-plugins-config-volume\") pod \"70a303db-3b6a-4cb5-81b2-4dea8288c66f\" (UID: \"70a303db-3b6a-4cb5-81b2-4dea8288c66f\") "
Sep 18 17:47:33.835446 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:33.835416    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70a303db-3b6a-4cb5-81b2-4dea8288c66f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "70a303db-3b6a-4cb5-81b2-4dea8288c66f" (UID: "70a303db-3b6a-4cb5-81b2-4dea8288c66f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:47:33.837144 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:33.837121    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70a303db-3b6a-4cb5-81b2-4dea8288c66f-kube-api-access-ghmmk" (OuterVolumeSpecName: "kube-api-access-ghmmk") pod "70a303db-3b6a-4cb5-81b2-4dea8288c66f" (UID: "70a303db-3b6a-4cb5-81b2-4dea8288c66f"). InnerVolumeSpecName "kube-api-access-ghmmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:47:33.859096 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:33.859065    2569 generic.go:356] "Generic (PLEG): container finished" podID="70a303db-3b6a-4cb5-81b2-4dea8288c66f" containerID="df41965de5ee4e68f8e1404751e959c9dad9303434d7543df133d9280a09136e" exitCode=0
Sep 18 17:47:33.859211 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:33.859108    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-88b9e1-7b945c94b-nlvzp"
Sep 18 17:47:33.859211 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:33.859155    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-88b9e1-7b945c94b-nlvzp" event={"ID":"70a303db-3b6a-4cb5-81b2-4dea8288c66f","Type":"ContainerDied","Data":"df41965de5ee4e68f8e1404751e959c9dad9303434d7543df133d9280a09136e"}
Sep 18 17:47:33.859211 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:33.859204    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-88b9e1-7b945c94b-nlvzp" event={"ID":"70a303db-3b6a-4cb5-81b2-4dea8288c66f","Type":"ContainerDied","Data":"01f6f945850643650decb7c0b85e32cdac2184c247d9b48b9e11394992dedbe8"}
Sep 18 17:47:33.859328 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:33.859225    2569 scope.go:117] "RemoveContainer" containerID="df41965de5ee4e68f8e1404751e959c9dad9303434d7543df133d9280a09136e"
Sep 18 17:47:33.868890 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:33.868855    2569 scope.go:117] "RemoveContainer" containerID="df41965de5ee4e68f8e1404751e959c9dad9303434d7543df133d9280a09136e"
Sep 18 17:47:33.869158 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:47:33.869137    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df41965de5ee4e68f8e1404751e959c9dad9303434d7543df133d9280a09136e\": container with ID starting with df41965de5ee4e68f8e1404751e959c9dad9303434d7543df133d9280a09136e not found: ID does not exist" containerID="df41965de5ee4e68f8e1404751e959c9dad9303434d7543df133d9280a09136e"
Sep 18 17:47:33.869207 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:33.869166    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df41965de5ee4e68f8e1404751e959c9dad9303434d7543df133d9280a09136e"} err="failed to get container status \"df41965de5ee4e68f8e1404751e959c9dad9303434d7543df133d9280a09136e\": rpc error: code = NotFound desc = could not find container \"df41965de5ee4e68f8e1404751e959c9dad9303434d7543df133d9280a09136e\": container with ID starting with df41965de5ee4e68f8e1404751e959c9dad9303434d7543df133d9280a09136e not found: ID does not exist"
Sep 18 17:47:33.877265 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:33.877244    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-88b9e1-49h8l"]
Sep 18 17:47:33.881784 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:33.881762    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-88b9e1-49h8l"]
Sep 18 17:47:33.887052 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:33.887032    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-88b9e1-7b945c94b-nlvzp"]
Sep 18 17:47:33.892972 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:33.892952    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-88b9e1-7b945c94b-nlvzp"]
Sep 18 17:47:33.936499 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:33.936466    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghmmk\" (UniqueName: \"kubernetes.io/projected/70a303db-3b6a-4cb5-81b2-4dea8288c66f-kube-api-access-ghmmk\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:47:33.936499 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:33.936493    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a303db-3b6a-4cb5-81b2-4dea8288c66f-plugins-config-volume\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:47:35.764687 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:35.764649    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54721180-3e18-4f55-8d2b-a611b57b9c49" path="/var/lib/kubelet/pods/54721180-3e18-4f55-8d2b-a611b57b9c49/volumes"
Sep 18 17:47:35.765085 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:35.765024    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70a303db-3b6a-4cb5-81b2-4dea8288c66f" path="/var/lib/kubelet/pods/70a303db-3b6a-4cb5-81b2-4dea8288c66f/volumes"
Sep 18 17:47:48.270321 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:48.270245    2569 scope.go:117] "RemoveContainer" containerID="e5619660975970fb29420ecf651f31b5bd7eb80472200716c88dd6cc65464833"
Sep 18 17:47:48.278493 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:48.278473    2569 scope.go:117] "RemoveContainer" containerID="132fc4722ab5beca5aac7c4ffd531d61b1d3edbf00a07ff977c4e31d110dbff6"
Sep 18 17:47:59.615976 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:59.615936    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv"]
Sep 18 17:47:59.616430 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:59.616161    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv" podUID="69a7f0df-3f15-4367-91ff-79f0267c7ac4" containerName="maas-api" containerID="cri-o://09f502a6cee891ed2656a0111306c113c17871d0cdc19035bbbd1faa9e5cb37f" gracePeriod=30
Sep 18 17:47:59.798246 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:59.798154    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-828a86-59d7877675-x6fdt"]
Sep 18 17:47:59.798425 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:59.798397    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-828a86-59d7877675-x6fdt" podUID="58f0a29e-9531-4b07-a8dd-41cafe6c74d6" containerName="payload-processing" containerID="cri-o://4bb2231738ae1f0c572889ba0b8c7eec4bb0e93e079c98c723900d3bb8765020" gracePeriod=30
Sep 18 17:47:59.925368 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:59.925339    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv"
Sep 18 17:47:59.994091 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:59.994059    2569 generic.go:356] "Generic (PLEG): container finished" podID="69a7f0df-3f15-4367-91ff-79f0267c7ac4" containerID="09f502a6cee891ed2656a0111306c113c17871d0cdc19035bbbd1faa9e5cb37f" exitCode=0
Sep 18 17:47:59.994263 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:59.994105    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv"
Sep 18 17:47:59.994263 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:59.994137    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv" event={"ID":"69a7f0df-3f15-4367-91ff-79f0267c7ac4","Type":"ContainerDied","Data":"09f502a6cee891ed2656a0111306c113c17871d0cdc19035bbbd1faa9e5cb37f"}
Sep 18 17:47:59.994263 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:59.994190    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv" event={"ID":"69a7f0df-3f15-4367-91ff-79f0267c7ac4","Type":"ContainerDied","Data":"da120982d7f5ea75dcabe3963526dbcf3cc4017dfd1b60e8fb5dd3de6175a827"}
Sep 18 17:47:59.994263 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:59.994212    2569 scope.go:117] "RemoveContainer" containerID="09f502a6cee891ed2656a0111306c113c17871d0cdc19035bbbd1faa9e5cb37f"
Sep 18 17:47:59.995889 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:59.995852    2569 generic.go:356] "Generic (PLEG): container finished" podID="58f0a29e-9531-4b07-a8dd-41cafe6c74d6" containerID="4bb2231738ae1f0c572889ba0b8c7eec4bb0e93e079c98c723900d3bb8765020" exitCode=0
Sep 18 17:47:59.996009 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:47:59.995894    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-828a86-59d7877675-x6fdt" event={"ID":"58f0a29e-9531-4b07-a8dd-41cafe6c74d6","Type":"ContainerDied","Data":"4bb2231738ae1f0c572889ba0b8c7eec4bb0e93e079c98c723900d3bb8765020"}
Sep 18 17:48:00.004405 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:00.004387    2569 scope.go:117] "RemoveContainer" containerID="09f502a6cee891ed2656a0111306c113c17871d0cdc19035bbbd1faa9e5cb37f"
Sep 18 17:48:00.004655 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:48:00.004634    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09f502a6cee891ed2656a0111306c113c17871d0cdc19035bbbd1faa9e5cb37f\": container with ID starting with 09f502a6cee891ed2656a0111306c113c17871d0cdc19035bbbd1faa9e5cb37f not found: ID does not exist" containerID="09f502a6cee891ed2656a0111306c113c17871d0cdc19035bbbd1faa9e5cb37f"
Sep 18 17:48:00.004710 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:00.004663    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09f502a6cee891ed2656a0111306c113c17871d0cdc19035bbbd1faa9e5cb37f"} err="failed to get container status \"09f502a6cee891ed2656a0111306c113c17871d0cdc19035bbbd1faa9e5cb37f\": rpc error: code = NotFound desc = could not find container \"09f502a6cee891ed2656a0111306c113c17871d0cdc19035bbbd1faa9e5cb37f\": container with ID starting with 09f502a6cee891ed2656a0111306c113c17871d0cdc19035bbbd1faa9e5cb37f not found: ID does not exist"
Sep 18 17:48:00.047446 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:00.047421    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-828a86-59d7877675-x6fdt"
Sep 18 17:48:00.080192 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:00.080165    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69a7f0df-3f15-4367-91ff-79f0267c7ac4-maas-api-tls\") pod \"69a7f0df-3f15-4367-91ff-79f0267c7ac4\" (UID: \"69a7f0df-3f15-4367-91ff-79f0267c7ac4\") "
Sep 18 17:48:00.080340 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:00.080210    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9644\" (UniqueName: \"kubernetes.io/projected/69a7f0df-3f15-4367-91ff-79f0267c7ac4-kube-api-access-t9644\") pod \"69a7f0df-3f15-4367-91ff-79f0267c7ac4\" (UID: \"69a7f0df-3f15-4367-91ff-79f0267c7ac4\") "
Sep 18 17:48:00.080340 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:00.080244    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69a7f0df-3f15-4367-91ff-79f0267c7ac4-metrics-certs\") pod \"69a7f0df-3f15-4367-91ff-79f0267c7ac4\" (UID: \"69a7f0df-3f15-4367-91ff-79f0267c7ac4\") "
Sep 18 17:48:00.080476 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:00.080339    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69a7f0df-3f15-4367-91ff-79f0267c7ac4-openshift-service-ca\") pod \"69a7f0df-3f15-4367-91ff-79f0267c7ac4\" (UID: \"69a7f0df-3f15-4367-91ff-79f0267c7ac4\") "
Sep 18 17:48:00.080742 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:00.080712    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69a7f0df-3f15-4367-91ff-79f0267c7ac4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "69a7f0df-3f15-4367-91ff-79f0267c7ac4" (UID: "69a7f0df-3f15-4367-91ff-79f0267c7ac4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:48:00.082417 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:00.082385    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69a7f0df-3f15-4367-91ff-79f0267c7ac4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "69a7f0df-3f15-4367-91ff-79f0267c7ac4" (UID: "69a7f0df-3f15-4367-91ff-79f0267c7ac4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:48:00.082510 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:00.082413    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69a7f0df-3f15-4367-91ff-79f0267c7ac4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "69a7f0df-3f15-4367-91ff-79f0267c7ac4" (UID: "69a7f0df-3f15-4367-91ff-79f0267c7ac4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:48:00.082510 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:00.082441    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a7f0df-3f15-4367-91ff-79f0267c7ac4-kube-api-access-t9644" (OuterVolumeSpecName: "kube-api-access-t9644") pod "69a7f0df-3f15-4367-91ff-79f0267c7ac4" (UID: "69a7f0df-3f15-4367-91ff-79f0267c7ac4"). InnerVolumeSpecName "kube-api-access-t9644". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:48:00.181945 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:00.181827    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f0a29e-9531-4b07-a8dd-41cafe6c74d6-plugins-config-volume\") pod \"58f0a29e-9531-4b07-a8dd-41cafe6c74d6\" (UID: \"58f0a29e-9531-4b07-a8dd-41cafe6c74d6\") "
Sep 18 17:48:00.181945 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:00.181934    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xgh5\" (UniqueName: \"kubernetes.io/projected/58f0a29e-9531-4b07-a8dd-41cafe6c74d6-kube-api-access-6xgh5\") pod \"58f0a29e-9531-4b07-a8dd-41cafe6c74d6\" (UID: \"58f0a29e-9531-4b07-a8dd-41cafe6c74d6\") "
Sep 18 17:48:00.182214 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:00.182196    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69a7f0df-3f15-4367-91ff-79f0267c7ac4-maas-api-tls\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:48:00.182280 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:00.182202    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58f0a29e-9531-4b07-a8dd-41cafe6c74d6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "58f0a29e-9531-4b07-a8dd-41cafe6c74d6" (UID: "58f0a29e-9531-4b07-a8dd-41cafe6c74d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:48:00.182280 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:00.182218    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9644\" (UniqueName: \"kubernetes.io/projected/69a7f0df-3f15-4367-91ff-79f0267c7ac4-kube-api-access-t9644\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:48:00.182280 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:00.182234    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69a7f0df-3f15-4367-91ff-79f0267c7ac4-metrics-certs\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:48:00.182280 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:00.182251    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69a7f0df-3f15-4367-91ff-79f0267c7ac4-openshift-service-ca\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:48:00.183935 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:00.183913    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58f0a29e-9531-4b07-a8dd-41cafe6c74d6-kube-api-access-6xgh5" (OuterVolumeSpecName: "kube-api-access-6xgh5") pod "58f0a29e-9531-4b07-a8dd-41cafe6c74d6" (UID: "58f0a29e-9531-4b07-a8dd-41cafe6c74d6"). InnerVolumeSpecName "kube-api-access-6xgh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:48:00.283302 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:00.283271    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f0a29e-9531-4b07-a8dd-41cafe6c74d6-plugins-config-volume\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:48:00.283302 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:00.283297    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xgh5\" (UniqueName: \"kubernetes.io/projected/58f0a29e-9531-4b07-a8dd-41cafe6c74d6-kube-api-access-6xgh5\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:48:00.326285 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:00.326250    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv"]
Sep 18 17:48:00.333557 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:00.333526    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-828a86-76d85cb6db-7lrjv"]
Sep 18 17:48:01.003203 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:01.003169    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-828a86-59d7877675-x6fdt" event={"ID":"58f0a29e-9531-4b07-a8dd-41cafe6c74d6","Type":"ContainerDied","Data":"aae5c176b92707ab9c4450553cfee89cb7234ec12cefa4fc80605d2c50212954"}
Sep 18 17:48:01.003203 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:01.003204    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-828a86-59d7877675-x6fdt"
Sep 18 17:48:01.003718 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:01.003243    2569 scope.go:117] "RemoveContainer" containerID="4bb2231738ae1f0c572889ba0b8c7eec4bb0e93e079c98c723900d3bb8765020"
Sep 18 17:48:01.055000 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:01.054966    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-828a86-59d7877675-x6fdt"]
Sep 18 17:48:01.061141 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:01.061117    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-828a86-59d7877675-x6fdt"]
Sep 18 17:48:01.764382 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:01.764351    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58f0a29e-9531-4b07-a8dd-41cafe6c74d6" path="/var/lib/kubelet/pods/58f0a29e-9531-4b07-a8dd-41cafe6c74d6/volumes"
Sep 18 17:48:01.764744 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:01.764731    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69a7f0df-3f15-4367-91ff-79f0267c7ac4" path="/var/lib/kubelet/pods/69a7f0df-3f15-4367-91ff-79f0267c7ac4/volumes"
Sep 18 17:48:05.497368 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:05.497330    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd"]
Sep 18 17:48:20.516397 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:20.516336    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd" podUID="61456fa8-cbab-4be5-ae6b-99091a1d595b" containerName="main" containerID="cri-o://ccd79cfb6bfd48986defc6e5844a7bd4013bfb03282a6fa743a67092ab86b843" gracePeriod=45
Sep 18 17:48:20.757795 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:20.757770    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd"
Sep 18 17:48:20.863555 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:20.863459    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61456fa8-cbab-4be5-ae6b-99091a1d595b-home\") pod \"61456fa8-cbab-4be5-ae6b-99091a1d595b\" (UID: \"61456fa8-cbab-4be5-ae6b-99091a1d595b\") "
Sep 18 17:48:20.863555 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:20.863508    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2n9s5\" (UniqueName: \"kubernetes.io/projected/61456fa8-cbab-4be5-ae6b-99091a1d595b-kube-api-access-2n9s5\") pod \"61456fa8-cbab-4be5-ae6b-99091a1d595b\" (UID: \"61456fa8-cbab-4be5-ae6b-99091a1d595b\") "
Sep 18 17:48:20.863555 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:20.863539    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61456fa8-cbab-4be5-ae6b-99091a1d595b-tls-certs\") pod \"61456fa8-cbab-4be5-ae6b-99091a1d595b\" (UID: \"61456fa8-cbab-4be5-ae6b-99091a1d595b\") "
Sep 18 17:48:20.863849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:20.863588    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61456fa8-cbab-4be5-ae6b-99091a1d595b-dshm\") pod \"61456fa8-cbab-4be5-ae6b-99091a1d595b\" (UID: \"61456fa8-cbab-4be5-ae6b-99091a1d595b\") "
Sep 18 17:48:20.863849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:20.863636    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61456fa8-cbab-4be5-ae6b-99091a1d595b-model-cache\") pod \"61456fa8-cbab-4be5-ae6b-99091a1d595b\" (UID: \"61456fa8-cbab-4be5-ae6b-99091a1d595b\") "
Sep 18 17:48:20.863849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:20.863666    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61456fa8-cbab-4be5-ae6b-99091a1d595b-tmp-dir\") pod \"61456fa8-cbab-4be5-ae6b-99091a1d595b\" (UID: \"61456fa8-cbab-4be5-ae6b-99091a1d595b\") "
Sep 18 17:48:20.863849 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:20.863805    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61456fa8-cbab-4be5-ae6b-99091a1d595b-home" (OuterVolumeSpecName: "home") pod "61456fa8-cbab-4be5-ae6b-99091a1d595b" (UID: "61456fa8-cbab-4be5-ae6b-99091a1d595b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:48:20.864096 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:20.863905    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61456fa8-cbab-4be5-ae6b-99091a1d595b-model-cache" (OuterVolumeSpecName: "model-cache") pod "61456fa8-cbab-4be5-ae6b-99091a1d595b" (UID: "61456fa8-cbab-4be5-ae6b-99091a1d595b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:48:20.864096 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:20.863948    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61456fa8-cbab-4be5-ae6b-99091a1d595b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "61456fa8-cbab-4be5-ae6b-99091a1d595b" (UID: "61456fa8-cbab-4be5-ae6b-99091a1d595b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:48:20.864096 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:20.863972    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61456fa8-cbab-4be5-ae6b-99091a1d595b-home\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:48:20.865897 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:20.865875    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61456fa8-cbab-4be5-ae6b-99091a1d595b-kube-api-access-2n9s5" (OuterVolumeSpecName: "kube-api-access-2n9s5") pod "61456fa8-cbab-4be5-ae6b-99091a1d595b" (UID: "61456fa8-cbab-4be5-ae6b-99091a1d595b"). InnerVolumeSpecName "kube-api-access-2n9s5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:48:20.865986 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:20.865877    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61456fa8-cbab-4be5-ae6b-99091a1d595b-dshm" (OuterVolumeSpecName: "dshm") pod "61456fa8-cbab-4be5-ae6b-99091a1d595b" (UID: "61456fa8-cbab-4be5-ae6b-99091a1d595b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:48:20.865986 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:20.865952    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61456fa8-cbab-4be5-ae6b-99091a1d595b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "61456fa8-cbab-4be5-ae6b-99091a1d595b" (UID: "61456fa8-cbab-4be5-ae6b-99091a1d595b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:48:20.965046 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:20.965006    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61456fa8-cbab-4be5-ae6b-99091a1d595b-dshm\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:48:20.965046 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:20.965040    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61456fa8-cbab-4be5-ae6b-99091a1d595b-model-cache\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:48:20.965046 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:20.965050    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61456fa8-cbab-4be5-ae6b-99091a1d595b-tmp-dir\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:48:20.965283 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:20.965059    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n9s5\" (UniqueName: \"kubernetes.io/projected/61456fa8-cbab-4be5-ae6b-99091a1d595b-kube-api-access-2n9s5\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:48:20.965283 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:20.965068    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61456fa8-cbab-4be5-ae6b-99091a1d595b-tls-certs\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:48:21.104874 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:21.104836    2569 generic.go:356] "Generic (PLEG): container finished" podID="61456fa8-cbab-4be5-ae6b-99091a1d595b" containerID="ccd79cfb6bfd48986defc6e5844a7bd4013bfb03282a6fa743a67092ab86b843" exitCode=0
Sep 18 17:48:21.105043 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:21.104900    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd"
Sep 18 17:48:21.105043 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:21.104900    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd" event={"ID":"61456fa8-cbab-4be5-ae6b-99091a1d595b","Type":"ContainerDied","Data":"ccd79cfb6bfd48986defc6e5844a7bd4013bfb03282a6fa743a67092ab86b843"}
Sep 18 17:48:21.105043 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:21.104950    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd" event={"ID":"61456fa8-cbab-4be5-ae6b-99091a1d595b","Type":"ContainerDied","Data":"a96601d5d4f97b86e4fb51dd7751bc6396eb2f58891c63f897ff260c4fbfc4c1"}
Sep 18 17:48:21.105043 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:21.104972    2569 scope.go:117] "RemoveContainer" containerID="ccd79cfb6bfd48986defc6e5844a7bd4013bfb03282a6fa743a67092ab86b843"
Sep 18 17:48:21.113598 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:21.113555    2569 scope.go:117] "RemoveContainer" containerID="ccd79cfb6bfd48986defc6e5844a7bd4013bfb03282a6fa743a67092ab86b843"
Sep 18 17:48:21.113877 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:48:21.113840    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccd79cfb6bfd48986defc6e5844a7bd4013bfb03282a6fa743a67092ab86b843\": container with ID starting with ccd79cfb6bfd48986defc6e5844a7bd4013bfb03282a6fa743a67092ab86b843 not found: ID does not exist" containerID="ccd79cfb6bfd48986defc6e5844a7bd4013bfb03282a6fa743a67092ab86b843"
Sep 18 17:48:21.113942 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:21.113902    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccd79cfb6bfd48986defc6e5844a7bd4013bfb03282a6fa743a67092ab86b843"} err="failed to get container status \"ccd79cfb6bfd48986defc6e5844a7bd4013bfb03282a6fa743a67092ab86b843\": rpc error: code = NotFound desc = could not find container \"ccd79cfb6bfd48986defc6e5844a7bd4013bfb03282a6fa743a67092ab86b843\": container with ID starting with ccd79cfb6bfd48986defc6e5844a7bd4013bfb03282a6fa743a67092ab86b843 not found: ID does not exist"
Sep 18 17:48:21.127472 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:21.127449    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd"]
Sep 18 17:48:21.131134 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:21.131109    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-828a86/ipp-route-828a86-kserve-76cd6cc67f-hrvvd"]
Sep 18 17:48:21.764637 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:21.764601    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61456fa8-cbab-4be5-ae6b-99091a1d595b" path="/var/lib/kubelet/pods/61456fa8-cbab-4be5-ae6b-99091a1d595b/volumes"
Sep 18 17:48:56.223882 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:56.223827    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8bdd6-dc2jq"]
Sep 18 17:48:56.224480 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:56.224303    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54721180-3e18-4f55-8d2b-a611b57b9c49" containerName="revoke-keys"
Sep 18 17:48:56.224480 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:56.224324    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="54721180-3e18-4f55-8d2b-a611b57b9c49" containerName="revoke-keys"
Sep 18 17:48:56.224480 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:56.224340    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69a7f0df-3f15-4367-91ff-79f0267c7ac4" containerName="maas-api"
Sep 18 17:48:56.224480 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:56.224349    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="69a7f0df-3f15-4367-91ff-79f0267c7ac4" containerName="maas-api"
Sep 18 17:48:56.224480 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:56.224370    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58f0a29e-9531-4b07-a8dd-41cafe6c74d6" containerName="payload-processing"
Sep 18 17:48:56.224480 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:56.224382    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="58f0a29e-9531-4b07-a8dd-41cafe6c74d6" containerName="payload-processing"
Sep 18 17:48:56.224480 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:56.224399    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70a303db-3b6a-4cb5-81b2-4dea8288c66f" containerName="payload-pre-processing"
Sep 18 17:48:56.224480 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:56.224409    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="70a303db-3b6a-4cb5-81b2-4dea8288c66f" containerName="payload-pre-processing"
Sep 18 17:48:56.224480 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:56.224422    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61456fa8-cbab-4be5-ae6b-99091a1d595b" containerName="main"
Sep 18 17:48:56.224480 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:56.224431    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="61456fa8-cbab-4be5-ae6b-99091a1d595b" containerName="main"
Sep 18 17:48:56.225011 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:56.224519    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="70a303db-3b6a-4cb5-81b2-4dea8288c66f" containerName="payload-pre-processing"
Sep 18 17:48:56.225011 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:56.224534    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="61456fa8-cbab-4be5-ae6b-99091a1d595b" containerName="main"
Sep 18 17:48:56.225011 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:56.224543    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="69a7f0df-3f15-4367-91ff-79f0267c7ac4" containerName="maas-api"
Sep 18 17:48:56.225011 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:56.224552    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="58f0a29e-9531-4b07-a8dd-41cafe6c74d6" containerName="payload-processing"
Sep 18 17:48:56.225011 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:56.224563    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="54721180-3e18-4f55-8d2b-a611b57b9c49" containerName="revoke-keys"
Sep 18 17:48:56.227615 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:56.227592    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8bdd6-dc2jq"
Sep 18 17:48:56.230193 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:56.230172    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qt978\""
Sep 18 17:48:56.239008 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:56.238983    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8bdd6-dc2jq"]
Sep 18 17:48:56.372408 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:56.372365    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a84918e0-035e-4f9f-b93c-1897e4833e66-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a8bdd6-dc2jq\" (UID: \"a84918e0-035e-4f9f-b93c-1897e4833e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8bdd6-dc2jq"
Sep 18 17:48:56.473406 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:56.473370    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a84918e0-035e-4f9f-b93c-1897e4833e66-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a8bdd6-dc2jq\" (UID: \"a84918e0-035e-4f9f-b93c-1897e4833e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8bdd6-dc2jq"
Sep 18 17:48:56.474113 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:56.474052    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a84918e0-035e-4f9f-b93c-1897e4833e66-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a8bdd6-dc2jq\" (UID: \"a84918e0-035e-4f9f-b93c-1897e4833e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8bdd6-dc2jq"
Sep 18 17:48:56.538549 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:56.538515    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8bdd6-dc2jq"
Sep 18 17:48:56.827753 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:56.827717    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8bdd6-dc2jq"]
Sep 18 17:48:56.828102 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:48:56.828079    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda84918e0_035e_4f9f_b93c_1897e4833e66.slice/crio-99a56e504a7ce5553a2c65f07e98a40c3acaabf149adb9d741ceac9bae3562a0 WatchSource:0}: Error finding container 99a56e504a7ce5553a2c65f07e98a40c3acaabf149adb9d741ceac9bae3562a0: Status 404 returned error can't find the container with id 99a56e504a7ce5553a2c65f07e98a40c3acaabf149adb9d741ceac9bae3562a0
Sep 18 17:48:57.277527 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:57.277494    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8bdd6-dc2jq" event={"ID":"a84918e0-035e-4f9f-b93c-1897e4833e66","Type":"ContainerStarted","Data":"15ef1e934d20654ea4100085178582c1f072d7e169f6c8a7cc0975d215901b2a"}
Sep 18 17:48:57.278011 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:57.277544    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8bdd6-dc2jq" event={"ID":"a84918e0-035e-4f9f-b93c-1897e4833e66","Type":"ContainerStarted","Data":"99a56e504a7ce5553a2c65f07e98a40c3acaabf149adb9d741ceac9bae3562a0"}
Sep 18 17:48:57.321081 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:57.321025    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8bdd6-dc2jq" podStartSLOduration=1.321009805 podStartE2EDuration="1.321009805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:48:57.319808739 +0000 UTC m=+2050.013736267" watchObservedRunningTime="2026-09-18 17:48:57.321009805 +0000 UTC m=+2050.014937333"
Sep 18 17:48:58.283003 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:58.282966    2569 generic.go:356] "Generic (PLEG): container finished" podID="a84918e0-035e-4f9f-b93c-1897e4833e66" containerID="15ef1e934d20654ea4100085178582c1f072d7e169f6c8a7cc0975d215901b2a" exitCode=0
Sep 18 17:48:58.283432 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:58.283056    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8bdd6-dc2jq" event={"ID":"a84918e0-035e-4f9f-b93c-1897e4833e66","Type":"ContainerDied","Data":"15ef1e934d20654ea4100085178582c1f072d7e169f6c8a7cc0975d215901b2a"}
Sep 18 17:48:59.415958 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:59.415929    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8bdd6-dc2jq"
Sep 18 17:48:59.500977 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:59.500942    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a84918e0-035e-4f9f-b93c-1897e4833e66-maas-api-service-ca\") pod \"a84918e0-035e-4f9f-b93c-1897e4833e66\" (UID: \"a84918e0-035e-4f9f-b93c-1897e4833e66\") "
Sep 18 17:48:59.501295 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:59.501273    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a84918e0-035e-4f9f-b93c-1897e4833e66-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a84918e0-035e-4f9f-b93c-1897e4833e66" (UID: "a84918e0-035e-4f9f-b93c-1897e4833e66"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:48:59.601506 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:48:59.601408    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a84918e0-035e-4f9f-b93c-1897e4833e66-maas-api-service-ca\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:49:00.294374 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:00.294344    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8bdd6-dc2jq"
Sep 18 17:49:00.294575 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:00.294373    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8bdd6-dc2jq" event={"ID":"a84918e0-035e-4f9f-b93c-1897e4833e66","Type":"ContainerDied","Data":"99a56e504a7ce5553a2c65f07e98a40c3acaabf149adb9d741ceac9bae3562a0"}
Sep 18 17:49:00.294575 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:00.294413    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99a56e504a7ce5553a2c65f07e98a40c3acaabf149adb9d741ceac9bae3562a0"
Sep 18 17:49:06.897781 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:06.897749    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj"]
Sep 18 17:49:06.898169 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:06.897983    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj" podUID="f5c54736-0b79-4269-85e1-eb478223401a" containerName="maas-api" containerID="cri-o://5ea4c198682b20e8a3130c1d245f024cb13374f27dd3eb0a26a5c5d705f2c90f" gracePeriod=30
Sep 18 17:49:06.988231 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:06.988194    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8bdd6-5b8b68d6b-tzl8m"]
Sep 18 17:49:06.988407 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:06.988386    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8bdd6-5b8b68d6b-tzl8m" podUID="09808280-044f-42e3-90a4-d0da31ce0420" containerName="payload-pre-processing" containerID="cri-o://2c6850e8eb95d6de33d03a84d6ddd9ac8365827b9960e535f1900611cd9efcec" gracePeriod=30
Sep 18 17:49:07.181767 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.181683    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8bdd6-dc2jq"]
Sep 18 17:49:07.183394 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.183362    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8bdd6-dc2jq"]
Sep 18 17:49:07.234078 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.234052    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj"
Sep 18 17:49:07.264261 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.264230    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8bdd6-5b8b68d6b-tzl8m"
Sep 18 17:49:07.265035 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.265012    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8r28\" (UniqueName: \"kubernetes.io/projected/f5c54736-0b79-4269-85e1-eb478223401a-kube-api-access-j8r28\") pod \"f5c54736-0b79-4269-85e1-eb478223401a\" (UID: \"f5c54736-0b79-4269-85e1-eb478223401a\") "
Sep 18 17:49:07.265160 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.265060    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5c54736-0b79-4269-85e1-eb478223401a-maas-api-tls\") pod \"f5c54736-0b79-4269-85e1-eb478223401a\" (UID: \"f5c54736-0b79-4269-85e1-eb478223401a\") "
Sep 18 17:49:07.265160 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.265084    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5c54736-0b79-4269-85e1-eb478223401a-openshift-service-ca\") pod \"f5c54736-0b79-4269-85e1-eb478223401a\" (UID: \"f5c54736-0b79-4269-85e1-eb478223401a\") "
Sep 18 17:49:07.265160 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.265146    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5c54736-0b79-4269-85e1-eb478223401a-metrics-certs\") pod \"f5c54736-0b79-4269-85e1-eb478223401a\" (UID: \"f5c54736-0b79-4269-85e1-eb478223401a\") "
Sep 18 17:49:07.265573 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.265544    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5c54736-0b79-4269-85e1-eb478223401a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f5c54736-0b79-4269-85e1-eb478223401a" (UID: "f5c54736-0b79-4269-85e1-eb478223401a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:49:07.267292 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.267268    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c54736-0b79-4269-85e1-eb478223401a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f5c54736-0b79-4269-85e1-eb478223401a" (UID: "f5c54736-0b79-4269-85e1-eb478223401a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:49:07.267376 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.267254    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c54736-0b79-4269-85e1-eb478223401a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f5c54736-0b79-4269-85e1-eb478223401a" (UID: "f5c54736-0b79-4269-85e1-eb478223401a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:49:07.267376 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.267303    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5c54736-0b79-4269-85e1-eb478223401a-kube-api-access-j8r28" (OuterVolumeSpecName: "kube-api-access-j8r28") pod "f5c54736-0b79-4269-85e1-eb478223401a" (UID: "f5c54736-0b79-4269-85e1-eb478223401a"). InnerVolumeSpecName "kube-api-access-j8r28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:49:07.330324 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.330286    2569 generic.go:356] "Generic (PLEG): container finished" podID="09808280-044f-42e3-90a4-d0da31ce0420" containerID="2c6850e8eb95d6de33d03a84d6ddd9ac8365827b9960e535f1900611cd9efcec" exitCode=0
Sep 18 17:49:07.330535 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.330330    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8bdd6-5b8b68d6b-tzl8m"
Sep 18 17:49:07.330535 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.330375    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8bdd6-5b8b68d6b-tzl8m" event={"ID":"09808280-044f-42e3-90a4-d0da31ce0420","Type":"ContainerDied","Data":"2c6850e8eb95d6de33d03a84d6ddd9ac8365827b9960e535f1900611cd9efcec"}
Sep 18 17:49:07.330535 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.330472    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8bdd6-5b8b68d6b-tzl8m" event={"ID":"09808280-044f-42e3-90a4-d0da31ce0420","Type":"ContainerDied","Data":"6f6e00af9eff33f50d61680ebd4f6f83f812639827c7a89af7fde41f32b38529"}
Sep 18 17:49:07.330535 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.330510    2569 scope.go:117] "RemoveContainer" containerID="2c6850e8eb95d6de33d03a84d6ddd9ac8365827b9960e535f1900611cd9efcec"
Sep 18 17:49:07.331930 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.331910    2569 generic.go:356] "Generic (PLEG): container finished" podID="f5c54736-0b79-4269-85e1-eb478223401a" containerID="5ea4c198682b20e8a3130c1d245f024cb13374f27dd3eb0a26a5c5d705f2c90f" exitCode=0
Sep 18 17:49:07.332070 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.331960    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj"
Sep 18 17:49:07.332070 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.331994    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj" event={"ID":"f5c54736-0b79-4269-85e1-eb478223401a","Type":"ContainerDied","Data":"5ea4c198682b20e8a3130c1d245f024cb13374f27dd3eb0a26a5c5d705f2c90f"}
Sep 18 17:49:07.332070 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.332059    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj" event={"ID":"f5c54736-0b79-4269-85e1-eb478223401a","Type":"ContainerDied","Data":"966f0383d5cc5f0c9d6b492701840617fbf13289dbd3ff3515ec8805888c8d1f"}
Sep 18 17:49:07.340387 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.340369    2569 scope.go:117] "RemoveContainer" containerID="2c6850e8eb95d6de33d03a84d6ddd9ac8365827b9960e535f1900611cd9efcec"
Sep 18 17:49:07.340689 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:49:07.340670    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c6850e8eb95d6de33d03a84d6ddd9ac8365827b9960e535f1900611cd9efcec\": container with ID starting with 2c6850e8eb95d6de33d03a84d6ddd9ac8365827b9960e535f1900611cd9efcec not found: ID does not exist" containerID="2c6850e8eb95d6de33d03a84d6ddd9ac8365827b9960e535f1900611cd9efcec"
Sep 18 17:49:07.340776 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.340696    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c6850e8eb95d6de33d03a84d6ddd9ac8365827b9960e535f1900611cd9efcec"} err="failed to get container status \"2c6850e8eb95d6de33d03a84d6ddd9ac8365827b9960e535f1900611cd9efcec\": rpc error: code = NotFound desc = could not find container \"2c6850e8eb95d6de33d03a84d6ddd9ac8365827b9960e535f1900611cd9efcec\": container with ID starting with 2c6850e8eb95d6de33d03a84d6ddd9ac8365827b9960e535f1900611cd9efcec not found: ID does not exist"
Sep 18 17:49:07.340776 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.340713    2569 scope.go:117] "RemoveContainer" containerID="5ea4c198682b20e8a3130c1d245f024cb13374f27dd3eb0a26a5c5d705f2c90f"
Sep 18 17:49:07.348608 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.348590    2569 scope.go:117] "RemoveContainer" containerID="5ea4c198682b20e8a3130c1d245f024cb13374f27dd3eb0a26a5c5d705f2c90f"
Sep 18 17:49:07.348883 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:49:07.348852    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ea4c198682b20e8a3130c1d245f024cb13374f27dd3eb0a26a5c5d705f2c90f\": container with ID starting with 5ea4c198682b20e8a3130c1d245f024cb13374f27dd3eb0a26a5c5d705f2c90f not found: ID does not exist" containerID="5ea4c198682b20e8a3130c1d245f024cb13374f27dd3eb0a26a5c5d705f2c90f"
Sep 18 17:49:07.348941 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.348891    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ea4c198682b20e8a3130c1d245f024cb13374f27dd3eb0a26a5c5d705f2c90f"} err="failed to get container status \"5ea4c198682b20e8a3130c1d245f024cb13374f27dd3eb0a26a5c5d705f2c90f\": rpc error: code = NotFound desc = could not find container \"5ea4c198682b20e8a3130c1d245f024cb13374f27dd3eb0a26a5c5d705f2c90f\": container with ID starting with 5ea4c198682b20e8a3130c1d245f024cb13374f27dd3eb0a26a5c5d705f2c90f not found: ID does not exist"
Sep 18 17:49:07.360099 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.360073    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj"]
Sep 18 17:49:07.364485 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.364467    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8bdd6-56bdd6c7fc-4r6zj"]
Sep 18 17:49:07.365850 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.365834    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sz5x9\" (UniqueName: \"kubernetes.io/projected/09808280-044f-42e3-90a4-d0da31ce0420-kube-api-access-sz5x9\") pod \"09808280-044f-42e3-90a4-d0da31ce0420\" (UID: \"09808280-044f-42e3-90a4-d0da31ce0420\") "
Sep 18 17:49:07.365920 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.365891    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09808280-044f-42e3-90a4-d0da31ce0420-plugins-config-volume\") pod \"09808280-044f-42e3-90a4-d0da31ce0420\" (UID: \"09808280-044f-42e3-90a4-d0da31ce0420\") "
Sep 18 17:49:07.366122 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.366108    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5c54736-0b79-4269-85e1-eb478223401a-openshift-service-ca\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:49:07.366122 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.366121    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5c54736-0b79-4269-85e1-eb478223401a-metrics-certs\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:49:07.366257 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.366130    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8r28\" (UniqueName: \"kubernetes.io/projected/f5c54736-0b79-4269-85e1-eb478223401a-kube-api-access-j8r28\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:49:07.366257 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.366139    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5c54736-0b79-4269-85e1-eb478223401a-maas-api-tls\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:49:07.366257 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.366241    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09808280-044f-42e3-90a4-d0da31ce0420-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "09808280-044f-42e3-90a4-d0da31ce0420" (UID: "09808280-044f-42e3-90a4-d0da31ce0420"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:49:07.367794 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.367774    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09808280-044f-42e3-90a4-d0da31ce0420-kube-api-access-sz5x9" (OuterVolumeSpecName: "kube-api-access-sz5x9") pod "09808280-044f-42e3-90a4-d0da31ce0420" (UID: "09808280-044f-42e3-90a4-d0da31ce0420"). InnerVolumeSpecName "kube-api-access-sz5x9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:49:07.466620 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.466535    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz5x9\" (UniqueName: \"kubernetes.io/projected/09808280-044f-42e3-90a4-d0da31ce0420-kube-api-access-sz5x9\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:49:07.466620 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.466565    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09808280-044f-42e3-90a4-d0da31ce0420-plugins-config-volume\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:49:07.652260 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.652222    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8bdd6-5b8b68d6b-tzl8m"]
Sep 18 17:49:07.661428 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.661400    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8bdd6-5b8b68d6b-tzl8m"]
Sep 18 17:49:07.764476 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.764447    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09808280-044f-42e3-90a4-d0da31ce0420" path="/var/lib/kubelet/pods/09808280-044f-42e3-90a4-d0da31ce0420/volumes"
Sep 18 17:49:07.764791 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.764778    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a84918e0-035e-4f9f-b93c-1897e4833e66" path="/var/lib/kubelet/pods/a84918e0-035e-4f9f-b93c-1897e4833e66/volumes"
Sep 18 17:49:07.765181 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:07.765168    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5c54736-0b79-4269-85e1-eb478223401a" path="/var/lib/kubelet/pods/f5c54736-0b79-4269-85e1-eb478223401a/volumes"
Sep 18 17:49:08.258718 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:08.258679    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8bdd6-5b8b68d6b-tzl8m" podUID="09808280-044f-42e3-90a4-d0da31ce0420" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.110:9004: i/o timeout"
Sep 18 17:49:14.868545 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:14.868466    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eeab6e-cxx7s"]
Sep 18 17:49:14.868934 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:14.868795    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09808280-044f-42e3-90a4-d0da31ce0420" containerName="payload-pre-processing"
Sep 18 17:49:14.868934 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:14.868805    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="09808280-044f-42e3-90a4-d0da31ce0420" containerName="payload-pre-processing"
Sep 18 17:49:14.868934 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:14.868824    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a84918e0-035e-4f9f-b93c-1897e4833e66" containerName="revoke-keys"
Sep 18 17:49:14.868934 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:14.868830    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a84918e0-035e-4f9f-b93c-1897e4833e66" containerName="revoke-keys"
Sep 18 17:49:14.868934 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:14.868838    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5c54736-0b79-4269-85e1-eb478223401a" containerName="maas-api"
Sep 18 17:49:14.868934 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:14.868843    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5c54736-0b79-4269-85e1-eb478223401a" containerName="maas-api"
Sep 18 17:49:14.868934 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:14.868914    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5c54736-0b79-4269-85e1-eb478223401a" containerName="maas-api"
Sep 18 17:49:14.868934 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:14.868922    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="09808280-044f-42e3-90a4-d0da31ce0420" containerName="payload-pre-processing"
Sep 18 17:49:14.868934 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:14.868929    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a84918e0-035e-4f9f-b93c-1897e4833e66" containerName="revoke-keys"
Sep 18 17:49:14.873310 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:14.873294    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eeab6e-cxx7s"
Sep 18 17:49:14.876220 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:14.876198    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qt978\""
Sep 18 17:49:14.877182 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:14.877164    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 17:49:14.890526 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:14.890503    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eeab6e-cxx7s"]
Sep 18 17:49:14.930923 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:14.930892    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94ecb00e-d094-4af5-b7fb-dcecaf85de28-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-eeab6e-cxx7s\" (UID: \"94ecb00e-d094-4af5-b7fb-dcecaf85de28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eeab6e-cxx7s"
Sep 18 17:49:15.031581 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:15.031546    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94ecb00e-d094-4af5-b7fb-dcecaf85de28-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-eeab6e-cxx7s\" (UID: \"94ecb00e-d094-4af5-b7fb-dcecaf85de28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eeab6e-cxx7s"
Sep 18 17:49:15.032202 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:15.032180    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94ecb00e-d094-4af5-b7fb-dcecaf85de28-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-eeab6e-cxx7s\" (UID: \"94ecb00e-d094-4af5-b7fb-dcecaf85de28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eeab6e-cxx7s"
Sep 18 17:49:15.183322 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:15.183240    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eeab6e-cxx7s"
Sep 18 17:49:15.500785 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:15.500761    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eeab6e-cxx7s"]
Sep 18 17:49:15.503346 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:49:15.503320    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ecb00e_d094_4af5_b7fb_dcecaf85de28.slice/crio-f999a5d86ab8443f6d87ea7860f4271850738247c23ef6c3a58eab312bc0d7cb WatchSource:0}: Error finding container f999a5d86ab8443f6d87ea7860f4271850738247c23ef6c3a58eab312bc0d7cb: Status 404 returned error can't find the container with id f999a5d86ab8443f6d87ea7860f4271850738247c23ef6c3a58eab312bc0d7cb
Sep 18 17:49:16.379907 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:16.379846    2569 generic.go:356] "Generic (PLEG): container finished" podID="94ecb00e-d094-4af5-b7fb-dcecaf85de28" containerID="cfff776f6e33af07e84147c8ee980a0c27bb26853aeb40301ce545469fed58a3" exitCode=0
Sep 18 17:49:16.380303 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:16.379938    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eeab6e-cxx7s" event={"ID":"94ecb00e-d094-4af5-b7fb-dcecaf85de28","Type":"ContainerDied","Data":"cfff776f6e33af07e84147c8ee980a0c27bb26853aeb40301ce545469fed58a3"}
Sep 18 17:49:16.380303 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:16.380002    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eeab6e-cxx7s" event={"ID":"94ecb00e-d094-4af5-b7fb-dcecaf85de28","Type":"ContainerStarted","Data":"f999a5d86ab8443f6d87ea7860f4271850738247c23ef6c3a58eab312bc0d7cb"}
Sep 18 17:49:17.508053 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:17.508026    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eeab6e-cxx7s"
Sep 18 17:49:17.553124 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:17.553091    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94ecb00e-d094-4af5-b7fb-dcecaf85de28-maas-api-service-ca\") pod \"94ecb00e-d094-4af5-b7fb-dcecaf85de28\" (UID: \"94ecb00e-d094-4af5-b7fb-dcecaf85de28\") "
Sep 18 17:49:17.553422 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:17.553400    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94ecb00e-d094-4af5-b7fb-dcecaf85de28-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "94ecb00e-d094-4af5-b7fb-dcecaf85de28" (UID: "94ecb00e-d094-4af5-b7fb-dcecaf85de28"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:49:17.653697 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:17.653617    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94ecb00e-d094-4af5-b7fb-dcecaf85de28-maas-api-service-ca\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:49:18.390924 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:18.390858    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eeab6e-cxx7s" event={"ID":"94ecb00e-d094-4af5-b7fb-dcecaf85de28","Type":"ContainerDied","Data":"f999a5d86ab8443f6d87ea7860f4271850738247c23ef6c3a58eab312bc0d7cb"}
Sep 18 17:49:18.391136 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:18.390943    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f999a5d86ab8443f6d87ea7860f4271850738247c23ef6c3a58eab312bc0d7cb"
Sep 18 17:49:18.391136 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:18.390895    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eeab6e-cxx7s"
Sep 18 17:49:25.464029 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:25.463996    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-eeab6e-645b4665b7-h78xd"]
Sep 18 17:49:25.464474 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:25.464193    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eeab6e-645b4665b7-h78xd" podUID="e5fd32a3-79f9-4b0a-9b59-a8f6866838c5" containerName="payload-processing" containerID="cri-o://ef2a7bfdc4112f009ec8aa4c945dcb14488cda1229571f53394eb7c997d651ea" gracePeriod=30
Sep 18 17:49:25.645526 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:25.645498    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eeab6e-cxx7s"]
Sep 18 17:49:25.650305 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:25.650279    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-eeab6e-cxx7s"]
Sep 18 17:49:25.714672 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:25.714610    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eeab6e-645b4665b7-h78xd"
Sep 18 17:49:25.766144 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:25.766112    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94ecb00e-d094-4af5-b7fb-dcecaf85de28" path="/var/lib/kubelet/pods/94ecb00e-d094-4af5-b7fb-dcecaf85de28/volumes"
Sep 18 17:49:25.823316 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:25.823287    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5fd32a3-79f9-4b0a-9b59-a8f6866838c5-plugins-config-volume\") pod \"e5fd32a3-79f9-4b0a-9b59-a8f6866838c5\" (UID: \"e5fd32a3-79f9-4b0a-9b59-a8f6866838c5\") "
Sep 18 17:49:25.823525 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:25.823505    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7trxg\" (UniqueName: \"kubernetes.io/projected/e5fd32a3-79f9-4b0a-9b59-a8f6866838c5-kube-api-access-7trxg\") pod \"e5fd32a3-79f9-4b0a-9b59-a8f6866838c5\" (UID: \"e5fd32a3-79f9-4b0a-9b59-a8f6866838c5\") "
Sep 18 17:49:25.823732 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:25.823710    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5fd32a3-79f9-4b0a-9b59-a8f6866838c5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e5fd32a3-79f9-4b0a-9b59-a8f6866838c5" (UID: "e5fd32a3-79f9-4b0a-9b59-a8f6866838c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:49:25.823815 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:25.823790    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5fd32a3-79f9-4b0a-9b59-a8f6866838c5-plugins-config-volume\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:49:25.825622 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:25.825598    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5fd32a3-79f9-4b0a-9b59-a8f6866838c5-kube-api-access-7trxg" (OuterVolumeSpecName: "kube-api-access-7trxg") pod "e5fd32a3-79f9-4b0a-9b59-a8f6866838c5" (UID: "e5fd32a3-79f9-4b0a-9b59-a8f6866838c5"). InnerVolumeSpecName "kube-api-access-7trxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:49:25.924692 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:25.924663    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7trxg\" (UniqueName: \"kubernetes.io/projected/e5fd32a3-79f9-4b0a-9b59-a8f6866838c5-kube-api-access-7trxg\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:49:26.428800 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:26.428717    2569 generic.go:356] "Generic (PLEG): container finished" podID="e5fd32a3-79f9-4b0a-9b59-a8f6866838c5" containerID="ef2a7bfdc4112f009ec8aa4c945dcb14488cda1229571f53394eb7c997d651ea" exitCode=0
Sep 18 17:49:26.428800 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:26.428765    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eeab6e-645b4665b7-h78xd"
Sep 18 17:49:26.429037 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:26.428808    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eeab6e-645b4665b7-h78xd" event={"ID":"e5fd32a3-79f9-4b0a-9b59-a8f6866838c5","Type":"ContainerDied","Data":"ef2a7bfdc4112f009ec8aa4c945dcb14488cda1229571f53394eb7c997d651ea"}
Sep 18 17:49:26.429037 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:26.428889    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-eeab6e-645b4665b7-h78xd" event={"ID":"e5fd32a3-79f9-4b0a-9b59-a8f6866838c5","Type":"ContainerDied","Data":"88cc3905ee24853dc74c53afb66fd54793f823c1eaee57e8f2be6ab1361e3a45"}
Sep 18 17:49:26.429037 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:26.428919    2569 scope.go:117] "RemoveContainer" containerID="ef2a7bfdc4112f009ec8aa4c945dcb14488cda1229571f53394eb7c997d651ea"
Sep 18 17:49:26.438767 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:26.438749    2569 scope.go:117] "RemoveContainer" containerID="ef2a7bfdc4112f009ec8aa4c945dcb14488cda1229571f53394eb7c997d651ea"
Sep 18 17:49:26.439065 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:49:26.439045    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef2a7bfdc4112f009ec8aa4c945dcb14488cda1229571f53394eb7c997d651ea\": container with ID starting with ef2a7bfdc4112f009ec8aa4c945dcb14488cda1229571f53394eb7c997d651ea not found: ID does not exist" containerID="ef2a7bfdc4112f009ec8aa4c945dcb14488cda1229571f53394eb7c997d651ea"
Sep 18 17:49:26.439125 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:26.439073    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef2a7bfdc4112f009ec8aa4c945dcb14488cda1229571f53394eb7c997d651ea"} err="failed to get container status \"ef2a7bfdc4112f009ec8aa4c945dcb14488cda1229571f53394eb7c997d651ea\": rpc error: code = NotFound desc = could not find container \"ef2a7bfdc4112f009ec8aa4c945dcb14488cda1229571f53394eb7c997d651ea\": container with ID starting with ef2a7bfdc4112f009ec8aa4c945dcb14488cda1229571f53394eb7c997d651ea not found: ID does not exist"
Sep 18 17:49:26.458955 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:26.458933    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-eeab6e-645b4665b7-h78xd"]
Sep 18 17:49:26.468484 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:26.468460    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-eeab6e-645b4665b7-h78xd"]
Sep 18 17:49:27.764441 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:27.764409    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5fd32a3-79f9-4b0a-9b59-a8f6866838c5" path="/var/lib/kubelet/pods/e5fd32a3-79f9-4b0a-9b59-a8f6866838c5/volumes"
Sep 18 17:49:32.959389 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:32.959355    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-808470-c6gc7"]
Sep 18 17:49:32.959769 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:32.959647    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5fd32a3-79f9-4b0a-9b59-a8f6866838c5" containerName="payload-processing"
Sep 18 17:49:32.959769 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:32.959658    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5fd32a3-79f9-4b0a-9b59-a8f6866838c5" containerName="payload-processing"
Sep 18 17:49:32.959769 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:32.959679    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94ecb00e-d094-4af5-b7fb-dcecaf85de28" containerName="revoke-keys"
Sep 18 17:49:32.959769 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:32.959684    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="94ecb00e-d094-4af5-b7fb-dcecaf85de28" containerName="revoke-keys"
Sep 18 17:49:32.959769 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:32.959733    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="94ecb00e-d094-4af5-b7fb-dcecaf85de28" containerName="revoke-keys"
Sep 18 17:49:32.959769 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:32.959740    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5fd32a3-79f9-4b0a-9b59-a8f6866838c5" containerName="payload-processing"
Sep 18 17:49:32.964060 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:32.964042    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-808470-c6gc7"
Sep 18 17:49:32.969334 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:32.969207    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 17:49:32.969334 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:32.969246    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qt978\""
Sep 18 17:49:32.972920 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:32.972891    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-808470-c6gc7"]
Sep 18 17:49:33.088134 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:33.088101    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1789f9-e3af-432f-9fc2-ca89405baa11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-808470-c6gc7\" (UID: \"3c1789f9-e3af-432f-9fc2-ca89405baa11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-808470-c6gc7"
Sep 18 17:49:33.188720 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:33.188686    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1789f9-e3af-432f-9fc2-ca89405baa11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-808470-c6gc7\" (UID: \"3c1789f9-e3af-432f-9fc2-ca89405baa11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-808470-c6gc7"
Sep 18 17:49:33.189360 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:33.189336    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1789f9-e3af-432f-9fc2-ca89405baa11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-808470-c6gc7\" (UID: \"3c1789f9-e3af-432f-9fc2-ca89405baa11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-808470-c6gc7"
Sep 18 17:49:33.275663 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:33.275631    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-808470-c6gc7"
Sep 18 17:49:33.546157 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:33.546128    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-808470-c6gc7"]
Sep 18 17:49:33.548085 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:49:33.548056    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c1789f9_e3af_432f_9fc2_ca89405baa11.slice/crio-305264e4e4a9debd777e7cb6c77ee1775694c3c57ab8175b4eb99d601b92a83a WatchSource:0}: Error finding container 305264e4e4a9debd777e7cb6c77ee1775694c3c57ab8175b4eb99d601b92a83a: Status 404 returned error can't find the container with id 305264e4e4a9debd777e7cb6c77ee1775694c3c57ab8175b4eb99d601b92a83a
Sep 18 17:49:34.476336 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:34.476297    2569 generic.go:356] "Generic (PLEG): container finished" podID="3c1789f9-e3af-432f-9fc2-ca89405baa11" containerID="0561d5ef278917f75ee7af91e3b3e261c0add196231afb52c02915a23b1251b1" exitCode=0
Sep 18 17:49:34.476336 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:34.476340    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-808470-c6gc7" event={"ID":"3c1789f9-e3af-432f-9fc2-ca89405baa11","Type":"ContainerDied","Data":"0561d5ef278917f75ee7af91e3b3e261c0add196231afb52c02915a23b1251b1"}
Sep 18 17:49:34.476762 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:34.476408    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-808470-c6gc7" event={"ID":"3c1789f9-e3af-432f-9fc2-ca89405baa11","Type":"ContainerStarted","Data":"305264e4e4a9debd777e7cb6c77ee1775694c3c57ab8175b4eb99d601b92a83a"}
Sep 18 17:49:35.601584 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:35.601555    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-808470-c6gc7"
Sep 18 17:49:35.711875 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:35.711834    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1789f9-e3af-432f-9fc2-ca89405baa11-maas-api-service-ca\") pod \"3c1789f9-e3af-432f-9fc2-ca89405baa11\" (UID: \"3c1789f9-e3af-432f-9fc2-ca89405baa11\") "
Sep 18 17:49:35.712242 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:35.712216    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c1789f9-e3af-432f-9fc2-ca89405baa11-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3c1789f9-e3af-432f-9fc2-ca89405baa11" (UID: "3c1789f9-e3af-432f-9fc2-ca89405baa11"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:49:35.812458 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:35.812421    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1789f9-e3af-432f-9fc2-ca89405baa11-maas-api-service-ca\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:49:36.486700 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:36.486675    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-808470-c6gc7"
Sep 18 17:49:36.486908 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:36.486706    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-808470-c6gc7" event={"ID":"3c1789f9-e3af-432f-9fc2-ca89405baa11","Type":"ContainerDied","Data":"305264e4e4a9debd777e7cb6c77ee1775694c3c57ab8175b4eb99d601b92a83a"}
Sep 18 17:49:36.486908 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:36.486748    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="305264e4e4a9debd777e7cb6c77ee1775694c3c57ab8175b4eb99d601b92a83a"
Sep 18 17:49:43.857197 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:43.857157    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-808470-c6gc7"]
Sep 18 17:49:43.858892 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:43.858852    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-808470-c6gc7"]
Sep 18 17:49:45.764783 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:49:45.764752    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c1789f9-e3af-432f-9fc2-ca89405baa11" path="/var/lib/kubelet/pods/3c1789f9-e3af-432f-9fc2-ca89405baa11/volumes"
Sep 18 17:51:48.348570 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:51:48.348537    2569 scope.go:117] "RemoveContainer" containerID="d65628d111b7f80a002c0455308fc45562ce4d8e8213a60283fffa4b422e543e"
Sep 18 17:53:48.376010 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:53:48.375932    2569 scope.go:117] "RemoveContainer" containerID="c664cd3e3fc435537d3e480e0b0ef91437405dc92b1f0005a30fa375311979a3"
Sep 18 17:54:40.117555 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:40.117521    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9"]
Sep 18 17:54:40.117996 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:40.117851    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c1789f9-e3af-432f-9fc2-ca89405baa11" containerName="revoke-keys"
Sep 18 17:54:40.117996 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:40.117879    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c1789f9-e3af-432f-9fc2-ca89405baa11" containerName="revoke-keys"
Sep 18 17:54:40.117996 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:40.117937    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c1789f9-e3af-432f-9fc2-ca89405baa11" containerName="revoke-keys"
Sep 18 17:54:40.120765 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:40.120748    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9"
Sep 18 17:54:40.123618 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:40.123600    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-3844c469-kserve-self-signed-certs\""
Sep 18 17:54:40.133361 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:40.133336    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9"]
Sep 18 17:54:40.227240 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:40.227200    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3143ac28-e2fe-4164-8639-8b29ae7c7c21-model-cache\") pod \"e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9\" (UID: \"3143ac28-e2fe-4164-8639-8b29ae7c7c21\") " pod="llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9"
Sep 18 17:54:40.227443 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:40.227264    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3143ac28-e2fe-4164-8639-8b29ae7c7c21-tmp-dir\") pod \"e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9\" (UID: \"3143ac28-e2fe-4164-8639-8b29ae7c7c21\") " pod="llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9"
Sep 18 17:54:40.227443 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:40.227299    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3143ac28-e2fe-4164-8639-8b29ae7c7c21-home\") pod \"e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9\" (UID: \"3143ac28-e2fe-4164-8639-8b29ae7c7c21\") " pod="llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9"
Sep 18 17:54:40.227443 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:40.227345    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9lp5\" (UniqueName: \"kubernetes.io/projected/3143ac28-e2fe-4164-8639-8b29ae7c7c21-kube-api-access-h9lp5\") pod \"e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9\" (UID: \"3143ac28-e2fe-4164-8639-8b29ae7c7c21\") " pod="llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9"
Sep 18 17:54:40.227443 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:40.227377    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3143ac28-e2fe-4164-8639-8b29ae7c7c21-tls-certs\") pod \"e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9\" (UID: \"3143ac28-e2fe-4164-8639-8b29ae7c7c21\") " pod="llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9"
Sep 18 17:54:40.227443 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:40.227399    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3143ac28-e2fe-4164-8639-8b29ae7c7c21-dshm\") pod \"e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9\" (UID: \"3143ac28-e2fe-4164-8639-8b29ae7c7c21\") " pod="llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9"
Sep 18 17:54:40.328144 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:40.328109    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9lp5\" (UniqueName: \"kubernetes.io/projected/3143ac28-e2fe-4164-8639-8b29ae7c7c21-kube-api-access-h9lp5\") pod \"e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9\" (UID: \"3143ac28-e2fe-4164-8639-8b29ae7c7c21\") " pod="llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9"
Sep 18 17:54:40.328272 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:40.328153    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3143ac28-e2fe-4164-8639-8b29ae7c7c21-tls-certs\") pod \"e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9\" (UID: \"3143ac28-e2fe-4164-8639-8b29ae7c7c21\") " pod="llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9"
Sep 18 17:54:40.328272 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:40.328172    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3143ac28-e2fe-4164-8639-8b29ae7c7c21-dshm\") pod \"e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9\" (UID: \"3143ac28-e2fe-4164-8639-8b29ae7c7c21\") " pod="llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9"
Sep 18 17:54:40.328272 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:40.328198    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3143ac28-e2fe-4164-8639-8b29ae7c7c21-model-cache\") pod \"e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9\" (UID: \"3143ac28-e2fe-4164-8639-8b29ae7c7c21\") " pod="llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9"
Sep 18 17:54:40.328272 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:40.328230    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3143ac28-e2fe-4164-8639-8b29ae7c7c21-tmp-dir\") pod \"e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9\" (UID: \"3143ac28-e2fe-4164-8639-8b29ae7c7c21\") " pod="llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9"
Sep 18 17:54:40.328272 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:40.328252    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3143ac28-e2fe-4164-8639-8b29ae7c7c21-home\") pod \"e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9\" (UID: \"3143ac28-e2fe-4164-8639-8b29ae7c7c21\") " pod="llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9"
Sep 18 17:54:40.328683 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:40.328658    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3143ac28-e2fe-4164-8639-8b29ae7c7c21-tmp-dir\") pod \"e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9\" (UID: \"3143ac28-e2fe-4164-8639-8b29ae7c7c21\") " pod="llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9"
Sep 18 17:54:40.328751 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:40.328668    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3143ac28-e2fe-4164-8639-8b29ae7c7c21-home\") pod \"e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9\" (UID: \"3143ac28-e2fe-4164-8639-8b29ae7c7c21\") " pod="llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9"
Sep 18 17:54:40.328751 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:40.328710    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3143ac28-e2fe-4164-8639-8b29ae7c7c21-model-cache\") pod \"e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9\" (UID: \"3143ac28-e2fe-4164-8639-8b29ae7c7c21\") " pod="llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9"
Sep 18 17:54:40.330448 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:40.330426    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3143ac28-e2fe-4164-8639-8b29ae7c7c21-dshm\") pod \"e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9\" (UID: \"3143ac28-e2fe-4164-8639-8b29ae7c7c21\") " pod="llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9"
Sep 18 17:54:40.330745 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:40.330727    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3143ac28-e2fe-4164-8639-8b29ae7c7c21-tls-certs\") pod \"e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9\" (UID: \"3143ac28-e2fe-4164-8639-8b29ae7c7c21\") " pod="llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9"
Sep 18 17:54:40.338521 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:40.338502    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9lp5\" (UniqueName: \"kubernetes.io/projected/3143ac28-e2fe-4164-8639-8b29ae7c7c21-kube-api-access-h9lp5\") pod \"e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9\" (UID: \"3143ac28-e2fe-4164-8639-8b29ae7c7c21\") " pod="llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9"
Sep 18 17:54:40.430238 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:40.430151    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9"
Sep 18 17:54:40.714518 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:40.714488    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9"]
Sep 18 17:54:40.717086 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:54:40.717056    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3143ac28_e2fe_4164_8639_8b29ae7c7c21.slice/crio-3d28873f495ddafc9d838e761da81411e32c89115009a18f7ae4c1b58857195e WatchSource:0}: Error finding container 3d28873f495ddafc9d838e761da81411e32c89115009a18f7ae4c1b58857195e: Status 404 returned error can't find the container with id 3d28873f495ddafc9d838e761da81411e32c89115009a18f7ae4c1b58857195e
Sep 18 17:54:40.936192 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:40.936153    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9" event={"ID":"3143ac28-e2fe-4164-8639-8b29ae7c7c21","Type":"ContainerStarted","Data":"43823a2d16dd6027f8fd9ff012be5c97cc134f1a5e7152019fc521bed1579c7c"}
Sep 18 17:54:40.936192 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:40.936195    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9" event={"ID":"3143ac28-e2fe-4164-8639-8b29ae7c7c21","Type":"ContainerStarted","Data":"3d28873f495ddafc9d838e761da81411e32c89115009a18f7ae4c1b58857195e"}
Sep 18 17:54:40.958182 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:40.958131    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9" podStartSLOduration=0.95811615 podStartE2EDuration="958.11615ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:54:40.957094439 +0000 UTC m=+2393.651021982" watchObservedRunningTime="2026-09-18 17:54:40.95811615 +0000 UTC m=+2393.652043680"
Sep 18 17:54:50.431035 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:50.430993    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9"
Sep 18 17:54:50.431553 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:50.431050    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9"
Sep 18 17:54:50.443573 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:50.443545    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9"
Sep 18 17:54:50.994503 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:50.994474    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9"
Sep 18 17:54:51.339437 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:51.339344    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9"]
Sep 18 17:54:51.342824 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:51.342801    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9"
Sep 18 17:54:51.346076 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:51.346057    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-3844c469-kserve-self-signed-certs\""
Sep 18 17:54:51.358562 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:51.358540    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9"]
Sep 18 17:54:51.431463 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:51.431425    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-home\") pod \"e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9\" (UID: \"3a4b5a89-c1c1-4660-b01f-4a2ee9621f10\") " pod="llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9"
Sep 18 17:54:51.431463 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:51.431458    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-model-cache\") pod \"e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9\" (UID: \"3a4b5a89-c1c1-4660-b01f-4a2ee9621f10\") " pod="llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9"
Sep 18 17:54:51.431945 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:51.431485    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6dcz\" (UniqueName: \"kubernetes.io/projected/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-kube-api-access-j6dcz\") pod \"e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9\" (UID: \"3a4b5a89-c1c1-4660-b01f-4a2ee9621f10\") " pod="llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9"
Sep 18 17:54:51.431945 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:51.431620    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-tls-certs\") pod \"e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9\" (UID: \"3a4b5a89-c1c1-4660-b01f-4a2ee9621f10\") " pod="llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9"
Sep 18 17:54:51.431945 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:51.431671    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-tmp-dir\") pod \"e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9\" (UID: \"3a4b5a89-c1c1-4660-b01f-4a2ee9621f10\") " pod="llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9"
Sep 18 17:54:51.431945 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:51.431691    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-dshm\") pod \"e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9\" (UID: \"3a4b5a89-c1c1-4660-b01f-4a2ee9621f10\") " pod="llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9"
Sep 18 17:54:51.532477 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:51.532445    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-tls-certs\") pod \"e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9\" (UID: \"3a4b5a89-c1c1-4660-b01f-4a2ee9621f10\") " pod="llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9"
Sep 18 17:54:51.532668 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:51.532492    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-tmp-dir\") pod \"e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9\" (UID: \"3a4b5a89-c1c1-4660-b01f-4a2ee9621f10\") " pod="llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9"
Sep 18 17:54:51.532668 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:51.532604    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-dshm\") pod \"e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9\" (UID: \"3a4b5a89-c1c1-4660-b01f-4a2ee9621f10\") " pod="llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9"
Sep 18 17:54:51.532668 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:51.532668    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-home\") pod \"e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9\" (UID: \"3a4b5a89-c1c1-4660-b01f-4a2ee9621f10\") " pod="llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9"
Sep 18 17:54:51.532828 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:51.532688    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-model-cache\") pod \"e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9\" (UID: \"3a4b5a89-c1c1-4660-b01f-4a2ee9621f10\") " pod="llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9"
Sep 18 17:54:51.532828 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:51.532717    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6dcz\" (UniqueName: \"kubernetes.io/projected/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-kube-api-access-j6dcz\") pod \"e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9\" (UID: \"3a4b5a89-c1c1-4660-b01f-4a2ee9621f10\") " pod="llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9"
Sep 18 17:54:51.532978 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:51.532924    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-tmp-dir\") pod \"e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9\" (UID: \"3a4b5a89-c1c1-4660-b01f-4a2ee9621f10\") " pod="llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9"
Sep 18 17:54:51.533096 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:51.533076    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-home\") pod \"e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9\" (UID: \"3a4b5a89-c1c1-4660-b01f-4a2ee9621f10\") " pod="llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9"
Sep 18 17:54:51.533157 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:51.533115    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-model-cache\") pod \"e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9\" (UID: \"3a4b5a89-c1c1-4660-b01f-4a2ee9621f10\") " pod="llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9"
Sep 18 17:54:51.534830 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:51.534806    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-dshm\") pod \"e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9\" (UID: \"3a4b5a89-c1c1-4660-b01f-4a2ee9621f10\") " pod="llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9"
Sep 18 17:54:51.535131 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:51.535116    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-tls-certs\") pod \"e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9\" (UID: \"3a4b5a89-c1c1-4660-b01f-4a2ee9621f10\") " pod="llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9"
Sep 18 17:54:51.542959 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:51.542937    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6dcz\" (UniqueName: \"kubernetes.io/projected/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-kube-api-access-j6dcz\") pod \"e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9\" (UID: \"3a4b5a89-c1c1-4660-b01f-4a2ee9621f10\") " pod="llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9"
Sep 18 17:54:51.651975 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:51.651879    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9"
Sep 18 17:54:51.945189 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:51.945119    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9"]
Sep 18 17:54:51.948262 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:54:51.948221    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a4b5a89_c1c1_4660_b01f_4a2ee9621f10.slice/crio-565586b73ada0c5dbf6d16ffe066326fec7b90d20c37e8ba832440259217521a WatchSource:0}: Error finding container 565586b73ada0c5dbf6d16ffe066326fec7b90d20c37e8ba832440259217521a: Status 404 returned error can't find the container with id 565586b73ada0c5dbf6d16ffe066326fec7b90d20c37e8ba832440259217521a
Sep 18 17:54:51.989197 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:51.989164    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9" event={"ID":"3a4b5a89-c1c1-4660-b01f-4a2ee9621f10","Type":"ContainerStarted","Data":"565586b73ada0c5dbf6d16ffe066326fec7b90d20c37e8ba832440259217521a"}
Sep 18 17:54:52.995748 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:52.995713    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9" event={"ID":"3a4b5a89-c1c1-4660-b01f-4a2ee9621f10","Type":"ContainerStarted","Data":"d36536cf280bebea3ce44200318825a6170b2a05c260af4561b9b0d350e6bdb2"}
Sep 18 17:54:53.024033 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:54:53.023982    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9" podStartSLOduration=2.023966005 podStartE2EDuration="2.023966005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:54:53.020559355 +0000 UTC m=+2405.714486882" watchObservedRunningTime="2026-09-18 17:54:53.023966005 +0000 UTC m=+2405.717893530"
Sep 18 17:55:01.652831 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:01.652791    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9"
Sep 18 17:55:01.653433 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:01.652842    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9"
Sep 18 17:55:01.665730 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:01.665705    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9"
Sep 18 17:55:02.050493 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:02.050462    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9"
Sep 18 17:55:09.040792 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:09.040756    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9"]
Sep 18 17:55:11.381331 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:11.381257    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9"]
Sep 18 17:55:24.059604 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:24.059558    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9" podUID="3143ac28-e2fe-4164-8639-8b29ae7c7c21" containerName="main" containerID="cri-o://43823a2d16dd6027f8fd9ff012be5c97cc134f1a5e7152019fc521bed1579c7c" gracePeriod=45
Sep 18 17:55:24.302781 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:24.302758    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9"
Sep 18 17:55:24.401697 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:24.401623    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9lp5\" (UniqueName: \"kubernetes.io/projected/3143ac28-e2fe-4164-8639-8b29ae7c7c21-kube-api-access-h9lp5\") pod \"3143ac28-e2fe-4164-8639-8b29ae7c7c21\" (UID: \"3143ac28-e2fe-4164-8639-8b29ae7c7c21\") "
Sep 18 17:55:24.401697 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:24.401656    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3143ac28-e2fe-4164-8639-8b29ae7c7c21-dshm\") pod \"3143ac28-e2fe-4164-8639-8b29ae7c7c21\" (UID: \"3143ac28-e2fe-4164-8639-8b29ae7c7c21\") "
Sep 18 17:55:24.401697 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:24.401688    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3143ac28-e2fe-4164-8639-8b29ae7c7c21-model-cache\") pod \"3143ac28-e2fe-4164-8639-8b29ae7c7c21\" (UID: \"3143ac28-e2fe-4164-8639-8b29ae7c7c21\") "
Sep 18 17:55:24.401991 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:24.401741    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3143ac28-e2fe-4164-8639-8b29ae7c7c21-home\") pod \"3143ac28-e2fe-4164-8639-8b29ae7c7c21\" (UID: \"3143ac28-e2fe-4164-8639-8b29ae7c7c21\") "
Sep 18 17:55:24.401991 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:24.401760    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3143ac28-e2fe-4164-8639-8b29ae7c7c21-tmp-dir\") pod \"3143ac28-e2fe-4164-8639-8b29ae7c7c21\" (UID: \"3143ac28-e2fe-4164-8639-8b29ae7c7c21\") "
Sep 18 17:55:24.401991 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:24.401790    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3143ac28-e2fe-4164-8639-8b29ae7c7c21-tls-certs\") pod \"3143ac28-e2fe-4164-8639-8b29ae7c7c21\" (UID: \"3143ac28-e2fe-4164-8639-8b29ae7c7c21\") "
Sep 18 17:55:24.402154 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:24.402000    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3143ac28-e2fe-4164-8639-8b29ae7c7c21-model-cache" (OuterVolumeSpecName: "model-cache") pod "3143ac28-e2fe-4164-8639-8b29ae7c7c21" (UID: "3143ac28-e2fe-4164-8639-8b29ae7c7c21"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:55:24.402154 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:24.402043    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3143ac28-e2fe-4164-8639-8b29ae7c7c21-home" (OuterVolumeSpecName: "home") pod "3143ac28-e2fe-4164-8639-8b29ae7c7c21" (UID: "3143ac28-e2fe-4164-8639-8b29ae7c7c21"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:55:24.402154 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:24.402076    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3143ac28-e2fe-4164-8639-8b29ae7c7c21-model-cache\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:55:24.402154 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:24.402104    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3143ac28-e2fe-4164-8639-8b29ae7c7c21-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3143ac28-e2fe-4164-8639-8b29ae7c7c21" (UID: "3143ac28-e2fe-4164-8639-8b29ae7c7c21"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:55:24.403940 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:24.403905    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3143ac28-e2fe-4164-8639-8b29ae7c7c21-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3143ac28-e2fe-4164-8639-8b29ae7c7c21" (UID: "3143ac28-e2fe-4164-8639-8b29ae7c7c21"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:55:24.404066 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:24.403979    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3143ac28-e2fe-4164-8639-8b29ae7c7c21-dshm" (OuterVolumeSpecName: "dshm") pod "3143ac28-e2fe-4164-8639-8b29ae7c7c21" (UID: "3143ac28-e2fe-4164-8639-8b29ae7c7c21"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:55:24.404066 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:24.403994    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3143ac28-e2fe-4164-8639-8b29ae7c7c21-kube-api-access-h9lp5" (OuterVolumeSpecName: "kube-api-access-h9lp5") pod "3143ac28-e2fe-4164-8639-8b29ae7c7c21" (UID: "3143ac28-e2fe-4164-8639-8b29ae7c7c21"). InnerVolumeSpecName "kube-api-access-h9lp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:55:24.502645 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:24.502615    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3143ac28-e2fe-4164-8639-8b29ae7c7c21-tls-certs\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:55:24.502645 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:24.502639    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9lp5\" (UniqueName: \"kubernetes.io/projected/3143ac28-e2fe-4164-8639-8b29ae7c7c21-kube-api-access-h9lp5\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:55:24.502645 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:24.502652    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3143ac28-e2fe-4164-8639-8b29ae7c7c21-dshm\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:55:24.502907 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:24.502663    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3143ac28-e2fe-4164-8639-8b29ae7c7c21-home\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:55:24.502907 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:24.502670    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3143ac28-e2fe-4164-8639-8b29ae7c7c21-tmp-dir\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:55:25.150173 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:25.150138    2569 generic.go:356] "Generic (PLEG): container finished" podID="3143ac28-e2fe-4164-8639-8b29ae7c7c21" containerID="43823a2d16dd6027f8fd9ff012be5c97cc134f1a5e7152019fc521bed1579c7c" exitCode=0
Sep 18 17:55:25.150698 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:25.150187    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9"
Sep 18 17:55:25.150698 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:25.150225    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9" event={"ID":"3143ac28-e2fe-4164-8639-8b29ae7c7c21","Type":"ContainerDied","Data":"43823a2d16dd6027f8fd9ff012be5c97cc134f1a5e7152019fc521bed1579c7c"}
Sep 18 17:55:25.150698 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:25.150267    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9" event={"ID":"3143ac28-e2fe-4164-8639-8b29ae7c7c21","Type":"ContainerDied","Data":"3d28873f495ddafc9d838e761da81411e32c89115009a18f7ae4c1b58857195e"}
Sep 18 17:55:25.150698 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:25.150283    2569 scope.go:117] "RemoveContainer" containerID="43823a2d16dd6027f8fd9ff012be5c97cc134f1a5e7152019fc521bed1579c7c"
Sep 18 17:55:25.159173 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:25.159155    2569 scope.go:117] "RemoveContainer" containerID="43823a2d16dd6027f8fd9ff012be5c97cc134f1a5e7152019fc521bed1579c7c"
Sep 18 17:55:25.159413 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:55:25.159393    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43823a2d16dd6027f8fd9ff012be5c97cc134f1a5e7152019fc521bed1579c7c\": container with ID starting with 43823a2d16dd6027f8fd9ff012be5c97cc134f1a5e7152019fc521bed1579c7c not found: ID does not exist" containerID="43823a2d16dd6027f8fd9ff012be5c97cc134f1a5e7152019fc521bed1579c7c"
Sep 18 17:55:25.159469 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:25.159425    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43823a2d16dd6027f8fd9ff012be5c97cc134f1a5e7152019fc521bed1579c7c"} err="failed to get container status \"43823a2d16dd6027f8fd9ff012be5c97cc134f1a5e7152019fc521bed1579c7c\": rpc error: code = NotFound desc = could not find container \"43823a2d16dd6027f8fd9ff012be5c97cc134f1a5e7152019fc521bed1579c7c\": container with ID starting with 43823a2d16dd6027f8fd9ff012be5c97cc134f1a5e7152019fc521bed1579c7c not found: ID does not exist"
Sep 18 17:55:25.173032 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:25.173010    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9"]
Sep 18 17:55:25.175315 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:25.175294    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-3844c469-kserve-5d4bd4b855-kmzq9"]
Sep 18 17:55:25.767769 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:25.767731    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3143ac28-e2fe-4164-8639-8b29ae7c7c21" path="/var/lib/kubelet/pods/3143ac28-e2fe-4164-8639-8b29ae7c7c21/volumes"
Sep 18 17:55:26.399628 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:26.399590    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9" podUID="3a4b5a89-c1c1-4660-b01f-4a2ee9621f10" containerName="main" containerID="cri-o://d36536cf280bebea3ce44200318825a6170b2a05c260af4561b9b0d350e6bdb2" gracePeriod=45
Sep 18 17:55:26.636854 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:26.636824    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9"
Sep 18 17:55:26.722118 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:26.722090    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6dcz\" (UniqueName: \"kubernetes.io/projected/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-kube-api-access-j6dcz\") pod \"3a4b5a89-c1c1-4660-b01f-4a2ee9621f10\" (UID: \"3a4b5a89-c1c1-4660-b01f-4a2ee9621f10\") "
Sep 18 17:55:26.722305 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:26.722130    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-model-cache\") pod \"3a4b5a89-c1c1-4660-b01f-4a2ee9621f10\" (UID: \"3a4b5a89-c1c1-4660-b01f-4a2ee9621f10\") "
Sep 18 17:55:26.722305 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:26.722184    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-tmp-dir\") pod \"3a4b5a89-c1c1-4660-b01f-4a2ee9621f10\" (UID: \"3a4b5a89-c1c1-4660-b01f-4a2ee9621f10\") "
Sep 18 17:55:26.722305 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:26.722214    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-dshm\") pod \"3a4b5a89-c1c1-4660-b01f-4a2ee9621f10\" (UID: \"3a4b5a89-c1c1-4660-b01f-4a2ee9621f10\") "
Sep 18 17:55:26.722305 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:26.722245    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-tls-certs\") pod \"3a4b5a89-c1c1-4660-b01f-4a2ee9621f10\" (UID: \"3a4b5a89-c1c1-4660-b01f-4a2ee9621f10\") "
Sep 18 17:55:26.722305 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:26.722283    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-home\") pod \"3a4b5a89-c1c1-4660-b01f-4a2ee9621f10\" (UID: \"3a4b5a89-c1c1-4660-b01f-4a2ee9621f10\") "
Sep 18 17:55:26.722571 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:26.722418    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-model-cache" (OuterVolumeSpecName: "model-cache") pod "3a4b5a89-c1c1-4660-b01f-4a2ee9621f10" (UID: "3a4b5a89-c1c1-4660-b01f-4a2ee9621f10"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:55:26.722571 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:26.722502    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3a4b5a89-c1c1-4660-b01f-4a2ee9621f10" (UID: "3a4b5a89-c1c1-4660-b01f-4a2ee9621f10"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:55:26.722678 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:26.722589    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-home" (OuterVolumeSpecName: "home") pod "3a4b5a89-c1c1-4660-b01f-4a2ee9621f10" (UID: "3a4b5a89-c1c1-4660-b01f-4a2ee9621f10"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:55:26.722678 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:26.722647    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-home\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:55:26.722678 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:26.722661    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-model-cache\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:55:26.722678 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:26.722676    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-tmp-dir\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:55:26.724415 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:26.724384    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-kube-api-access-j6dcz" (OuterVolumeSpecName: "kube-api-access-j6dcz") pod "3a4b5a89-c1c1-4660-b01f-4a2ee9621f10" (UID: "3a4b5a89-c1c1-4660-b01f-4a2ee9621f10"). InnerVolumeSpecName "kube-api-access-j6dcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:55:26.724415 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:26.724386    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3a4b5a89-c1c1-4660-b01f-4a2ee9621f10" (UID: "3a4b5a89-c1c1-4660-b01f-4a2ee9621f10"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:55:26.724576 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:26.724436    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-dshm" (OuterVolumeSpecName: "dshm") pod "3a4b5a89-c1c1-4660-b01f-4a2ee9621f10" (UID: "3a4b5a89-c1c1-4660-b01f-4a2ee9621f10"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:55:26.823788 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:26.823706    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-dshm\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:55:26.823788 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:26.823737    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-tls-certs\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:55:26.823788 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:26.823748    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6dcz\" (UniqueName: \"kubernetes.io/projected/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10-kube-api-access-j6dcz\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:55:27.164594 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.164501    2569 generic.go:356] "Generic (PLEG): container finished" podID="3a4b5a89-c1c1-4660-b01f-4a2ee9621f10" containerID="d36536cf280bebea3ce44200318825a6170b2a05c260af4561b9b0d350e6bdb2" exitCode=0
Sep 18 17:55:27.164594 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.164546    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9"
Sep 18 17:55:27.164825 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.164592    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9" event={"ID":"3a4b5a89-c1c1-4660-b01f-4a2ee9621f10","Type":"ContainerDied","Data":"d36536cf280bebea3ce44200318825a6170b2a05c260af4561b9b0d350e6bdb2"}
Sep 18 17:55:27.164825 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.164642    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9" event={"ID":"3a4b5a89-c1c1-4660-b01f-4a2ee9621f10","Type":"ContainerDied","Data":"565586b73ada0c5dbf6d16ffe066326fec7b90d20c37e8ba832440259217521a"}
Sep 18 17:55:27.164825 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.164660    2569 scope.go:117] "RemoveContainer" containerID="d36536cf280bebea3ce44200318825a6170b2a05c260af4561b9b0d350e6bdb2"
Sep 18 17:55:27.174100 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.174086    2569 scope.go:117] "RemoveContainer" containerID="d36536cf280bebea3ce44200318825a6170b2a05c260af4561b9b0d350e6bdb2"
Sep 18 17:55:27.174409 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:55:27.174386    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d36536cf280bebea3ce44200318825a6170b2a05c260af4561b9b0d350e6bdb2\": container with ID starting with d36536cf280bebea3ce44200318825a6170b2a05c260af4561b9b0d350e6bdb2 not found: ID does not exist" containerID="d36536cf280bebea3ce44200318825a6170b2a05c260af4561b9b0d350e6bdb2"
Sep 18 17:55:27.174468 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.174418    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d36536cf280bebea3ce44200318825a6170b2a05c260af4561b9b0d350e6bdb2"} err="failed to get container status \"d36536cf280bebea3ce44200318825a6170b2a05c260af4561b9b0d350e6bdb2\": rpc error: code = NotFound desc = could not find container \"d36536cf280bebea3ce44200318825a6170b2a05c260af4561b9b0d350e6bdb2\": container with ID starting with d36536cf280bebea3ce44200318825a6170b2a05c260af4561b9b0d350e6bdb2 not found: ID does not exist"
Sep 18 17:55:27.205691 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.205667    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9"]
Sep 18 17:55:27.211581 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.211561    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-3844c469-kserve-77fd54f9cf-jssv9"]
Sep 18 17:55:27.540951 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.540910    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7dc567d6f7-w785p"]
Sep 18 17:55:27.541435 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.541339    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3143ac28-e2fe-4164-8639-8b29ae7c7c21" containerName="main"
Sep 18 17:55:27.541435 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.541357    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3143ac28-e2fe-4164-8639-8b29ae7c7c21" containerName="main"
Sep 18 17:55:27.541435 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.541377    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a4b5a89-c1c1-4660-b01f-4a2ee9621f10" containerName="main"
Sep 18 17:55:27.541435 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.541384    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a4b5a89-c1c1-4660-b01f-4a2ee9621f10" containerName="main"
Sep 18 17:55:27.541651 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.541486    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3143ac28-e2fe-4164-8639-8b29ae7c7c21" containerName="main"
Sep 18 17:55:27.541651 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.541498    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a4b5a89-c1c1-4660-b01f-4a2ee9621f10" containerName="main"
Sep 18 17:55:27.545705 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.545683    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7dc567d6f7-w785p"
Sep 18 17:55:27.549077 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.549045    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 17:55:27.549193 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.549080    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 17:55:27.549193 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.549114    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 18 17:55:27.549193 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.549136    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sjtq4\""
Sep 18 17:55:27.549193 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.549139    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 18 17:55:27.563079 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.563055    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7dc567d6f7-w785p"]
Sep 18 17:55:27.631736 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.631702    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lxq7\" (UniqueName: \"kubernetes.io/projected/1378c22b-14ff-4b28-af33-a49b8d4b37e9-kube-api-access-5lxq7\") pod \"maas-api-7dc567d6f7-w785p\" (UID: \"1378c22b-14ff-4b28-af33-a49b8d4b37e9\") " pod="odh-ai-gateway-infra/maas-api-7dc567d6f7-w785p"
Sep 18 17:55:27.631924 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.631749    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1378c22b-14ff-4b28-af33-a49b8d4b37e9-maas-api-tls\") pod \"maas-api-7dc567d6f7-w785p\" (UID: \"1378c22b-14ff-4b28-af33-a49b8d4b37e9\") " pod="odh-ai-gateway-infra/maas-api-7dc567d6f7-w785p"
Sep 18 17:55:27.631924 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.631814    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1378c22b-14ff-4b28-af33-a49b8d4b37e9-openshift-service-ca\") pod \"maas-api-7dc567d6f7-w785p\" (UID: \"1378c22b-14ff-4b28-af33-a49b8d4b37e9\") " pod="odh-ai-gateway-infra/maas-api-7dc567d6f7-w785p"
Sep 18 17:55:27.631924 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.631851    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1378c22b-14ff-4b28-af33-a49b8d4b37e9-metrics-certs\") pod \"maas-api-7dc567d6f7-w785p\" (UID: \"1378c22b-14ff-4b28-af33-a49b8d4b37e9\") " pod="odh-ai-gateway-infra/maas-api-7dc567d6f7-w785p"
Sep 18 17:55:27.732571 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.732527    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1378c22b-14ff-4b28-af33-a49b8d4b37e9-maas-api-tls\") pod \"maas-api-7dc567d6f7-w785p\" (UID: \"1378c22b-14ff-4b28-af33-a49b8d4b37e9\") " pod="odh-ai-gateway-infra/maas-api-7dc567d6f7-w785p"
Sep 18 17:55:27.732571 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.732581    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1378c22b-14ff-4b28-af33-a49b8d4b37e9-openshift-service-ca\") pod \"maas-api-7dc567d6f7-w785p\" (UID: \"1378c22b-14ff-4b28-af33-a49b8d4b37e9\") " pod="odh-ai-gateway-infra/maas-api-7dc567d6f7-w785p"
Sep 18 17:55:27.732847 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.732605    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1378c22b-14ff-4b28-af33-a49b8d4b37e9-metrics-certs\") pod \"maas-api-7dc567d6f7-w785p\" (UID: \"1378c22b-14ff-4b28-af33-a49b8d4b37e9\") " pod="odh-ai-gateway-infra/maas-api-7dc567d6f7-w785p"
Sep 18 17:55:27.732847 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.732687    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lxq7\" (UniqueName: \"kubernetes.io/projected/1378c22b-14ff-4b28-af33-a49b8d4b37e9-kube-api-access-5lxq7\") pod \"maas-api-7dc567d6f7-w785p\" (UID: \"1378c22b-14ff-4b28-af33-a49b8d4b37e9\") " pod="odh-ai-gateway-infra/maas-api-7dc567d6f7-w785p"
Sep 18 17:55:27.733355 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.733332    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1378c22b-14ff-4b28-af33-a49b8d4b37e9-openshift-service-ca\") pod \"maas-api-7dc567d6f7-w785p\" (UID: \"1378c22b-14ff-4b28-af33-a49b8d4b37e9\") " pod="odh-ai-gateway-infra/maas-api-7dc567d6f7-w785p"
Sep 18 17:55:27.735080 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.735058    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1378c22b-14ff-4b28-af33-a49b8d4b37e9-maas-api-tls\") pod \"maas-api-7dc567d6f7-w785p\" (UID: \"1378c22b-14ff-4b28-af33-a49b8d4b37e9\") " pod="odh-ai-gateway-infra/maas-api-7dc567d6f7-w785p"
Sep 18 17:55:27.735224 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.735203    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1378c22b-14ff-4b28-af33-a49b8d4b37e9-metrics-certs\") pod \"maas-api-7dc567d6f7-w785p\" (UID: \"1378c22b-14ff-4b28-af33-a49b8d4b37e9\") " pod="odh-ai-gateway-infra/maas-api-7dc567d6f7-w785p"
Sep 18 17:55:27.747899 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.747852    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lxq7\" (UniqueName: \"kubernetes.io/projected/1378c22b-14ff-4b28-af33-a49b8d4b37e9-kube-api-access-5lxq7\") pod \"maas-api-7dc567d6f7-w785p\" (UID: \"1378c22b-14ff-4b28-af33-a49b8d4b37e9\") " pod="odh-ai-gateway-infra/maas-api-7dc567d6f7-w785p"
Sep 18 17:55:27.764555 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.764531    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a4b5a89-c1c1-4660-b01f-4a2ee9621f10" path="/var/lib/kubelet/pods/3a4b5a89-c1c1-4660-b01f-4a2ee9621f10/volumes"
Sep 18 17:55:27.856051 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:27.855966    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7dc567d6f7-w785p"
Sep 18 17:55:28.128503 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:28.128319    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7dc567d6f7-w785p"]
Sep 18 17:55:28.131018 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:55:28.130989    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1378c22b_14ff_4b28_af33_a49b8d4b37e9.slice/crio-ef6cbbac591bf81403e31d9d4e797f3a77485a64793d1adc6a217d1e0838191f WatchSource:0}: Error finding container ef6cbbac591bf81403e31d9d4e797f3a77485a64793d1adc6a217d1e0838191f: Status 404 returned error can't find the container with id ef6cbbac591bf81403e31d9d4e797f3a77485a64793d1adc6a217d1e0838191f
Sep 18 17:55:28.132395 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:28.132379    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 17:55:28.170085 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:28.170053    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7dc567d6f7-w785p" event={"ID":"1378c22b-14ff-4b28-af33-a49b8d4b37e9","Type":"ContainerStarted","Data":"ef6cbbac591bf81403e31d9d4e797f3a77485a64793d1adc6a217d1e0838191f"}
Sep 18 17:55:29.177001 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:29.176963    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7dc567d6f7-w785p" event={"ID":"1378c22b-14ff-4b28-af33-a49b8d4b37e9","Type":"ContainerStarted","Data":"4e444c242534f6f8596c413223811a5805bec923a2e9a20b807dcb8990d13e59"}
Sep 18 17:55:29.177451 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:29.177060    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7dc567d6f7-w785p"
Sep 18 17:55:29.197446 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:29.197396    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7dc567d6f7-w785p" podStartSLOduration=1.691670032 podStartE2EDuration="2.197382602s" totalImagesPullingTime="505.71257ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:55:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:55:29.195445886 +0000 UTC m=+2441.889373435" watchObservedRunningTime="2026-09-18 17:55:29.197382602 +0000 UTC m=+2441.891310140"
Sep 18 17:55:35.187130 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:35.187100    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7dc567d6f7-w785p"
Sep 18 17:55:35.500340 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:35.500304    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7dc567d6f7-w785p"]
Sep 18 17:55:35.500551 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:35.500520    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7dc567d6f7-w785p" podUID="1378c22b-14ff-4b28-af33-a49b8d4b37e9" containerName="maas-api" containerID="cri-o://4e444c242534f6f8596c413223811a5805bec923a2e9a20b807dcb8990d13e59" gracePeriod=30
Sep 18 17:55:35.739405 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:35.739380    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7dc567d6f7-w785p"
Sep 18 17:55:35.806712 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:35.806634    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1378c22b-14ff-4b28-af33-a49b8d4b37e9-maas-api-tls\") pod \"1378c22b-14ff-4b28-af33-a49b8d4b37e9\" (UID: \"1378c22b-14ff-4b28-af33-a49b8d4b37e9\") "
Sep 18 17:55:35.806908 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:35.806745    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lxq7\" (UniqueName: \"kubernetes.io/projected/1378c22b-14ff-4b28-af33-a49b8d4b37e9-kube-api-access-5lxq7\") pod \"1378c22b-14ff-4b28-af33-a49b8d4b37e9\" (UID: \"1378c22b-14ff-4b28-af33-a49b8d4b37e9\") "
Sep 18 17:55:35.806908 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:35.806766    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1378c22b-14ff-4b28-af33-a49b8d4b37e9-metrics-certs\") pod \"1378c22b-14ff-4b28-af33-a49b8d4b37e9\" (UID: \"1378c22b-14ff-4b28-af33-a49b8d4b37e9\") "
Sep 18 17:55:35.806908 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:35.806786    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1378c22b-14ff-4b28-af33-a49b8d4b37e9-openshift-service-ca\") pod \"1378c22b-14ff-4b28-af33-a49b8d4b37e9\" (UID: \"1378c22b-14ff-4b28-af33-a49b8d4b37e9\") "
Sep 18 17:55:35.807220 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:35.807192    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1378c22b-14ff-4b28-af33-a49b8d4b37e9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1378c22b-14ff-4b28-af33-a49b8d4b37e9" (UID: "1378c22b-14ff-4b28-af33-a49b8d4b37e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:55:35.808830 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:35.808804    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1378c22b-14ff-4b28-af33-a49b8d4b37e9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1378c22b-14ff-4b28-af33-a49b8d4b37e9" (UID: "1378c22b-14ff-4b28-af33-a49b8d4b37e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:55:35.808941 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:35.808849    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1378c22b-14ff-4b28-af33-a49b8d4b37e9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1378c22b-14ff-4b28-af33-a49b8d4b37e9" (UID: "1378c22b-14ff-4b28-af33-a49b8d4b37e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:55:35.808941 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:35.808848    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1378c22b-14ff-4b28-af33-a49b8d4b37e9-kube-api-access-5lxq7" (OuterVolumeSpecName: "kube-api-access-5lxq7") pod "1378c22b-14ff-4b28-af33-a49b8d4b37e9" (UID: "1378c22b-14ff-4b28-af33-a49b8d4b37e9"). InnerVolumeSpecName "kube-api-access-5lxq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:55:35.908245 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:35.908213    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1378c22b-14ff-4b28-af33-a49b8d4b37e9-maas-api-tls\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:55:35.908245 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:35.908240    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lxq7\" (UniqueName: \"kubernetes.io/projected/1378c22b-14ff-4b28-af33-a49b8d4b37e9-kube-api-access-5lxq7\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:55:35.908245 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:35.908250    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1378c22b-14ff-4b28-af33-a49b8d4b37e9-metrics-certs\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:55:35.908245 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:35.908261    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1378c22b-14ff-4b28-af33-a49b8d4b37e9-openshift-service-ca\") on node \"ip-10-0-139-122.ec2.internal\" DevicePath \"\""
Sep 18 17:55:36.211115 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:36.211029    2569 generic.go:356] "Generic (PLEG): container finished" podID="1378c22b-14ff-4b28-af33-a49b8d4b37e9" containerID="4e444c242534f6f8596c413223811a5805bec923a2e9a20b807dcb8990d13e59" exitCode=0
Sep 18 17:55:36.211115 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:36.211090    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7dc567d6f7-w785p"
Sep 18 17:55:36.211574 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:36.211118    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7dc567d6f7-w785p" event={"ID":"1378c22b-14ff-4b28-af33-a49b8d4b37e9","Type":"ContainerDied","Data":"4e444c242534f6f8596c413223811a5805bec923a2e9a20b807dcb8990d13e59"}
Sep 18 17:55:36.211574 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:36.211196    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7dc567d6f7-w785p" event={"ID":"1378c22b-14ff-4b28-af33-a49b8d4b37e9","Type":"ContainerDied","Data":"ef6cbbac591bf81403e31d9d4e797f3a77485a64793d1adc6a217d1e0838191f"}
Sep 18 17:55:36.211574 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:36.211218    2569 scope.go:117] "RemoveContainer" containerID="4e444c242534f6f8596c413223811a5805bec923a2e9a20b807dcb8990d13e59"
Sep 18 17:55:36.220835 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:36.220817    2569 scope.go:117] "RemoveContainer" containerID="4e444c242534f6f8596c413223811a5805bec923a2e9a20b807dcb8990d13e59"
Sep 18 17:55:36.221115 ip-10-0-139-122 kubenswrapper[2569]: E0918 17:55:36.221097    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e444c242534f6f8596c413223811a5805bec923a2e9a20b807dcb8990d13e59\": container with ID starting with 4e444c242534f6f8596c413223811a5805bec923a2e9a20b807dcb8990d13e59 not found: ID does not exist" containerID="4e444c242534f6f8596c413223811a5805bec923a2e9a20b807dcb8990d13e59"
Sep 18 17:55:36.221165 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:36.221123    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e444c242534f6f8596c413223811a5805bec923a2e9a20b807dcb8990d13e59"} err="failed to get container status \"4e444c242534f6f8596c413223811a5805bec923a2e9a20b807dcb8990d13e59\": rpc error: code = NotFound desc = could not find container \"4e444c242534f6f8596c413223811a5805bec923a2e9a20b807dcb8990d13e59\": container with ID starting with 4e444c242534f6f8596c413223811a5805bec923a2e9a20b807dcb8990d13e59 not found: ID does not exist"
Sep 18 17:55:36.239606 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:36.239583    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7dc567d6f7-w785p"]
Sep 18 17:55:36.243605 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:36.243583    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7dc567d6f7-w785p"]
Sep 18 17:55:37.764549 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:37.764514    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1378c22b-14ff-4b28-af33-a49b8d4b37e9" path="/var/lib/kubelet/pods/1378c22b-14ff-4b28-af33-a49b8d4b37e9/volumes"
Sep 18 17:55:48.401236 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:48.401200    2569 scope.go:117] "RemoveContainer" containerID="cfff776f6e33af07e84147c8ee980a0c27bb26853aeb40301ce545469fed58a3"
Sep 18 17:55:48.409287 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:48.409269    2569 scope.go:117] "RemoveContainer" containerID="0561d5ef278917f75ee7af91e3b3e261c0add196231afb52c02915a23b1251b1"
Sep 18 17:55:48.416696 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:55:48.416670    2569 scope.go:117] "RemoveContainer" containerID="15ef1e934d20654ea4100085178582c1f072d7e169f6c8a7cc0975d215901b2a"
Sep 18 17:56:01.549650 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:01.549619    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-dc59d87f8-q2x86_f431da55-9418-45da-be88-cd303b6af64a/authorino/0.log"
Sep 18 17:56:07.691337 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:07.691304    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9496889b9-8mhvz_f0748df9-cc4f-4426-9035-21be3c6519e7/copy-manifests/0.log"
Sep 18 17:56:07.697643 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:07.697611    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9496889b9-8mhvz_f0748df9-cc4f-4426-9035-21be3c6519e7/manager/0.log"
Sep 18 17:56:08.355341 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:08.355313    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29829210-mn2xl_3e1310bd-d032-45ee-bbc6-7adee2636dad/cleanup/0.log"
Sep 18 17:56:08.465948 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:08.465916    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29829225-gffbg_763b5e7b-e33e-4163-9e71-2d25bfba9634/cleanup/0.log"
Sep 18 17:56:09.322647 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:09.322618    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-dc59d87f8-q2x86_f431da55-9418-45da-be88-cd303b6af64a/authorino/0.log"
Sep 18 17:56:09.697174 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:09.697098    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-7vgst_a2253623-1c4c-4134-92b7-ac254c2c0569/kuadrant-console-plugin/0.log"
Sep 18 17:56:10.606553 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:10.606521    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65b7cd5988-ng2jw_812e1c0f-7f7a-495a-96c7-7e03d0b2599c/kube-auth-proxy/0.log"
Sep 18 17:56:11.189914 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:11.189887    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76465d565b-r2rdc_63db05d0-62bd-4fe6-999f-0d19ef2ff266/router/0.log"
Sep 18 17:56:11.674068 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:11.674036    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-x2z66_8b847607-8f6e-4cd3-94b8-283fbd05fa46/main/0.log"
Sep 18 17:56:12.060695 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:12.060666    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp5qvz_5d104c24-7c53-4359-b2a4-20f7e2805a9a/main/0.log"
Sep 18 17:56:12.338139 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:12.338052    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-757lh_55cebd16-d2c2-4a44-ad3a-9a0c3f0daf23/main/0.log"
Sep 18 17:56:18.096159 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:18.096123    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xbl54/must-gather-9cz46"]
Sep 18 17:56:18.096545 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:18.096423    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1378c22b-14ff-4b28-af33-a49b8d4b37e9" containerName="maas-api"
Sep 18 17:56:18.096545 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:18.096434    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1378c22b-14ff-4b28-af33-a49b8d4b37e9" containerName="maas-api"
Sep 18 17:56:18.096545 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:18.096496    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1378c22b-14ff-4b28-af33-a49b8d4b37e9" containerName="maas-api"
Sep 18 17:56:18.099493 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:18.099475    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xbl54/must-gather-9cz46"
Sep 18 17:56:18.101969 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:18.101943    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xbl54\"/\"openshift-service-ca.crt\""
Sep 18 17:56:18.102112 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:18.102044    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xbl54\"/\"kube-root-ca.crt\""
Sep 18 17:56:18.102714 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:18.102700    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xbl54\"/\"default-dockercfg-222tl\""
Sep 18 17:56:18.117549 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:18.111708    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xbl54/must-gather-9cz46"]
Sep 18 17:56:18.160280 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:18.160245    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/91904d71-b9a5-4406-a01b-621b6808ac05-must-gather-output\") pod \"must-gather-9cz46\" (UID: \"91904d71-b9a5-4406-a01b-621b6808ac05\") " pod="openshift-must-gather-xbl54/must-gather-9cz46"
Sep 18 17:56:18.160450 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:18.160301    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5mkm\" (UniqueName: \"kubernetes.io/projected/91904d71-b9a5-4406-a01b-621b6808ac05-kube-api-access-j5mkm\") pod \"must-gather-9cz46\" (UID: \"91904d71-b9a5-4406-a01b-621b6808ac05\") " pod="openshift-must-gather-xbl54/must-gather-9cz46"
Sep 18 17:56:18.261319 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:18.261278    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/91904d71-b9a5-4406-a01b-621b6808ac05-must-gather-output\") pod \"must-gather-9cz46\" (UID: \"91904d71-b9a5-4406-a01b-621b6808ac05\") " pod="openshift-must-gather-xbl54/must-gather-9cz46"
Sep 18 17:56:18.261319 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:18.261325    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5mkm\" (UniqueName: \"kubernetes.io/projected/91904d71-b9a5-4406-a01b-621b6808ac05-kube-api-access-j5mkm\") pod \"must-gather-9cz46\" (UID: \"91904d71-b9a5-4406-a01b-621b6808ac05\") " pod="openshift-must-gather-xbl54/must-gather-9cz46"
Sep 18 17:56:18.261707 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:18.261685    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/91904d71-b9a5-4406-a01b-621b6808ac05-must-gather-output\") pod \"must-gather-9cz46\" (UID: \"91904d71-b9a5-4406-a01b-621b6808ac05\") " pod="openshift-must-gather-xbl54/must-gather-9cz46"
Sep 18 17:56:18.271692 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:18.271659    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5mkm\" (UniqueName: \"kubernetes.io/projected/91904d71-b9a5-4406-a01b-621b6808ac05-kube-api-access-j5mkm\") pod \"must-gather-9cz46\" (UID: \"91904d71-b9a5-4406-a01b-621b6808ac05\") " pod="openshift-must-gather-xbl54/must-gather-9cz46"
Sep 18 17:56:18.408367 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:18.408292    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xbl54/must-gather-9cz46"
Sep 18 17:56:18.683618 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:18.683540    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xbl54/must-gather-9cz46"]
Sep 18 17:56:18.687294 ip-10-0-139-122 kubenswrapper[2569]: W0918 17:56:18.687262    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91904d71_b9a5_4406_a01b_621b6808ac05.slice/crio-113ed0c41bb168cec2fbd5dcec9e18f94cdc7a61e1ba96d2411130bc93f0330b WatchSource:0}: Error finding container 113ed0c41bb168cec2fbd5dcec9e18f94cdc7a61e1ba96d2411130bc93f0330b: Status 404 returned error can't find the container with id 113ed0c41bb168cec2fbd5dcec9e18f94cdc7a61e1ba96d2411130bc93f0330b
Sep 18 17:56:19.408686 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:19.408641    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xbl54/must-gather-9cz46" event={"ID":"91904d71-b9a5-4406-a01b-621b6808ac05","Type":"ContainerStarted","Data":"113ed0c41bb168cec2fbd5dcec9e18f94cdc7a61e1ba96d2411130bc93f0330b"}
Sep 18 17:56:20.415586 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:20.415548    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xbl54/must-gather-9cz46" event={"ID":"91904d71-b9a5-4406-a01b-621b6808ac05","Type":"ContainerStarted","Data":"b713ecc11a35b35e532b04c42193c255833d15d218ccbdea2914edac99c2eb85"}
Sep 18 17:56:20.415586 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:20.415595    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xbl54/must-gather-9cz46" event={"ID":"91904d71-b9a5-4406-a01b-621b6808ac05","Type":"ContainerStarted","Data":"fa34bd9928cda1fe3f2002771f5bdb6d71c03db185bcfd9ebfcd002847476789"}
Sep 18 17:56:20.449560 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:20.449497    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xbl54/must-gather-9cz46" podStartSLOduration=1.656184597 podStartE2EDuration="2.449474777s" totalImagesPullingTime="793.29018ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:56:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:56:20.445681847 +0000 UTC m=+2493.139609374" watchObservedRunningTime="2026-09-18 17:56:20.449474777 +0000 UTC m=+2493.143402307"
Sep 18 17:56:21.535121 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:21.535093    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7kwfm_491a6ab9-f579-4c60-a0ea-6b6aaad27d52/global-pull-secret-syncer/0.log"
Sep 18 17:56:21.841327 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:21.841254    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-blmjk_7e343c56-b6c8-42a6-bf92-dd46fe5d247b/konnectivity-agent/0.log"
Sep 18 17:56:22.071414 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:22.071386    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-122.ec2.internal_96da070bb6fafb62858754f170b042c5/haproxy/0.log"
Sep 18 17:56:26.225889 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:26.225567    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-dc59d87f8-q2x86_f431da55-9418-45da-be88-cd303b6af64a/authorino/0.log"
Sep 18 17:56:26.325253 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:26.325218    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-7vgst_a2253623-1c4c-4134-92b7-ac254c2c0569/kuadrant-console-plugin/0.log"
Sep 18 17:56:28.406377 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:28.406330    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9tw2m_0d81ebf0-5191-4fe8-820b-f16d136e9044/node-exporter/0.log"
Sep 18 17:56:28.433880 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:28.433825    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9tw2m_0d81ebf0-5191-4fe8-820b-f16d136e9044/kube-rbac-proxy/0.log"
Sep 18 17:56:28.455293 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:28.455213    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9tw2m_0d81ebf0-5191-4fe8-820b-f16d136e9044/init-textfile/0.log"
Sep 18 17:56:29.617927 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:29.617898    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5995d9b5fc-hj95d_0c0dd879-ad03-42f8-8d1c-44f9dae55b42/telemeter-client/0.log"
Sep 18 17:56:29.643687 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:29.643662    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5995d9b5fc-hj95d_0c0dd879-ad03-42f8-8d1c-44f9dae55b42/reload/0.log"
Sep 18 17:56:29.679770 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:29.679740    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5995d9b5fc-hj95d_0c0dd879-ad03-42f8-8d1c-44f9dae55b42/kube-rbac-proxy/0.log"
Sep 18 17:56:31.218154 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:31.218119    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-n6wb8_b214c735-b2fd-42a5-9f4e-ee9258c4bfae/networking-console-plugin/0.log"
Sep 18 17:56:31.523288 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:31.523252    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xbl54/perf-node-gather-daemonset-kdz5p"]
Sep 18 17:56:31.530987 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:31.530954    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-kdz5p"
Sep 18 17:56:31.543135 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:31.542158    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xbl54/perf-node-gather-daemonset-kdz5p"]
Sep 18 17:56:31.602945 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:31.602908    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d3f574ea-5fca-4629-8b0b-18755a5c0caa-podres\") pod \"perf-node-gather-daemonset-kdz5p\" (UID: \"d3f574ea-5fca-4629-8b0b-18755a5c0caa\") " pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-kdz5p"
Sep 18 17:56:31.603265 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:31.603229    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3f574ea-5fca-4629-8b0b-18755a5c0caa-lib-modules\") pod \"perf-node-gather-daemonset-kdz5p\" (UID: \"d3f574ea-5fca-4629-8b0b-18755a5c0caa\") " pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-kdz5p"
Sep 18 17:56:31.603426 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:31.603410    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3f574ea-5fca-4629-8b0b-18755a5c0caa-sys\") pod \"perf-node-gather-daemonset-kdz5p\" (UID: \"d3f574ea-5fca-4629-8b0b-18755a5c0caa\") " pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-kdz5p"
Sep 18 17:56:31.603576 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:31.603554    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d3f574ea-5fca-4629-8b0b-18755a5c0caa-proc\") pod \"perf-node-gather-daemonset-kdz5p\" (UID: \"d3f574ea-5fca-4629-8b0b-18755a5c0caa\") " pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-kdz5p"
Sep 18 17:56:31.603704 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:31.603688    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgx5l\" (UniqueName: \"kubernetes.io/projected/d3f574ea-5fca-4629-8b0b-18755a5c0caa-kube-api-access-bgx5l\") pod \"perf-node-gather-daemonset-kdz5p\" (UID: \"d3f574ea-5fca-4629-8b0b-18755a5c0caa\") " pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-kdz5p"
Sep 18 17:56:31.704996 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:31.704956    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3f574ea-5fca-4629-8b0b-18755a5c0caa-lib-modules\") pod \"perf-node-gather-daemonset-kdz5p\" (UID: \"d3f574ea-5fca-4629-8b0b-18755a5c0caa\") " pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-kdz5p"
Sep 18 17:56:31.705406 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:31.705387    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3f574ea-5fca-4629-8b0b-18755a5c0caa-sys\") pod \"perf-node-gather-daemonset-kdz5p\" (UID: \"d3f574ea-5fca-4629-8b0b-18755a5c0caa\") " pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-kdz5p"
Sep 18 17:56:31.705558 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:31.705545    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d3f574ea-5fca-4629-8b0b-18755a5c0caa-proc\") pod \"perf-node-gather-daemonset-kdz5p\" (UID: \"d3f574ea-5fca-4629-8b0b-18755a5c0caa\") " pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-kdz5p"
Sep 18 17:56:31.705658 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:31.705643    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgx5l\" (UniqueName: \"kubernetes.io/projected/d3f574ea-5fca-4629-8b0b-18755a5c0caa-kube-api-access-bgx5l\") pod \"perf-node-gather-daemonset-kdz5p\" (UID: \"d3f574ea-5fca-4629-8b0b-18755a5c0caa\") " pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-kdz5p"
Sep 18 17:56:31.705820 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:31.705809    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d3f574ea-5fca-4629-8b0b-18755a5c0caa-podres\") pod \"perf-node-gather-daemonset-kdz5p\" (UID: \"d3f574ea-5fca-4629-8b0b-18755a5c0caa\") " pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-kdz5p"
Sep 18 17:56:31.706072 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:31.706057    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d3f574ea-5fca-4629-8b0b-18755a5c0caa-podres\") pod \"perf-node-gather-daemonset-kdz5p\" (UID: \"d3f574ea-5fca-4629-8b0b-18755a5c0caa\") " pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-kdz5p"
Sep 18 17:56:31.706182 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:31.705337    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3f574ea-5fca-4629-8b0b-18755a5c0caa-lib-modules\") pod \"perf-node-gather-daemonset-kdz5p\" (UID: \"d3f574ea-5fca-4629-8b0b-18755a5c0caa\") " pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-kdz5p"
Sep 18 17:56:31.706301 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:31.706289    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3f574ea-5fca-4629-8b0b-18755a5c0caa-sys\") pod \"perf-node-gather-daemonset-kdz5p\" (UID: \"d3f574ea-5fca-4629-8b0b-18755a5c0caa\") " pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-kdz5p"
Sep 18 17:56:31.706785 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:31.706741    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d3f574ea-5fca-4629-8b0b-18755a5c0caa-proc\") pod \"perf-node-gather-daemonset-kdz5p\" (UID: \"d3f574ea-5fca-4629-8b0b-18755a5c0caa\") " pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-kdz5p"
Sep 18 17:56:31.717107 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:31.717077    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgx5l\" (UniqueName: \"kubernetes.io/projected/d3f574ea-5fca-4629-8b0b-18755a5c0caa-kube-api-access-bgx5l\") pod \"perf-node-gather-daemonset-kdz5p\" (UID: \"d3f574ea-5fca-4629-8b0b-18755a5c0caa\") " pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-kdz5p"
Sep 18 17:56:31.844898 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:31.844787    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-kdz5p"
Sep 18 17:56:32.154194 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:32.153150    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xbl54/perf-node-gather-daemonset-kdz5p"]
Sep 18 17:56:32.514411 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:32.514313    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-kdz5p" event={"ID":"d3f574ea-5fca-4629-8b0b-18755a5c0caa","Type":"ContainerStarted","Data":"4721e601edfe3adbc26501c6d4bdec3b18fd0b6a20446f92b2387ce67a4be98e"}
Sep 18 17:56:32.514411 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:32.514375    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-kdz5p" event={"ID":"d3f574ea-5fca-4629-8b0b-18755a5c0caa","Type":"ContainerStarted","Data":"62b965ccbe0872606fd59eb4b96934f39605bc51146cc142b56b1e8c119f25d9"}
Sep 18 17:56:32.515063 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:32.515028    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-kdz5p"
Sep 18 17:56:32.539235 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:32.539188    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-kdz5p" podStartSLOduration=1.539172684 podStartE2EDuration="1.539172684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:56:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:56:32.53621542 +0000 UTC m=+2505.230142971" watchObservedRunningTime="2026-09-18 17:56:32.539172684 +0000 UTC m=+2505.233100211"
Sep 18 17:56:34.678638 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:34.678606    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-s566m_80573a4b-9369-42f3-94b4-c0673f122e62/dns/0.log"
Sep 18 17:56:34.730975 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:34.730947    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-s566m_80573a4b-9369-42f3-94b4-c0673f122e62/kube-rbac-proxy/0.log"
Sep 18 17:56:34.928678 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:34.928610    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zjhnk_fa3b7283-8f5d-4d10-a7ab-6bd10036542f/dns-node-resolver/0.log"
Sep 18 17:56:35.669589 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:35.669553    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-86b9797894-fbhdx_9e770835-5d2a-4225-8465-88b59b66badb/registry/0.log"
Sep 18 17:56:35.811305 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:35.811267    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z6nxd_2af39eb4-05e3-4eca-87d1-87558b5ae1c1/node-ca/0.log"
Sep 18 17:56:37.592318 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:37.592294    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65b7cd5988-ng2jw_812e1c0f-7f7a-495a-96c7-7e03d0b2599c/kube-auth-proxy/0.log"
Sep 18 17:56:37.829851 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:37.829819    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76465d565b-r2rdc_63db05d0-62bd-4fe6-999f-0d19ef2ff266/router/0.log"
Sep 18 17:56:38.921055 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:38.921033    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nwncv_1549ca4a-cf93-4295-8f95-c869e2b6a1c6/serve-healthcheck-canary/0.log"
Sep 18 17:56:39.540215 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:39.540155    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-kdz5p"
Sep 18 17:56:39.662374 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:39.662341    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-v4nfn_cdd63e16-6865-47a1-86ca-c1870d2eae7d/insights-operator/0.log"
Sep 18 17:56:39.663041 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:39.663020    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-v4nfn_cdd63e16-6865-47a1-86ca-c1870d2eae7d/insights-operator/1.log"
Sep 18 17:56:39.713764 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:39.713728    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5lm2g_08f50d0f-2329-428f-ab3e-64159a5bfe35/kube-rbac-proxy/0.log"
Sep 18 17:56:39.737389 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:39.737361    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5lm2g_08f50d0f-2329-428f-ab3e-64159a5bfe35/exporter/0.log"
Sep 18 17:56:39.763120 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:39.763092    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5lm2g_08f50d0f-2329-428f-ab3e-64159a5bfe35/extractor/0.log"
Sep 18 17:56:41.841603 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:41.841560    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65b7cd5988-ng2jw_812e1c0f-7f7a-495a-96c7-7e03d0b2599c/kube-auth-proxy/0.log"
Sep 18 17:56:41.949629 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:41.949579    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76465d565b-r2rdc_63db05d0-62bd-4fe6-999f-0d19ef2ff266/router/0.log"
Sep 18 17:56:43.542328 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:43.542297    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9496889b9-8mhvz_f0748df9-cc4f-4426-9035-21be3c6519e7/manager/0.log"
Sep 18 17:56:43.680939 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:43.680896    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9496889b9-8mhvz_f0748df9-cc4f-4426-9035-21be3c6519e7/copy-manifests/0.log"
Sep 18 17:56:44.009800 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:44.009768    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65b7cd5988-ng2jw_812e1c0f-7f7a-495a-96c7-7e03d0b2599c/kube-auth-proxy/0.log"
Sep 18 17:56:44.125853 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:44.125825    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76465d565b-r2rdc_63db05d0-62bd-4fe6-999f-0d19ef2ff266/router/0.log"
Sep 18 17:56:45.212679 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:45.212651    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5b67bbc446-crkqn_01955b9b-e9a1-456b-b671-a20d74c803d8/manager/0.log"
Sep 18 17:56:45.315476 ip-10-0-139-122 kubenswrapper[2569]: I0918 17:56:45.315446    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-76k8x_d578c9ff-051b-453e-936c-d5d1f409a042/openshift-lws-operator/0.log"